Data Engineer

Job Title

Technical Lead Microsoft Fabric Data Engineering

Location

Noida, India

Overview of the Role

As a Technical Lead Microsoft Fabric Data Engineering, you will spearhead the design and implementation of advanced data solutions that harness the capabilities of Microsoft Fabric. Your expertise will drive robust, scalable, and high-quality data pipelines, enabling actionable business intelligence and supporting HCLTechs strategic transformation initiatives. You will play a critical role in ensuring data integrity, optimizing performance, and fostering a data-driven culture across the organization.

Detailed Responsibilities

• Architect, develop, and implement scalable ETL (Extract, Transform, Load) processes using Microsoft Fabric, integrating data from diverse sources into cloud-based data warehouses and lakehouses.
• Collaborate closely with data analysts, engineers, and business stakeholders to define requirements and translate them into effective and sustainable data solutions.
• Build, maintain, and optimize Power BI dashboards and reports, transforming complex datasets into actionable insights for informed business decisions.
• Ensure data quality, integrity, and accuracy throughout the ETL lifecycle, adhering to industry best practices for data management and governance.
• Monitor, troubleshoot, and optimize ETL processes, proactively identifying and resolving issues or performance bottlenecks to maintain high system reliability.
• Document ETL workflows, data mappings, and process modifications to facilitate transparency and clear communication across technical and non-technical teams.
• Implement advanced data transformation and ingestion patterns, including Change Data Capture (CDC), schema evolution, error handling, and robust data operations.
• Uphold best practices in data security, privacy, and compliance throughout all data engineering activities.

Skill Requirements

Must-Have Skills & Qualifications

• Bachelors degree in Computer Science, Information Technology, Engineering, or related field.
• 7+ years of hands-on experience in ETL development, with recent projects utilizing Microsoft Fabric.
• Expertise in Microsoft Fabric Data Pipelines (activities, triggers, parameters) and Notebooks (PySpark/Spark SQL).
• Deep working knowledge of Delta Lake, including schema evolution, MERGE/UPSERT, OPTIMIZE/VACUUM, partitioning, and checkpointing.
• Strong proficiency in CDC ingestion patterns (watermarks, change tables, log-based ingestion).
• Advanced SQL skills (window functions, CTEs) and experience with PySpark for data transformations and performance tuning.
• Demonstrated experience in implementing data quality frameworks: rule patterns (null, range, referential, uniqueness), scorecards, and exception handling.
• Advanced error handling and observability: idempotency, retries, dead-letter handling, and alerting hooks.
• Solid understanding of business intelligence concepts with practical experience in Power BI for data visualization and reporting.
• Knowledge of data warehousing concepts, architectures, and modeling techniques.
• Familiarity with major cloud platforms (Azure, AWS, Google Cloud) and their respective data storage and processing services.
• Excellent analytical, troubleshooting, and problem-solving skills.
• Strong communication skills, with the ability to convey complex technical concepts to non-technical audiences.

Good-to-Have Skills

• Experience with additional ETL tools (e.g., SSIS) and data warehousing solutions.
• Understanding of data governance, data privacy, and compliance standards.
• Familiarity with programming languages such as Python or R for advanced data manipulation.
• Experience working with stored procedures within Fabric transformations.
• Knowledge of metadata-driven pipelines and parameterization (YAML/JSON controlled).
• Awareness of Power BI model readiness (star schema, aggregations).

Other Requirements (Optional)

• Professional certifications in cloud platforms (e.g., Azure Data Engineer Associate, AWS Certified Data Analytics) are advantageous.
• Experience in large-scale enterprise environments is a plus.
• Demonstrated ability to lead and mentor junior engineers or cross-functional teams.