Data Engineer

About the Role

We are seeking a highly skilled Data Engineer with deep expertise in the Microsoft Azure ecosystem and Microsoft Fabric to design, build, and optimize scalable data pipelines and Medallion lakehouse architectures. The candidate will provide thought leadership in transitioning legacy environments to unified Fabric workspaces, delivering robust, secure, and high-performance solutions that empower AI and analytics.

Key Responsibilities

• Fabric & Lakehouse Implementation: Design and maintain unified data environments using Microsoft Fabric (OneLake, Lakehouses, and Warehouses) and Azure Synapse.
• Pipeline Orchestration: Develop scalable data pipelines using Fabric Data Factory and Azure Data Factory to ingest data from diverse sources (APIs, On-premises, Cloud).
• Architecture Optimization: Build and optimize Medallion architectures (Bronze/Silver/Gold) using Delta Lake and Fabric Notebooks (Spark).
• Thought Leadership: Work with stakeholders to translate business requirements into technical roadmaps, specifically advising on when to use Fabric vs. Azure Databricks.
• Unified Governance: Leverage Microsoft Purview and Fabrics native security features to ensure data quality, consistency, and RBAC/Sensitivity labeling.
• Infrastructure & Automation: Implement Infrastructure as Code (IaC) using Terraform or Bicep for automated provisioning of Fabric capacities and Azure resources.
• Advanced Analytics Support: Design Star Schemas and utilize DirectLake mode in Power BI to provide high-performance reporting for data scientists and analysts.

Required Technical Skills

• Microsoft Fabric Ecosystem: Expert-level knowledge of OneLake, Fabric Capacities, Lakehouse/Warehouse artifacts, and Shortcuts.
• Azure Data Services: Proven experience with Azure Data Factory, Azure Databricks, and Azure Data Lake Storage Gen2.
• Data Processing: Strong proficiency in PySpark (Spark SQL & DataFrames) for complex transformations and performance tuning.
• Languages: Advanced SQL and Python skills for pipeline development and orchestration.
• DevOps & IaC: Experience with Terraform/Bicep and CI/CD practices using Azure DevOps or GitHub Actions.
• Real-Time Data: Experience with streaming data using Fabric Eventstreams or Azure Event Hubs.

Soft Skills & Mindset

• Strategic Thinking: Ability to navigate the evolving Microsoft roadmap and choose the right tool for the specific scale/cost requirement.
• Collaboration: Strong ability to bridge the gap between "Pro-code" engineering and "Low-code" analytics users.
• Communication: Robust client management skills with the ability to explain complex cloud architectures to non-technical stakeholders.

Preferred Qualifications

• Certifications: DP-600 (Implementing Analytics Solutions Using Microsoft Fabric) or DP-203 (Azure Data Engineer).
• Advanced Tools: Experience with Unity Catalog, Delta Live Tables, or Fabrics Copilot features.