Azure Data Engineer Lead

Job Title: Azure Data Engineer Lead

Experience: 8-12 years

Job Summary:

We are seeking a skilled and motivated Senior Azure Data Engineer with 8-12 years of relevant experience. Candidate will play a pivotal role in designing, implementing, and optimizing large-scale data solutions on the Microsoft Azure platform. Candidate will work closely with other cross-functional teams to ensure data solutions are scalable, secure, and high-performance. The ideal candidate will have a strong background in data engineering, deep expertise in Azure Databricks and proven leadership skills to guide and mentor the team.

Key Responsibilities:

• Architect and Design Data Solutions: Lead the design and implementation of scalable, secure and high-performing data solutions on Azure, ensuring alignment with business goals and technical requirements.
• Data Pipeline Development: Build and optimize end-to-end data pipelines to collect, process and analyze data from various sources using Azure Databricks, Azure Data Factory and other Azure data services.
• Cloud Infrastructure Management: Architect and maintain cloud data infrastructure using Azure Data Lake, Azure Synapse Analytics, Azure SQL Data Warehouse and Azure Blob Storage.
• Collaborate with Cross-functional Teams: Work closely with data architects, business analysts and developers to gather requirements, define data models and ensure smooth integration of data across systems.
• Design and build scalable data pipelines to support AI/ML use cases, including feature engineering, model training datasets, and real-time inference workflows using Azure-native services.
• Collaborate with Data Scientists and ML Engineers to productionize machine learning models by implementing robust data workflows, feature stores, model monitoring pipelines, and MLOps best practices on Azure (e.g., integration with Azure ML, CI/CD for ML pipelines).
• Develop and optimize data architectures to support advanced analytics, AI, and Generative AI workloads, ensuring data quality, governance, lineage, and compliance across structured and unstructured data sources.
• Performance Tuning and Optimization: Continuously monitor and optimize the performance of data workflows and pipelines to ensure efficient data processing.
• Data Governance & Security: Implement best practices for data governance, security and compliance, ensuring that data solutions adhere to company and regulatory standards.
• Mentor and Lead Teams: Provide technical leadership and mentorship to junior data engineers, guiding them in the best practices of data engineering and cloud technologies.

Required Qualifications:

• Bachelor's or Masters degree in Computer Science, Engineering, or a related field
• 8-12 years of experience in data engineering, with at least 5+ years of hands-on experience with Microsoft Azure cloud technologies.
• Strong experience with Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, Azure Blob Storage, Azure Data Lake and other related Azure services.
• Hands-on experience using these services: PySpark, Notebooks, Structured Streaming, Unity Catalog, Delta Live Tables, Workflows and SQL Warehouse
• In-depth knowledge of data modeling, ETL/ELT processes, data warehousing and data pipeline architecture.
• Proven experience in building and optimizing large-scale data pipelines in a cloud environment.
• Strong experience with SQL, NoSQL databases and big data processing technologies (e.g., Spark, Hadoop, Kafka).
• Hands-on experience with data transformation, data integration and data quality frameworks.
• Expertise in data security and compliance standards in a cloud-based environment.
• Ability to design and implement data governance strategies for large datasets.
• Strong leadership, communication and collaboration skills, with a proven ability to guide teams and manage complex projects.

Good to have:

• Microsoft Azure Data Engineer Certification
• Databricks Certified Data Engineer Professional
• Familiarity with GenAI Concepts
• Familiar with DevOps Practices