Data Engineer (Azure)_96930

Company:IT Services Organization

Key Skills: Data Engineer, Azure, Python, Pyspark, MySQL, Azure Databricks

Roles and Responsibilities:

• Design and implement scalable, metadata-driven frameworks for data ingestion, quality, and transformation across both batch and streaming datasets.
• Develop and optimize end-to-end data pipelines to process structured and unstructured data, enabling the creation of analytical data products.
• Build robust exception handling, logging, and monitoring mechanisms for better observability and operational support.
• Take ownership of complex modules and lead the development of critical data workflows and components.
• Provide guidance to data engineers and peers on best practices, conduct code reviews, and enforces standards.
• Collaborate with cross-functional teams--including business consultants, data architects, scientists, and application developers--to deliver impactful analytics solutions.

Skills Required:

• 5-10 years of total IT experience, including at least 3 years in big data engineering on Microsoft Azure
• Strong SQL expertise with experience in Azure SQL, Synapse, or similar cloud databases
• Proven experience in building high-performance data pipelines using Azure Databricks (PySpark, Spark SQL)
• Hands-on experience with Azure Data Factory for pipeline orchestration
• Experience with batch and streaming data processing
• Delivered at least one end-to-end Data Lakehouse solution (Medallion Architecture)
• Strong programming and debugging skills in Python, PySpark, and SQL
• Knowledge of data governance, security, and lifecycle management

Good to Have

• Exposure to LLM / Generative AI applications
• Knowledge of NoSQL databases
• Experience supporting BI and Data Science teams
• Azure / Databricks certifications

Education: Bachelor's Degree in related field