Azure Databricks Lead

Position Summary:

We are seeking a talented Data Engineer with a strong background in data engineering to join our team. You will play a key role in designing, building, and maintaining data pipelines using a variety of technologies, with a focus on the Microsoft Azure cloud platform.

Responsibilities:

• Design, develop, and implement data pipelines using Azure Data Factory (ADF) or other orchestration tools.
• Write efficient SQL queries to extract, transform, and load (ETL) data from various sources into Databricks.
• Utilize PySpark and Python for complex data processing tasks on large datasets within Azure Databricks.
• Databricks is Mandatory
• Collaborate with data analysts to understand data requirements and ensure data quality.
• Hands-on experience in designing and developing Datalakes and Warehouses
• Implement data governance practices to ensure data security and compliance.
• Monitor and maintain data pipelines for optimal performance and troubleshoot any issues.
• Develop and maintain unit tests for data pipeline code.
• Work collaboratively with other engineers and data professionals in an Agile development environment.

Preferred Skills & Experience:

• Good knowledge of PySpark & working knowledge of Python
• Full stack Azure Data Engineering skills (Azure Data Factory, DataBricks and Synapse Analytics)
• Experience with large dataset handling
• Hands-on experience in designing and developing Datalakes and Warehouses