Azure Devops Engineer

Responsibilities

As a Databricks Data Engineer, you will:

• Design, construct, install, test, and maintain highly scalable data management systems using Azure Databricks.
• Develop data processing pipelines using Databricks Notebooks, Spark SQL, and other Databricks utilities.
• Integrate Azure Databricks with other Azure services such as Azure Data Lake Storage, Azure SQL Data Warehouse, and Azure Cosmos DB.
• Ensure optimal performance of data processing and extraction, transformation, and loading (ETL) processes.
• Collaborate with data architects to implement effective data models and schemas within the Databricks environment.
• Monitor data reliability, efficiency, and quality, and implement improvements as needed.
• Work with stakeholders to assist with data-related technical issues and support their data infrastructure needs.
• Develop and maintain documentation related to data pipeline architecture, development processes, and data governance.
• Stay current with industry trends and developments in cloud computing, big data, and Azure Databricks.

Skills 

• Azure certifications such as Azure Data Engineer Associate or Azure Data Scientist Associate.
• Experience with machine learning frameworks and libraries.
• Knowledge of additional big data tools and platforms.
• Familiarity with data visualization tools and techniques.

Qualification 

• Bachelor's  degree in  Engineering or a related field.
• Proven experience as a Data Engineer with a focus on Azure Databricks and related Azure services.
• Strong programming skills in languages such as Python, Scala, or Java.
• Experience with Apache Spark and its integration with Databricks.
• Proficiency in SQL and experience with data modeling and data warehousing concepts.
• Familiarity with DevOps practices and CI/CD pipelines.
• Excellent analytical and problem-solving abilities.
• Effective communication skills and the ability to work collaboratively in a team environment.

Career Level - IC2