Azure Data Engineer

Creospan is a growing tech collective of makers, shakers, and problem solvers, offering solutions today that will propel businesses into a better tomorrow. Tomorrows ideas, built today! In addition to being able to work alongside equally brilliant and motivated developers, our consultants appreciate the opportunity to learn and apply new skills and methodologies to different clients and industries.

Title: DATAENGINEER - AZURE DATABRICKS

Location: Pune

• Looking for Immediate joiners or someone already serving notice.

Job Title: Azure Databricks Engineer

Experience: 610 Years

Location: Pune, India (Hybrid)

Employment Type: Full Time

About the Role

We are seeking an experienced Azure Databricks Engineer to join its data engineering team in Pune. This role focuses on building, optimizing, and supporting enterprise-scale data platforms on Microsoft Azure, leveraging Azure Databricks, Apache Spark, and modern data engineering practices. The engineer will work closely with global teams to deliver secure, scalable, and high-performance analytics solutions.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using Azure Databricks

Build and optimize batch and streaming ETL/ELT workflows using Apache Spark

Develop data processing solutions using PySpark, Spark SQL, and Delta Lake

Integrate data from multiple Azure sources such as Azure Data Lake Storage (ADLS Gen2), Azure SQL, and APIs

Optimize Spark jobs for performance, cost efficiency, and reliability on Azure

Implement data quality, validation, and reconciliation frameworks

Collaborate with data architects, analysts, and business stakeholders

Ensure adherence to data security, governance, and compliance standards

Support production deployments and troubleshoot pipeline failures

Required Skills & Qualifications

68 years of experience in Data Engineering

Strong hands-on experience with Azure Databricks (mandatory)

Deep expertise in Apache Spark, PySpark, and Spark SQL

Experience working with Azure Data Lake (ADLS Gen2)

Hands-on experience with Azure Data Factory (ADF) for orchestration

Strong SQL skills and experience with relational databases

Experience with CI/CD pipelines (Azure DevOps preferred)

Working knowledge of cloud security and access controls on Azure