Data Engineer

Our Company

Creospan is a subsidiary of Creospan Inc., our parent company, headquartered in Chicago, IL. From our humble beginnings in 1999 with just a handful of employees and a mission to help our clients leverage emerging web technologies to build next-generation products technology has changed dramatically, yet our curiosity has remained constant. Our expertise spans across Telecom, Technology, Manufacturing, Ecommerce, Insurance, Banking, Transportation, and Healthcare domains.

To find out more about us, visit our website: www.creospan.comJob Title: Azure Databricks Engineer

Experience: 68 Years

Location: Pune, India (Hybrid)

Employment Type: Full Time

Key Responsibilities

Design, develop, and maintain scalable data pipelines using Azure Databricks

Build and optimize batch and streaming ETL/ELT workflows using Apache Spark

Develop data processing solutions using PySpark, Spark SQL, and Delta Lake

Integrate data from multiple Azure sources such as Azure Data Lake Storage (ADLS Gen2), Azure

SQL, and APIs

Optimize Spark jobs for performance, cost efficiency, and reliability on Azure

Implement data quality, validation, and reconciliation frameworks

Collaborate with data architects, analysts, and business stakeholders

Ensure adherence to data security, governance, and compliance standards

Support production deployments and troubleshoot pipeline failures

Required Skills & Qualifications

68 years of experience in Data Engineering

Strong hands-on experience with Azure Databricks (mandatory)

Deep expertise in Apache Spark, PySpark, and Spark SQL

Experience working with Azure Data Lake (ADLS Gen2)

Hands-on experience with Azure Data Factory (ADF) for orchestration

Strong SQL skills and experience with relational databases

Experience with CI/CD pipelines (Azure DevOps preferred)

Working knowledge of cloud security and access controls on Azure