Senior Data Engineer

Job Role: Senior Data Engineer

Experience: 6+ Years

Location: South Bangalore, KA

Key Responsibilities:

Design, develop, and optimize ETL/ELT pipelines using Azure Databricks and Azure Data Factory.

Build scalable batch and streaming data processing solutions using Python and PySpark.

Implement and maintain Delta Lake and Lakehouse architectures.

Design and manage data models, including Dimensional Modeling and SCD Type 1 & Type 2 implementations.

Develop data storage solutions using Azure Data Lake Storage (Gen2, Delta Tables, and Parquet formats).

Integrate external systems through REST APIs and secure authentication mechanisms.

Build and maintain CI/CD pipelines using Azure DevOps and Git.

Implement data governance, security, and access control best practices.

Optimize SQL queries, data processing performance, and workload efficiency.

Collaborate with cross-functional teams to deliver scalable and reliable data solutions.

Required Skills:

6+ years of experience in Data Engineering.

Strong hands-on experience with Azure Databricks, Python, and PySpark.

Experience with Azure Data Factory (ADF) and Azure Data Lake Storage.

Strong knowledge of Azure DevOps, Git, and CI/CD implementation.

Experience with REST API integrations and authentication mechanisms.

Strong SQL skills including query optimization, CTEs, and window functions.

Experience with Delta Lake concepts including ACID transactions, Time Travel, Vacuum, and Z-Ordering.

Understanding of distributed Data Processing, Data Governance, and security best practices.

Experience with Databricks Asset Bundles (DABs).

Exposure to event-driven architectures and streaming data solutions.