Senior Data Engineer

We are seeking a seasoned Data Engineer to design, build, and operate scalable, reliable, productiongrade data pipelines on Azure. This role requires deep, handson expertise in PySpark, Python, Azure Synapse, and modern data orchestration frameworks (Dagster preferred). The role demands a strong understanding of distributed data processing, cloud-based data lake architectures, and endtoend pipeline ownership, spanning raw data ingestion to analyticsready datasets.

Immediate joiners highly preferred (030 days) || Location: Pune/ Gurgaon (Hybrid: 23 days WFO)

Key competencies:

6+ years of experience in CORE DATA ENGINEERING.

Core Technical Skills

• Strong, hands-on PySpark experience in production environments
• Advanced Python skills for data engineering (beyond notebooks and scripts)
• Practical experience with Azure Synapse Analytics (Pipelines + Spark)
• Solid understanding of distributed systems and cloud data processing

Data Engineering Foundations

• Strong grasp of ETL / ELT design patterns
• Experience with large-scale data lakes (ADLS Gen2 or equivalent)
• Excellent SQL skills for transformation and analytics workflows
• Experience handling structured and semi-structured data

Key Responsibilities:

Distributed Data Engineering (PySpark)

• Design and build distributed data pipelines using PySpark on large-scale datasets
• Develop and optimize pipelines using:

o Spark DataFrames and Spark SQL

o Partitioning, caching, and join optimization strategies

o Efficient read/write patterns for cloud object storage

• Diagnose and remediate performance bottlenecks including skew, shuffle issues, and memory pressure
• Ensure pipelines meet SLAs for latency, reliability, and data quality

Python Engineering for Data Platforms

• Write production-grade Python code for data pipelines, transformations, and utilities
• Apply strong engineering practices:

o Modular code structure and reusable libraries

o Dependency and package management

o Logging, error handling, and testability

• Build framework-agnostic pipeline logic where appropriate (decoupled from orchestration layer)

Azure Synapse & Cloud Data Integration

• Hands-on development with Azure Synapse Analytics, including:

o Synapse Pipelines (data movement and orchestration)

o Synapse Notebooks (Spark-based transformations)

o Linked Services and Integration Runtimes

• Participate in or lead data platform migrations (on-prem Azure, legacy ETL Synapse / Spark)
• Optimize pipeline execution for cost, scalability, and reliability in Azure environments
• Integrate Synapse with ADLS Gen2, SQL pools, and downstream analytics tools

Data Orchestration & Dagster (Preferred)

• Design and operate Dagster-based orchestration using asset-centric patterns
• Implement:

o Software-defined data assets

o Sensors and schedules for event-driven and time-based processing

o Dependency-aware pipeline execution

• Monitor and debug pipeline runs, lineage, and failures
• (Preferred) Experience deploying and operating pipelines in Dagster+ Cloud or similar managed environments

Core Data Engineering & Architecture

• Design robust ETL / ELT pipelines aligned to analytics and reporting needs Job Description for [Role]
• Apply solid data modeling principles (fact/dimension models, curated layers, semantic consistency)
• Architect and maintain data lake solutions using ADLS Gen2 and medallion-style layering
• Write and optimize SQL for transformations, validation, and analytics consumption
• Enforce data quality, schema evolution controls, and observability across the platform

Interested candidates, please share your updated CVs at nihal.upadhyay@acuityanalytics.com