Lead Data Engineer

Experience: 10+ Years

Employment Type: Full-time

Work Mode: Hybrid

Location: Bengaluru

Role Overview:

We are seeking a Lead Data Engineer to define and drive an enterprise-scale data engineering strategy for a next-generation unified analytics platform spanning digital, physical retail, and marketplace channels.

This role owns the end-to-end data architecture roadmap , including the successful divestiture of Snowflake and transition to a Databricks/Spark Lakehouse on AWS , while ensuring 95%+ KPI alignment and metric consistency across the organization.

You will operate as both a hands-on technical leader and a strategic architect , influencing platform design, governance models, and modernization initiatives at global scale.

Key Responsibilities:

Architecture & Technical Leadership:

• Define the target-state data architecture using Databricks, Apache Spark, and AWS-native services
• Own and execute the Snowflake divestiture strategy , ensuring zero residual footprint and uninterrupted business reporting
• Design highly scalable, secure, and cost-efficient batch and streaming data pipelines
• Establish architectural standards for data modeling, storage formats, and performance optimization

Data Engineering & Platform Strategy:

• Design and implement ETL/ELT pipelines using Python, Spark, and SQL
• Build and optimize pipelines using AWS S3, Lambda, EMR, and Databricks
• Enable real-time and near-real-time data processing using Kafka, Kinesis, and Spark Streaming
• Drive containerized deployments using Docker and Kubernetes

Orchestration, CI/CD & Infrastructure

• Lead workflow orchestration standards using Apache Airflow
• Implement and govern CI/CD pipelines using Git and Jenkins
• Own infrastructure provisioning using Terraform and/or CloudFormation

Data Governance & Enterprise Metrics

• Establish enterprise-wide data lineage, cataloging, and access controls
• Define and manage metric dictionaries and KPI frameworks
• Partner with analytics, product, and business teams to ensure trusted insights and metric alignment

Observability & Operational Excellence

• Implement monitoring, alerting, and observability across data platforms
• Define SLAs, SLOs, and operational playbooks for mission-critical analytics
• Mentor and guide engineers, raising overall engineering standards

Must-Have Qualifications

• 10+ years of experience in data engineering, distributed systems, and platform architecture
• Deep expertise in AWS , including S3, Lambda, EMR, and Databricks
• Advanced Python for data processing, automation, and optimization
• Advanced SQL (complex queries, window functions, performance tuning)
• Proven experience modernizing legacy platforms and migrating to Databricks/Spark Lakehouse architectures
• Strong background in data governance, lineage, and enterprise metrics

Certifications (Mandatory)

• Databricks Certified Data Engineer Professional
• AWS Solutions Architect Associate or Professional (preferred)