Lead Data Engineer (AWS & Databricks)

Job Title: Lead Data Engineer (AWS & Databricks)

Location: Bengaluru

Employment Type: Full-Time

Experience: 10+ Years

Work Mode: Hybrid (3 Days WFO / 2 Days WFH)

Office Location: Bagmane Tech Park, Bengaluru

Notice Period: Immediate to 30 Days

Role Overview

We are looking for an experienced Lead Data Engineer (AWS & Databricks) to drive the enterprise data engineering strategy for a next-generation unified analytics platform spanning digital, retail, and marketplace channels.

This role will own the end-to-end data architecture roadmap, including migration from legacy data platforms to a modern Databricks/Spark Lakehouse ecosystem on AWS, ensuring high KPI alignment and metric consistency across the enterprise.

The ideal candidate will function as both a hands-on technical leader and strategic architect, influencing platform design, governance models, and modernization initiatives at scale.

Key Responsibilities

Architecture & Technical Leadership

• Define the target-state enterprise data architecture using Databricks, Spark, and AWS-native services.
• Lead the migration from legacy data platforms to a Databricks/Spark Lakehouse architecture.
• Design scalable, secure, and cost-efficient data pipelines supporting both batch and streaming workloads.
• Establish architectural standards for data modeling, storage formats, and performance optimization.

Data Engineering & Platform Strategy

• Design and implement ETL/ELT pipelines using Python, Spark, and SQL.
• Build scalable pipelines leveraging AWS S3, Lambda, EMR, and Databricks.
• Enable real-time and near-real-time data processing using Kafka, Kinesis, and Spark Streaming.
• Drive containerized deployments using Docker and Kubernetes.

Orchestration, CI/CD & Infrastructure

• Establish global orchestration standards using Apache Airflow.
• Implement CI/CD pipelines using Git and Jenkins.
• Manage infrastructure provisioning through Infrastructure as Code (Terraform / CloudFormation).
• Enforce best practices for automation, security, and platform reliability.

Data Governance & Enterprise Metrics

• Implement enterprise-wide data governance, lineage, and cataloging frameworks.
• Use Unity Catalog and metadata-driven designs for secure and structured data access.
• Define and manage enterprise KPI frameworks and metric dictionaries.
• Collaborate with analytics, product, and business teams to ensure consistent and trusted insights.

Observability & Operational Excellence

• Implement monitoring, alerting, and observability across data pipelines and platforms.
• Define SLAs, SLOs, and operational playbooks for critical analytics workloads.
• Mentor and guide data engineers and technical teams, ensuring engineering excellence.

Required Skills

• Strong expertise in AWS (S3, Lambda, EMR, Databricks)
• Advanced programming in Python for data engineering and automation
• Strong experience with Spark and distributed data processing
• Advanced SQL for data modeling, complex queries, and performance optimization
• Experience building large-scale ETL/ELT pipelines
• Experience with Kafka, Kinesis, or streaming data architectures
• Experience with Apache Airflow for workflow orchestration
• Experience with Docker and Kubernetes
• Experience implementing data governance and metadata frameworks

Required Experience

• 10+ years of experience in data engineering, distributed systems, and data platform architecture
• Proven experience modernizing legacy data platforms to Databricks/Spark Lakehouse architectures
• Strong hands-on experience designing enterprise-scale data platforms
• Experience working with cloud-based analytics ecosystems

Certifications (Preferred)

At least one certification is preferred. Strong candidates without certification may also be considered.

• Databricks Certified Data Engineer Professional
• AWS Solutions Architect Associate or Professional