Lead Data Engineer

Lead Data Engineer

Experience: 10+ Years

Location: Bangalore / Hyderabad / Noida

Mandatory Skills: Databricks, AWS, PySpark, Airflow

Role Overview

We are seeking a Lead Data Engineer to design, build, and maintain scalable data platforms and pipelines. This role requires strong hands-on expertise in cloud-based data engineering, distributed data processing, and workflow orchestration, along with the ability to guide teams and deliver high-quality data solutions in a fast-paced environment.

Core Responsibilities

• Design, develop, and manage scalable and reliable data pipelines using PySpark and Databricks .
• Build and maintain data orchestration workflows using Apache Airflow .
• Architect, implement, and optimize cloud-based data solutions on AWS .
• Work extensively with AWS services including S3, Glue, Redshift, EMR, Lambda, and IAM .
• Ensure high performance, reliability, and scalability of large-scale data processing systems.
• Perform data modeling, SQL optimization, and performance tuning.
• Collaborate with cross-functional teams to understand data requirements and deliver robust solutions.
• Lead technical discussions, mentor junior engineers, and provide technical guidance.
• Implement best practices for data quality, security, and governance.
• Support CI/CD pipelines and follow version control best practices using Git and automated deployments.
• Contribute to containerized deployments using Docker/Kubernetes where applicable.

Skills & Qualifications

• Bachelors or Masters degree in Computer Science, Engineering, or a related field.
• 4+ years of hands-on experience in Data Engineering .
• Strong expertise in PySpark, Databricks, and Apache Airflow .
• Solid hands-on experience with AWS cloud services .
• Strong proficiency in SQL and data performance tuning.
• Experience with CI/CD pipelines , version control (Git).
• Exposure to Docker and Kubernetes is an added advantage.
• Strong analytical, problem-solving, and communication skills.
• Ability to work effectively in a fast-paced, collaborative environment.