Lead Data Engineer

KPI Partners is looking for a highly skilled Lead Data Engineer to join our dynamic team. The ideal candidate will have extensive experience in managing and optimizing data pipelines, with strong expertise in Databricks, Airflow, AWS, and Pyspark.

Job Title: Lead Data Engineer ((Databricks, Airflow, AWS, Pyspark)

Location: Hybrid Hyderabad/Bangalore/Pune

Experience: 8+ years

Employment Type: Full-time

Key Responsibilities:

• Lead the design, development, and maintenance of scalable data pipelines that power data analytics and reporting solutions.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and provide solutions that meet business needs.
• Implement best practices for data engineering, including data quality, data governance, and performance optimization.
• Utilize Databricks for data processing and transformation tasks, ensuring high efficiency and reliability.
• Design and manage orchestration workflows with Apache Airflow to automate data processing tasks.
• Work with AWS services such as S3, Redshift, and EMR to build robust data infrastructure.
• Mentor and guide junior data engineers, fostering a culture of continuous learning and improvement within the team.
• Stay up-to-date with emerging data engineering technologies and trends, applying them to improve our data architecture and processes.

Qualifications:

• Bachelors degree in Computer Science, Engineering, or a related field; Masters degree preferred.
• 5+ years of experience in data engineering or related fields, with a focus on pipeline development and management.
• Hands-on experience with Databricks, Airflow, AWS, and Pyspark is essential.
• Strong programming skills in Python, with familiarity in SQL and other languages being a plus.
• Proven experience with database systems, data warehousing, and ETL processes.
• Excellent problem-solving skills and the ability to work independently and collaboratively in a fast-paced environment.
• Strong communication skills, capable of explaining complex technical concepts to non-technical stakeholders.

If youre passionate about building scalable data platforms and leading high-impact projects, wed love to connect!

Interested candidates can share their updated resume at saswati.paul@kpipartners.com