Lead Data Engineer - PySpark

Job Summary

We are looking for an experienced Data Engineer with 7+ years of expertise in building scalable data pipelines and modern data platforms. The ideal candidate will have strong hands-on experience in PySpark, SQL, Python, Airflow, dbt, and Snowflake, with the ability to design robust data architectures that support analytics and business intelligence needs.

Responsibilities

• Design, develop, and maintain scalable and efficient data pipelines using PySpark and Python
• Build and optimize data workflows and orchestration pipelines using Apache Airflow
• Develop and manage data transformation models using dbt (Data Build Tool)
• Implement and maintain data warehouse solutions on Snowflake
• Write complex and optimized SQL queries for data extraction, transformation, and analysis
• Ensure data quality, integrity, and reliability across pipelines and systems
• Collaborate with Data Scientists, Analysts, and Business teams to understand data requirements and deliver solutions
• Optimize data processing performance and cost efficiency in cloud environments
• Implement best practices for data governance, security, and compliance
• Troubleshoot and resolve data-related issues in production environments
• Mentor junior engineers and contribute to improving engineering standards and practices

Required Skills

• 7+ years of experience in Data Engineering or related roles
• Strong proficiency in Python and PySpark for large-scale data processing
• Advanced knowledge of SQL for data manipulation and performance tuning
• Hands-on experience with Apache Airflow for workflow orchestration
• Strong experience with dbt for data transformation and modeling
• Expertise in Snowflake including data modeling, performance tuning, and cost optimization
• Experience working with cloud platforms (AWS, Azure, or GCP)
• Strong understanding of data warehousing concepts and ETL/ELT frameworks
• Familiarity with version control systems (e.g., Git) and CI/CD pipelines
• Excellent problem-solving and analytical skills