AWS Data engineer

Job Summary

We are seeking a skilled Data Engineer with strong experience in AWS, PySpark, SQL, and Airflow. The ideal candidate will be responsible for building scalable data pipelines, optimizing data workflows, and supporting advanced analytics and data-driven decision-making.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark.
• Work with AWS services such as S3, Glue, Lambda, Redshift, and EMR.
• Develop and optimize complex SQL queries for data extraction and transformation.
• Build, schedule, and monitor workflows using Apache Airflow.
• Ensure data quality, integrity, and reliability across data platforms.
• Collaborate with data analysts, scientists, and business teams to understand data requirements.
• Troubleshoot and resolve data-related issues in a timely manner.
• Implement best practices for data governance, security, and performance optimization.

Required Skills & Qualifications

• Strong experience with AWS cloud platform.
• Hands-on experience with PySpark for large-scale data processing.
• Advanced knowledge of SQL (query optimization, joins, indexing).
• Experience with Apache Airflow for workflow orchestration.
• Good understanding of ETL/ELT concepts and data warehousing.
• Experience working with large datasets and distributed systems.
• Familiarity with version control systems (e.g., Git).

Preferred Qualifications

• Experience with data lakes and lakehouse architecture.
• Knowledge of AWS services like Glue, Athena, Redshift, and Kinesis.
• Exposure to CI/CD pipelines and DevOps practices.
• Basic understanding of Python beyond PySpark.
• Experience in Agile/Scrum environments.