Data Engineer

LinkedIn Job Post Data Engineer

Job Title: Data Engineer

Experience: 4+ Years

Location: India / Hybrid (as per client requirement)

Job Summary

We are looking for a skilled Data Engineer with strong experience in building and operationalizing large-scale data pipelines and ETL workflows. The ideal candidate should have hands-on expertise with PySpark, Apache Airflow, and AWS data services to design, develop, and maintain scalable data platforms and analytics solutions.

Key Responsibilities

• Build scalable ETL/ELT data pipelines using PySpark on distributed processing frameworks.
• Design and orchestrate workflows using Apache Airflow including DAG creation, scheduling, and monitoring.
• Develop data ingestion and transformation jobs using AWS Glue.
• Manage secure and compliant data access using AWS Lake Formation.
• Maintain and optimize AWS Glue Data Catalog for schema and metadata management.
• Collaborate with analytics teams to publish datasets for BI and dashboards.
• Build and support data visualizations using Amazon QuickSight.
• Ensure data quality, reliability, and performance across all data pipelines.

Required Skills

• Strong hands-on experience with PySpark for large-scale data processing
• Deep knowledge of Apache Airflow DAGs and workflow orchestration
• Expertise in AWS Glue (ETL jobs, Glue Studio, Crawlers)
• Experience with AWS Lake Formation for data governance
• Familiarity with AWS Glue Data Catalog and metadata management

Nice to Have

• Experience with AWS S3, Athena, Redshift, or EMR
• Knowledge of Python-based automation and testing
• Exposure to DevOps / Infrastructure as Code (Terraform or CloudFormation)

Top 5 Mandatory Skills (For Naukri / LinkedIn Search)

1. PySpark
2. Apache Airflow
3. AWS Glue
4. AWS Lake Formation
5. AWS Glue Data Catalog