Senior Data Engineer

Role: Senior Data Engineer

Experience Range: 3 to 9 years

Location: Bangalore

As a Senior Data Engineer, you will be responsible for building and delivering a Lakehouse-based data pipeline. This role is hands-on and focuses on implementing real-time and batch data ingestion, processing, and delivery workflows, while also ensuring monitoring, observability, and data quality across the pipeline.

Must have skills:

• 3+ years of hands-on experience in building large-scale data pipelines.
• Strong experience with Spark Streaming, AWS Glue, and EMR for processing both real-time and batch workloads
• Proficiency in Pyspark/Python, including building Kafka producers for data ingestion
• Experience working with Confluent Kafka/spark streaming for streaming ingestion from on-premise sources.
• Solid understanding of AWS services including S3, Redshift, Glue, CloudWatch, and Secrets Manager
• Experience working with Medallion architecture and hybrid data destinations (e.g., Redshift and on-premise Oracle)
• Ability to implement monitoring dashboards and observability for pipelines using tools like CloudWatch or Datadog
• Strong SQL skills for validating data accuracy and developing job-level metrics
• Experience with building alerting mechanisms for pipeline failures and performance issues
• Strong collaboration and communication skills
• Proven ability to take ownership of technical deliverables, driving execution from design to deployment
• Experience in guiding junior developers or data engineers, providing technical direction and code reviews to uphold best practices
• AWS Data Engineering - Associate Certified.

Good to have skills:

• Experience with orchestration tools like Apache Airflow or AWS Step Functions
• Exposure to Bigdata stacks like Sqoop,HDFS,Hive,Nifi
• Exposure to Terraform for infrastructure automation
• Familiarity with CI/CD pipelines for data workflows