Data Engineer II

Job Title: Data Engineer II

Experience: 35 Years

Location: Bangalore

Team: Data Platform & Engineering

Job Overview

We are seeking a highly skilled Data Engineer II to join our Data Platform team. In this role, you will be responsible for designing, building, and optimizing scalable data infrastructure capable of handling large-scale datasets. You will work on high-throughput distributed systems and contribute to modern cloud-native and AI-driven data solutions.

Key Responsibilities

Data Pipeline Development & Optimization

• Design, develop, and maintain scalable ETL/ELT pipelines using Scala and Apache Spark/Flink (Core, SQL, Streaming).
• Optimize Spark jobs and SQL queries for performance, scalability, and efficient resource utilization.
• Build and manage data pipelines for processing large-scale datasets with low latency.
• Implement modern Lakehouse architectures using Apache Iceberg, Hudi, or Delta Lake.

Data Management & Quality

• Design and implement data models using Medallion Architecture (Bronze, Silver, Gold layers).
• Ensure data accuracy and reliability through validation frameworks such as Deequ or Great Expectations.
• Monitor data pipelines with observability tools to track data freshness, lineage, and quality.

Cloud-Native Engineering

• Deploy and manage data workloads on GCP (DataProc, GCS, BigQuery) or equivalent cloud platforms.
• Work with Kubernetes (K8s) and Docker for containerized data processing.
• Contribute to Infrastructure as Code (IaC) and automation for scalable deployments.

Collaboration & Innovation

• Collaborate with cross-functional teams including architects, product managers, and data scientists in an Agile environment.
• Participate in code reviews to maintain high coding standards and best practices.
• Explore and implement GenAI and automation workflows to enhance data engineering processes.

Required Qualifications

Experience

• 35 years of hands-on experience in Data Engineering.

Technical Skills

• Strong proficiency in Scala and Apache Spark (Batch & Streaming).
• Solid understanding of SQL and distributed computing concepts.
• Experience with GCP (DataProc, GCS, BigQuery) or equivalent cloud platforms (AWS/Azure).
• Hands-on experience with Kubernetes and Docker.

Architecture & Data Modeling

• Experience with Lakehouse table formats (Iceberg, Hudi, Delta Lake).
• Strong understanding of data warehousing concepts (Star schema, Snowflake schema).

Soft Skills

• Strong problem-solving and analytical skills.
• Ability to work independently and in a team environment.
• Effective communication and collaboration skills.

Education

• Bachelors or Masters degree in Computer Science, Information Technology, Engineering, or a related field.