Data Engineer

Position: Data Engineer

Experience: 58 years

Location: Bengaluru

MustHave Skills

• SQL (Advanced) Joins, Window functions, query tuning.
• Python ETL scripting, PySpark, automation.
• Apache Kafka Realtime ingestion, stream processing, Kafka Connect.

Data Pipeline & ETL Development

• Design, develop, and maintain batch and realtime ETL/ELT pipelines using Python, PySpark, and Kafka.
• Build data ingestion frameworks to integrate RDBMS, APIs, Kafka streams, flat files, and cloud-native sources.
• Implement and optimize CDC, incremental loads, SCD Type2, partitioning, and bucketing strategies.

Big Data Engineering

• Develop and optimize pipelines using Spark, PySpark, Scala, Hive, Hadoop, and related tooling.
• Handle large-scale distributed datasets with advanced performance techniques such as:
• Broadcast joins
• Shuffle optimization
• Data skew resolution
• Caching & indexing

Cloud Engineering (AWS / Azure)

• Build end-to-end data solutions using:
• AWS: S3, Glue, Redshift, EMR, Lambda
• Azure: ADF, Synapse, Databricks, Data Lake
• Implement orchestration frameworks like Airflow, Oozie, ADF triggers, and Lambda automation.

Data Quality, Governance & Monitoring

• Implement end-to-end data validation, audit checks, logging, and monitoring frameworks.
• Build metadata-driven and reusable ETL frameworks.
• Work with monitoring tools such as CloudWatch or Databricks cluster logs.

Collaboration, Documentation & Agile Delivery

• Work closely with analysts, architects, QA, and business teams in an Agile/Scrum environment.
• Prepare technical documentation, architecture diagrams, and SOPs.
• Participate in code reviews and contribute to best engineering practices.