Data Engineer

Job Title - Data Engineer

Years of Experience - 5 to 8 Years

Location - Bangalore

Notice Period - Immediate to 30 Days

About the Role

We are looking for a skilled Data Engineer with hands-on experience in building and managing scalable data pipelines across cloud platforms. The ideal candidate should have strong expertise in ETL frameworks and exposure to multiple cloud environments.

Key Responsibilities

Design, develop, and maintain scalable end-to-end ETL/ELT data pipelines (batch and near real-time) across AWS, Azure, and GCP environments (AWS preferred).

Implement robust data ingestion patterns including CDC, API-based, file-based, and streaming integrations ensuring data quality, reliability, and performance.

Develop and maintain cloud-native data solutions using MySQL/SQL Server, Azure Data Factory, MFTS, GCP BigQuery, and GCS (preferred: AWS Glue, Redshift, EC2, RDS).

Build and manage workflow orchestration using Apache Airflow (or equivalent tools) for scheduling, monitoring, and dependency management.

Work on data ingestion, transformation, and integration across multiple enterprise systems and platforms.

Develop CI/CD pipelines for data infrastructure and ETL code using Git-based workflows and Infrastructure-as-Code (IaC) best practices.

Collaborate with cross-functional teams to understand data requirements and deliver scalable and efficient data solutions.

Work closely with security and compliance teams to implement encryption, IAM least privilege access, network segmentation (VPN/VPC), and governance controls.

Build monitoring, alerting, and auditing mechanisms for data pipelines including latency tracking, failure monitoring, retries, DLQ, and data quality metrics.

Optimize data storage, partitioning, and query performance for analytics workloads in BigQuery and Redshift environments.

Troubleshoot production issues, perform root-cause analysis, and implement corrective and preventive actions.

Produce and maintain technical documentation including architecture diagrams, runbooks, and data contracts while supporting onboarding and mentoring activities.

Provide ongoing sustainment and operational support for production data platforms and pipelines.

Required Qualifications

58 years of total experience in data engineering with at least 35 years specifically building cloud-native data pipelines

Strong experience with Azure Cloud services, especially Azure Data Factory (ADF), data ingestion, and transformation workflows

Hands-on expertise with GCP services such as BigQuery, Google Cloud Storage (GCS), and cloud-native analytics solutions

Proficiency in Apache Airflow for workflow orchestration, scheduling, monitoring, and dependency management

Experience implementing batch and near real-time data pipelines using CDC, API, file based, and streaming ingestion patterns

Strong understanding of data engineering concepts, data modeling, pipeline optimization, and performance tuning

Experience with CI/CD pipelines, Git-based workflows, and Infrastructure-as-Code (IaC) practices

Proficiency in SQL and scripting languages such as Python for data processing and automation

Experience with monitoring, troubleshooting, and production support for enterprise data platforms

Preferred Qualifications

Exposure to AWS services such as Glue, Redshift, EC2, and RDS

Experience working in multi-cloud enterprise environments (Azure, GCP, AWS)

Knowledge of data governance, IAM, encryption, and security best practices

Experience handling large-scale distributed and analytics data systems

Strong problem-solving skills and experience working in Agile environments