Senior/Lead Data Engineer

Job Title Senior/Lead Data Engineer

Location: Hyderabad/Bangalore(Hybrid)

Job Type : Fulltime

** Immediate joiners or within 15 days**

Job Description:

We are seeking an experienced Data Engineer Senior/Lead to design, develop, and maintain scalable data solutions on Azure and Databricks as part of our enterprise data modernization initiatives. The ideal candidate will have a strong background in data pipeline development, data integration frameworks, and cloud-based data engineering, with deep expertise in tools such as Databricks, Azure Data Factory, Alteryx, Ab Initio, Talend, and Informatica.

This role will lead the design and delivery of high-performance, governed data architectures for large-scale enterprise clients, driving data reliability, compliance, and analytics readiness.

Key Responsibilities

• Design and implement scalable and reusable data pipelines using Databricks (PySpark, Delta Lake) and Azure Data Factory, ensuring high throughput and low latency.
• Develop and maintain Common Data Platforms (CDP) and Medallion Architecture-based data lakes to consolidate legacy systems and improve data availability.
• Create and manage backend data services using Python, Spark, or C#, ensuring performance, reliability, and scalability.
• Develop data ingestion frameworks using metadata-driven and schema-drifthandling designs to support multi-source data integration.
• Lead data integration and migration projects using Talend, Ab Initio, Alteryx, and Informatica, modernizing traditional ETL workflows into cloud-native pipelines.
• Implement data quality, lineage, and governance frameworks (Unity Catalog, Data Stewardship, Data Catalog, MDM).
• Design and maintain GDPR/CCPA-compliant data processing and ensure adherence to global data privacy standards.
• Build and optimize Feature Store Data Marts to support AI/ML analytics teams and downstream BI consumers.
• Collaborate with cross-functional teams (Data Scientists, BI Developers, Product Owners) to define and deliver data-driven business outcomes.
• Mentor junior engineers in best practices of data engineering, CI/CD, and infrastructure-as-code deployments.
• Utilize Redis and caching mechanisms to optimize performance and reduce load latency in analytic workloads.
• Take end-to-end ownership of assigned modules ensuring timely delivery, documentation, and performance optimization.

Technical Environment

• Cloud Platforms: Microsoft Azure (Data Factory, Synapse, Data Lake, Databricks), AWS (S3, EC2, Glue, Redshift)
• Data Engineering Tools: Databricks (DLT, Auto Loader, Unity Catalog), Talend, Ab Initio, Informatica, Alteryx
• Programming Languages: Python, PySpark, SQL, Unix Shell Scripting
• Data Modeling: Dimensional, Relational, and Medallion architecture
• DevOps / CI-CD: GitHub Actions, Azure DevOps, Databricks Asset Bundles (DABs)
• Data Governance & Security: RBAC/ABAC, Data Stewardship, MDM, GDPR/CCPA compliance
• Monitoring & Scheduling: Airflow, Logic Apps, Azure Monitor

Required Qualifications

• 10+ years of experience in Data Engineering, Data Architecture, or ETL Development.
• 5 plus years Experience or familiarity using dbt.
• Proven experience leading Azure + Databricksbased modernization or migration programs.
• Deep understanding of data integration tools (Talend, Informatica, Ab Initio, Alteryx) and metadata-driven pipeline design.
• Hands-on experience with PySpark, SQL, and Azure Data Services.
• Strong understanding of data modeling, data quality, and governance frameworks.
• Experience with real-time and batch data ingestion, including Kafka, Kinesis, or Event Hub.
• Excellent problem-solving, communication, and stakeholder management skills.

Preferred Qualifications

• Bachelors degree in computer science, or a related field
• Strong analytical and problem-solving skills
• Azure Data Engineer Certification or equivalent
• Familiarity with AI/ML data preparation and Feature Store design
• Experience implementing cost optimization and cluster governance on Databricks

Why Join Us

• Be part of a data modernization journey with a global enterprise brand.
• Work with cutting-edge cloud and data engineering tools.
• Collaborate with a highly skilled, cross-functional team of architects, analysts, and data scientists.
• Opportunity to lead and innovate within a large-scale, multi-domain data transformation initiative.