Lead Data Engineer

Lead Data Engineer

Location: Banglore/Pune (Hybrid)

Mode: Hybrid

Shift Timing: 2 PM to 11 PM

Experience: 7-12 years. 3+ years in cloud data platforms and Databricks.

Purpose:

We are seeking a hands-on Technical Lead to drive the ingestion of high-volume mainframe RPC data into Databricks , enabling scalable machine learning workflows. This role is critical to building a robust data foundation for training thousands of AI models that detect anomalous behavior across applications, services, and functions.

Key Responsibilities

• Mainframe Data Ingestion:
• Design and implement scalable pipelines to extract, parse, and ingest RPC logs and technical attributes from mainframe systems into Delta Lake on Databricks.
• ML-Ready Data Engineering:
• Transform and structure data for time-series modelling and anomaly detection across thousands of models.
• ML Workflow Integration:
• Collaborate with ML engineers to ensure data pipelines support SARIMA, ANN, and other model types; enable automated retraining and scoring.
• Performance Optimization:
• Tune Spark jobs, Delta Lake storage, and cluster configurations for billions of records and real-time aggregation.
• FinOps & Cost Control:
• Monitor and optimize Databricks resource usage; implement auto-scaling and cost-aware job scheduling.
• Monitoring & Alerting:
• Integrate Databricks-native alerting for pipeline health, data anomalies, and job failures.

Required Skills

• Strong hands-on experience with Databricks , Apache Spark , Delta Lake , and MLflow
• Proven expertise in mainframe data integration (e.g., SMF, RPC logs, VSAM, DB2)
• Strong Python and PySpark programming skills

Preferred Skills

• Databricks experience with CI/CD tools GitHub Actions
• Knowledge of FinOps principles for cloud cost optimization