Lead Consultant- Databricks Developer !

Genpact

5 - 10 years

Hyderabad

Posted: 18/06/2025

Job Description

Responsibilities

Develop and maintain scalable ETL pipelines using Databricks with a focus on Unity Catalog for data asset management.

Implement data processing frameworks using Apache Spark for large-scale data transformation and aggregation.

Integrate real-time data streams using Apache Kafka and Databricks to enable near real-time data processing.

Develop data workflows and orchestrate data pipelines using Databricks Workflows or other orchestration tools.

Design and enforce data governance policies, access controls, and security protocols within Unity Catalog.

Monitor data pipeline performance, troubleshoot issues, and implement optimizations for scalability and efficiency.

Write efficient Python scripts for data extraction, transformation, and loading.

Collaborate with data scientists and analysts to deliver data solutions that meet business requirements.

Maintain data documentation, including data dictionaries, data lineage, and data governance frameworks.

Minimum Qualifications

Bachelors degree in Computer Science, Data Engineering, or a related field.

experience in data engineering with a focus on Databricks development.

Proven expertise in Databricks, Unity Catalog, and data lake management.

Strong programming skills in Python for data processing and automation.

Experience with Apache Spark for distributed data processing and optimization.

Hands-on experience with Apache Kafka for data streaming and event processing.

Proficiency in SQL for data querying and transformation.

Strong understanding of data governance, data security, and data quality frameworks.

Excellent communication skills and the ability to work in a cross-functional environ

Must have experience in Data Engineering domain .
Must have implemented at least 2 project end-to-end in Databricks.
Must have at least experience on databricks which consists of various components as below
o Delta lake
o dbConnect
o db API 2.0
o Databricks workflows orchestration
Must be well versed with Databricks Lakehouse concept and its implementation in enterprise environments.
Must have good understanding to create complex data pipeline
Must have good knowledge of Data structure & algorithms.
Must be strong in SQL and sprak-sql.
Must have strong performance optimization skills to improve efficiency and reduce cost.
Must have worked on both Batch and streaming data pipeline.
Must have extensive knowledge of Spark and Hive data processing framework.
Must have worked on any cloud (Azure, AWS, GCP) and most common services like ADLS/S3, ADF/Lambda, CosmosDB/DynamoDB, ASB/SQS, Cloud databases.
Must be strong in writing unit test case and integration test
Must have strong communication skills and have worked on the team of size 5 plus
Must have great attitude towards learning new skills and upskilling the existing skills.

Preferred Qualifications

Good to have Unity catalog and basic governance knowledge.
Good to have Databricks SQL Endpoint understanding.
Good To have CI/CD experience to build the pipeline for Databricks jobs.
Good to have if worked on migration project to build Unified data platform.
Good to have knowledge of DBT.
Good to have knowledge of docker and Kubernetes.

Why join Genpact?

Be a transformation leader Work at the cutting edge of AI, automation, and digital innovation

Make an impact Drive change for global enterprises and solve business challenges that matter

Accelerate your career Get hands-on experience, mentorship, and continuous learning opportunities

Work with the best Join 140,000+ bold thinkers and problem-solvers who push boundaries every day

Thrive in a values-driven culture Our courage, curiosity, and incisiveness - built on a foundation of integrity and inclusion - allow your ideas to fuel progress

Come join the tech shapers and growth makers at Genpact and take your career in the only direction that matters: Up.

Lets build tomorrow together.

About Company

Genpact is a global professional services firm that offers a wide range of digital transformation services and solutions. With a presence in over 30 countries, Genpact leverages its deep domain expertise in operations and analytics to help businesses transform their operations, improve efficiency, and enhance customer experience. The company combines digital technology, data science, and operational excellence to deliver business outcomes across various industries, including banking, insurance, manufacturing, and healthcare. Founded in 1997 as a subsidiary of GE, Genpact has grown into an independent, NYSE-listed company with a diverse workforce of over 90,000 employees globally.