Data Engineer

We are seeking a highly experienced Lead Data Engineer to lead the design, development, and delivery of scalable, reliable, and cost-efficient data platforms. The ideal candidate will possess deep expertise in modern data engineering technologies, including distributed data processing, ETL/ELT pipeline development, data modeling, and workflow orchestration, along with hands-on experience with the Databricks Lakehouse Platform and its medallion (Bronze/Silver/Gold) architecture.

This role also requires a solid understanding of Large Language Models (LLMs) and GenAI data ecosystems, enabling the development of high-quality datasets and retrieval-ready pipelines for AI-powered applications. As a technical leader, you will mentor engineering teams, establish best practices, and collaborate closely with architects, data scientists, and business stakeholders to deliver robust data solutions.

Key Responsibilities

Lead the architecture, development, and optimization of scalable data platforms supporting both batch and streaming workloads.

Design, implement, and maintain ETL/ELT pipelines for data ingestion, transformation, and curation using the Databricks Lakehouse Platform and medallion architecture.

Define data models, schemas, and storage strategies for data lakes and data warehouses to support analytics, reporting, machine learning, and GenAI initiatives.

Develop and curate high-quality datasets, feature engineering pipelines, and retrieval-ready data stores, including embeddings and vector-based data structures, for LLM-powered applications.

Establish engineering standards, coding best practices, code review processes, and CI/CD pipelines to ensure maintainable and reliable solutions.

Build and automate end-to-end data workflows using orchestration tools such as Apache Airflow or equivalent platforms.

Lead migrations from legacy on-premises or cloud-based data warehouses to modern cloud-native and lakehouse architectures.

Optimize performance and cost by implementing effective partitioning, caching, compute tuning, and distributed processing strategies.

Implement robust data governance, security, lineage, and access control frameworks aligned with organizational compliance requirements.

Build monitoring, logging, alerting, and data quality frameworks to ensure reliability and proactive issue resolution.

Mentor and guide data engineers while collaborating with architects, data scientists, and business stakeholders to translate business requirements into scalable technical solutions.

Participate in and lead Agile ceremonies, including sprint planning, stand-ups, retrospectives, and technical reviews.

Required Qualifications

10+ years of hands-on experience in data engineering, including leadership of enterprise-scale data platform initiatives.

Strong expertise with the Databricks Lakehouse Platform, including Delta Lake, Delta Live Tables, Databricks Workflows, and Unity Catalog.

Proven experience implementing the medallion (Bronze/Silver/Gold) architecture for enterprise data platforms.

Deep knowledge of distributed data processing using Apache Spark, including PySpark and Spark SQL.

Extensive experience building scalable ETL/ELT pipelines for both batch and streaming data processing.

Expert proficiency in Python and SQL for data engineering, transformation, validation, and pipeline development.

Strong experience designing and managing data lakes and data warehouses using dimensional and lakehouse modeling techniques.

Practical understanding of Large Language Models (LLMs) and GenAI concepts, including prompts, embeddings, vector databases, Retrieval-Augmented Generation (RAG), and supporting data pipelines.

Hands-on experience with at least one major cloud platform (AWS, Azure, or GCP) and its core data services.

Demonstrated success leading migrations from legacy platforms such as Hadoop or traditional data warehouses to modern cloud and lakehouse environments.

Strong expertise in distributed computing, data partitioning, and performance optimization techniques.

Experience implementing data security, governance, lineage, encryption, IAM, and metadata management.

Solid understanding of object-oriented programming principles, software design patterns, and CI/CD practices.

Experience working within Agile/Scrum environments and mentoring engineering teams.

Excellent analytical, problem-solving, stakeholder management, and communication skills.