Lead Data Engineer / Platform Engineer

Lead Data Engineer / Platform Engineer

Experience: 57 Years

Type: Full-Time, On-site

We are looking for a hands-on, high-ownership Lead Data / Platform Engineer who thrives in fast-moving startup environments. This is a build-from-zero, scale-to-thousands, own-the-outcome role.

You will architect, build, and scale a modern lakehouse and data platform stack powered by Spark, Trino, Kubernetes, and open-source technologies. You will lead from the frontwriting production code, designing architecture, mentoring engineers, and driving delivery.

We need someone who can take ideas from concept to production, drive architecture decisions, mentor engineers, and ensure delivery of high-quality software in a fast-paced product environment.

Self-starter who thrives in ambiguity

Highly hands-on (coding majority of the time)

Comfortable making technical decisions

Accountable for delivery timelines and production stability

Experienced in leading and mentoring engineers

Lead architecture and implementation of distributed data platforms

Design and build high-performance pipelines using Spark, PySpark, and Spark SQL

Build and optimize federated query systems using Trino

Design scalable orchestration workflows using Apache Airflow

Implement strong data governance frameworks (catalog, lineage, data contracts, data quality)

Own Kubernetes-based OSS stack deployment using Terraform and Helm

Architect ML data pipelines for feature engineering and model training

Drive SSO integrations (OAuth2 / OIDC) across platform components

Mentor engineers and enforce engineering excellence standards

Establish best practices for reliability, performance, and cost optimization

Strong hands-on experience with Apache Spark, PySpark, and Spark SQL

Experience working with TB-scale datasets

Query engine expertise with Trino

Strong Python programming skills

Deep understanding of distributed systems and data modeling

Production experience with Apache Airflow

Designing reliable DAGs with observability

CI/CD for data pipelines

Experience with data catalog tools (DataHub/Amundsen or equivalent)

Data contracts and schema enforcement

Data quality frameworks and lineage tracking

Experience supporting ML pipelines

Feature engineering and model lifecycle exposure

Understanding of MLflow or equivalent tools (preferred)

Kubernetes production deployments

Infrastructure as Code using Terraform

Managing OSS stacks via Helm charts

Performance tuning and cluster management

Experience implementing SSO (OAuth2 / OIDC)

Understanding of RBAC and multi-tenant security architectures

Experience building Agentic AI systems and multi-agent workflows

Hands-on experience with LangChain and LangGraph frameworks

Strong understanding of Prompt Engineering techniques (system prompts, tool use, RAG patterns)

Experience building Text-to-SQL, code generation, or AI assistant workflows

Knowledge of Retrieval-Augmented Generation (RAG) architectures

Experience integrating LLM APIs (OpenAI/Anthropic or equivalent)

Understanding of model evaluation, guardrails, and AI observability

Experience deploying AI/ML services on Kubernetes (KServe or equivalent preferred)

Lead and mentor a team of engineers

Drive delivery with clear milestones

Make scalable, long-term architectural decisions

Communicate effectively with product and business stakeholders

Raise the bar for engineering quality