Senior Data Engineer

Role Overview

We are looking for a highly skilled Senior Platform Engineer of 7+years of experience to design and implement a next-generation observability and reliability platform for distributed data pipelines.

This platform will automatically discover lineage, collect metrics, and aggregate logs across the technology stack, enabling proactive monitoring and rapid troubleshooting of failures. You will work closely with reliability and data engineering teams to build intelligent systems that enhance operational efficiency and resilience.

Key Responsibilities

Design and develop end-to-end observability solutions for distributed data pipelines

Build systems that automatically capture lineage, metrics, and logs across heterogeneous data platforms

Develop MCP proxies, orchestrators, and scalable data pipelines using Python

Implement alerting and monitoring frameworks to detect and resolve failures proactively

Integrate with modern observability stacks including metrics, logs, and tracing systems

Build and optimize REST/GraphQL APIs with async patterns and robust session management

Collaborate with reliability engineers to enable faster root cause analysis and troubleshooting

Develop intuitive UI dashboards (React or equivalent) for monitoring and insights

Work with cloud-native architectures, primarily on AWS

Required Skills & Experience

Strong experience in Python & PySpark for data engineering (Must)

Strong experience with Databricks platform for data processing (Must)

Strong expertise in Python for building scalable backend systems, orchestrators, and pipelines

Experience with REST and/or GraphQL APIs, async programming, and session handling

Hands-on experience with cloud platforms (AWS preferred)

Deep understanding of distributed systems, data pipelines, and observability concepts

Experience with data quality, anomaly detection, or validation systems

Hands on experience in:

o LLM orchestration frameworks

o RAG (Retrieval-Augmented Generation)

o MCP (Model Context Protocol)

o AWS Bedrock or equivalent AI platforms

o Experience with AWS Nova Pro

Frontend experience with React JS or similar frameworks

Good to Have

Experience building AI-powered observability or AIOps platforms

Exposure to data lineage tools and metadata management systems

Familiarity with Kubernetes and containerized deployments

Knowledge of CI/CD pipelines and DevOps practices

What Youll Build

A unified platform that provides:

o Automatic data lineage discovery

o Centralized logging, metrics, and tracing

o Intelligent alerting for failures

o AI-assisted troubleshooting workflows

Why Join Us

Opportunity to build a cutting-edge observability platform from scratch

Work at the intersection of Data Engineering, AI/ML, and Cloud

High ownership and impact in a fast-moving, innovation-driven environment