BE/MTech in Computer Science or an equivalent professional experience
9+ years of design, architecture, and development experience, tackling complex problems in largescale data pipelines
Solid foundation in Data Structures, Algorithms, Object-Oriented Programming, and Software Design
Architectural expertise in data modeling for productiongrade batch and streaming processing systems
Deep understanding of Spark-based processing with focus on resource optimization
Practical understanding of Airflow for orchestration and Kafka for streaming
Solid foundation in distributed systems: consistency, reliability, fault tolerance, retries, circuit breakers, and timeouts
Production experience with CI/CD (e.g., GitHub Actions/Jenkins), containers (Docker), Kubernetes, and infrastructure-as-code (Helm/Terraform)
Hands-on experience integrating LLM calls in data pipelines: prompt orchestration, batching, rate limiting, guardrails, output validation
Exposure to embedding generation and vector indexing as part of data processing pipelines.
Programming experience in Python (Spark). Strong SQL and exposure to at least one cloud
Develop batch and streaming ETL/ELT pipelines across APIs, databases, files, and event streams
Use SQL and optimized Spark pipelines to transform raw data into clean, standardized, query-ready datasets
Build reusable data marts and feature sets for downstream teams (analytics, ML, product)
Tune queries, partitioning, clustering, indexing, and storage formats (Parquet/ORC)
Optimize compute and storage costs; manage scaling strategies and right-size resources
Implement CI/CD for data code and pipelines; manage environments and releases
Translate business needs into technical specifications; document datasets, SLAs, and usage guidelines
Support incident response and root-cause analysis for data quality issues
Partner with analytics, ML, engineering, and product teams to define data requirements
Mentor junior engineers and contribute to engineering best practices
Drive architectural decisions and influence long-term data strategy

Lead Big Data Engineer

Talentiser

Job Description

Services you might be interested in

Improve Your Resume Today