Data Engineer

Data Engineer Data Platform Engineering

We are looking for a highly skilled Data Engineer to design, build, and scale a modern cloud-native Data Platform supporting batch, streaming, analytics, and real-time data use cases across the organization.

The ideal candidate should have strong expertise in distributed data systems, platform engineering, cloud infrastructure, and building reliable, scalable, and observable data ecosystems.

Key Responsibilities

• Design and develop scalable enterprise data platform architecture on AWS
• Build reusable frameworks for data ingestion, transformation, orchestration, and consumption
• Develop batch and real-time data pipelines using distributed processing technologies
• Build and maintain streaming platforms for event-driven architectures
• Create standardized data lake/lakehouse foundations for analytics and downstream applications
• Implement metadata management, schema evolution, lineage, governance, and access control
• Develop platform capabilities for data reliability, observability, monitoring, and alerting
• Enable self-service data access and scalable data consumption layers
• Build CDC and incremental ingestion frameworks from databases and external systems
• Optimize platform scalability, performance, fault tolerance, and cloud cost
• Automate deployments, infrastructure provisioning, and CI/CD for data workloads
• Collaborate with Analytics, Product, and Engineering teams to onboard new data domains
• Ensure platform security, compliance, and operational excellence

Technical Skills

Core Technologies

• Strong expertise in:
• Apache Spark
• Spark Streaming
• Apache Flink
• SQL
• Python
• Distributed data processing systems

Streaming & Event Platforms

• Hands-on experience with:
• Apache Kafka
• Kafka Connect / Kafka Streams
• Real-time event-driven architectures

AWS Data Platform

• Strong experience with:
• Amazon Web Services
• AWS Glue
• AWS Athena
• AWS DMS
• Amazon S3
• Amazon CloudWatch
• Understanding of IAM, networking, security, and cloud-native architecture

Data Platform Experience

• Experience building:
• Data lakes / lakehouse platforms
• Batch and streaming ingestion frameworks
• Metadata-driven ETL systems
• Data catalog and lineage systems
• Multi-tenant data platforms
• Data reliability and observability frameworks
• Data quality validation pipelines
• Scalable consumption layers for analytics and APIs
• Strong understanding of:
• Data partitioning and file optimization
• CDC architecture and incremental processing
• Schema registry and schema evolution
• High availability and fault-tolerant systems
• Data governance and access management

Good to Have

• Experience with:
• Apache Airflow
• dbt
• Delta Lake
• Apache Iceberg
• Docker / Kubernetes
• Terraform / Infrastructure as Code
• CI/CD pipelines and DevOps practices

Functional Expectations

• Strong system design and architecture skills
• Ability to build reusable and scalable platform components
• Strong debugging and production support capabilities
• Excellent communication and stakeholder collaboration
• Ownership mindset toward platform stability, scalability, and reliability
• Ability to drive platform modernization initiatives independently

Experience

• 3+ years of experience in Data Engineering / Data Platform Engineering
• Experience building enterprise-scale cloud data platforms preferred
• Prior experience with high-volume real-time systems is a plus