Senior Engineer - BigData

Required Technical Skills

Core Technologies

• Java (8+)
• Strong OOP fundamentals, collections, multithreading, and JVM concepts
• Apache Kafka
• Producers, consumers, consumer groups
• Partitioning, offset management
• Delivery semantics (at-least-once, exactly-once conceptual)
• Apache Spark
• Spark SQL, DataFrames/Datasets
• Structured Streaming
• Performance tuning basics

Containerization & Orchestration

• OnPrem Kubernetes
• Deploying and managing applications using Deployments, Services, ConfigMaps, and Secrets
• Understanding of pods, namespaces, scaling, and rolling updates
• Basic troubleshooting of Kubernetes workloads
• AWS ECS
• Experience with ECS task definitions and services
• Working knowledge of ECS on EC2
• Integration with ALB, IAM roles, and CloudWatch
• Docker
• Creating Dockerfiles
• Containerizing Java and Spark applications

Cloud (AWS Basic to Intermediate)

Hands-on experience with core AWS services:

• Amazon S3 data storage / data lake
• Amazon EC2 compute
• AWS ECS container orchestration
• IAM roles, policies, and permissions

Infrastructure as Code & DevOps (Good to Have)

• Terraform
• Writing and maintaining Terraform configurations and modules
• Managing environments (dev, test, prod)
• State management fundamentals
• CI/CD exposure using tools such as Jenkins, GitHub Actions, GitLab CI, or AWS CodePipeline
• Understanding of deployment strategies (rolling, blue-green)

Preferred / Good-to-Have Skills

• Kafka Streams experience
• Spark performance tuning at scale
• Experience with EKS or migrating workloads from onprem Kubernetes to cloud
• Knowledge of service mesh or ingress controllers
• Exposure to observability tools (Prometheus, Grafana, ELK, etc.)
• Understanding of security best practices in Kubernetes and AWS