Senior Data Engineer – PySpark, Cloud & Kafka - 5+ YoE - Immediate Joiner - Any UST Location

Candidates ready to join immediately can share their details via email for quick processing.

CCTC | ECTC | Notice Period | Location Preference

Act fast for immediate attention!

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark
• Build and manage batch and real-time data processing systems
• Develop and integrate Kafka-based streaming solutions
• Optimize Spark jobs for performance, cost, and scalability
• Work with cloud-native services to deploy and manage data solutions
• Ensure data quality, reliability, and security across platforms
• Collaborate with data scientists, analysts, and application teams
• Participate in code reviews, design discussions, and production support

Must-Have Skills

• Strong hands-on experience with PySpark / Apache Spark
• Solid understanding of distributed data processing concepts
• Experience with Apache Kafka (producers, consumers, topics, partitions)
• Hands-on experience with any one cloud platform :
• AWS (S3, EMR, Glue, EC2, IAM) or
• Azure (ADLS, Synapse, Databricks) or
• GCP (GCS, Dataproc, BigQuery)
• Proficiency in Python
• Strong experience with SQL and data modeling
• Experience working with large-scale datasets
• Familiarity with Linux/Unix environments
• Understanding of ETL/ELT frameworks
• Experience with CI/CD pipelines for data applications

Good-to-Have Skills

• Experience with Spark Structured Streaming
• Knowledge of Kafka Connect and Kafka Streams
• Exposure to Databricks
• Experience with NoSQL databases (Cassandra, MongoDB, HBase)
• Familiarity with workflow orchestration tools (Airflow, Oozie)
• Knowledge of containerization (Docker, Kubernetes)
• Experience with data lake architectures
• Understanding of security, governance, and compliance in cloud environments
• Exposure to Scala or Java is a plus
• Prior experience in Agile/Scrum environments