Data Engineer

Experience: 5-8

Location: Pune, Thiruvantnampuram, Hyderabad

Employment Type: Full-Time

Mandatory Skills: Data Engineer, Kafka, SQL, AWS, and Java

Job Summary:

We are seeking a skilled Data Engineer with strong experience in Kafka, SQL, AWS, and Java to design, develop, and maintain scalable data pipelines and streaming solutions. The ideal candidate will have hands-on experience in building real-time and batch data processing systems and working with cloud-based data platforms.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Java and Kafka.
• Develop and optimize complex SQL queries for data extraction, transformation, and loading (ETL).
• Build and manage real-time streaming solutions using Apache Kafka.
• Work with AWS services such as S3, EC2, Lambda, Glue, and Redshift for data storage and processing.
• Implement data quality checks and ensure data integrity across systems.
• Monitor and troubleshoot data pipelines and streaming workflows.
• Collaborate with data analysts, data scientists, and software engineers to support data needs.
• Optimize system performance and ensure high availability of data systems.

Required Skills:

• Strong programming experience in Java.
• Hands-on experience with Apache Kafka (producers, consumers, topics, partitions).
• Advanced knowledge of SQL and relational databases.
• Experience working with AWS cloud services.
• Good understanding of ETL processes and data pipeline architecture.
• Knowledge of data modeling and performance tuning.

Preferred Skills (Good to Have):

• Experience with Spark or PySpark.
• Familiarity with Docker and Kubernetes.
• Experience with CI/CD pipelines.
• Knowledge of NoSQL databases (MongoDB, Cassandra).
• Experience with data warehousing solutions.