Big Data Engineer

Software Engineer Big Data with 4+ yrs to 15 Years of experience

4+ years of professional experience in Big data platform. Responsible for producing excellent quality of code, adhering to expected coding standards and industry best practices.

Key Responsibilities:

• Design, develop and optimize big data processing pipelines using Spark/Scala and Hive (HQL).
• Implement and manage data ingestion workflows using Apache NiFi.
• Develop data streaming solutions leveraging Apache Kafka.
• Monitor and analyse logs and metrics using Kibana, Elasticsearch, and Logstash.
• Collaborate with DevOps teams to implement Continuous Integration and Continuous Deployment (CI/CD) pipelines using Git, Jenkins, and Nexus.
• Ensure data quality, consistency, and governance across big data platforms.
• Participate in architecture discussions and recommend best practices for data processing and pipeline design.
• Troubleshoot and resolve complex data-related issues across the big data stack.
• Mentor junior engineers and contribute to team knowledge sharing.
• Stay updated with emerging big data technologies and recommend adoption as appropriate
• The candidate should have been in client facing roles and interacted with customers in requirement gathering workshops.
• Experience working in Agile development environments and following Agile methodologies.
• Should have completed at least 2 implementation projects on the above-mentioned mandatory skills and should be an independent contributor.

Skill Set:

• SPARK/SCALA
• HQL
• Shell Scripting / Python
• Postgresql
• Scheduling Oozie and Control M
• CD/CI concepts & Technologies
• Big Data Ecosystem
• Git & Jenkins, Nexus, Ansible, Maven, Sonar
• NIFI, Kafka Good to have
• Kibana, Elastic Search Log stash Good to have