Data Engineer (Big Data and ETL Ops) for our banking client in Chennai

Key Roles & Responsibilities:

• System Monitoring: Continuously monitor ETL jobs, workflows, and data pipelines for performance and reliability.
• Job Scheduling: Ensure jobs are scheduled appropriately and are running as per the defined schedules.
• Issue Resolution: Troubleshoot and resolve issues related to data integration, ETL jobs, and workflows promptly.
• Root Cause Analysis: Perform root cause analysis for recurring issues and implement preventive measures.
• Tuning: Optimize ETL jobs and workflows for better performance and efficiency.
• Resource Management: Ensure optimal use of system resources like CPU, memory, and storage.
• Data Validation: Ensure data accuracy, completeness, and consistency across different systems.
• Error Handling: Implement robust error handling and recovery mechanisms in ETL processes.
• Data Reconciliation: Perform data reconciliation to verify that the data loaded into the target systems matches the source data.
• Code Deployment: Deploy ETL code and configurations to production environments following change management procedures.
• Automation: Develop scripts and automation tools to streamline routine tasks and processes.
• Cross-Functional Teams: Work closely with development teams, DBAs, system administrators, and business users to resolve issues and implement changes.
• Status Reporting: Provide regular updates and reports on system status, performance, and incidents to stakeholders.
• Documentation: Maintain up-to-date documentation for ETL jobs, workflows, processes, and standard operating procedures.

Technical Requirements:

• Advanced working skills and knowledge with both RDBMS and SQL, with established command and working with a variety of databases (both SQL and NoSQL).
• Experience building and optimizing big data data pipelines architectures and data sets; both batch-oriented and real time.
• Extensive experience working with Big Data tools and building data solutions for advanced analytics eg Cloudera, Big query, Data proc etc.
• Practical knowledge across ETL tools (Informatica, Talend) as well as more recent big data tools,
• Experience with the following tools and technologies:
• Big data stack, including Hadoop, Spark, Kafka, Informatica
• Relational SQL and NoSQL databases
• Data pipeline/workflow management and quality tools.
• Real-time data injection
• Object-oriented, procedural and functional programming languages such as Python, Java, C++, Scala, etc

Qualifications

• Education: Bachelor's degree in Computer Science or equivalent; Masters preferred
• Experience: 3-5 years of experience in a similar Data Engineer role, Data Platforms and Design.
• Ability to work and communicate collectively with business.
• Excellent analytical and problem solving, conflict resolution, communication and Interpersonal skills.