Senior Data Engineer_AVP-I

JOB DESCRIPTION

1. The Senior Data Engineer will oversee the department's data integration work, including developing a data model, maintaining a data warehouse and analytics environment and writing scripts for data integration and analysis
2. This role will work closely and collaboratively with members of Data Science & Decision Management and IT teams to define requirements, mine and analyze data, integrate data from a variety of sources, and deploy high quality data pipelines in support of the analytics needs of the Bank
3. They will also create and oversee an automated reporting system, data visualization applications and manage other internal systems / process
4. Deliver end to end, the projects in the data and MIS domain. He/she will collaborate with senior business leaders to determine and prioritize immediate requirements and anticipate future data pipelines and architecture, MIS and critical ad-hoc requests to ensure high quality delivery
5. Role not only covers the complete data value chain but also to provides tools, services and data analytics to enable new use cases, business capabilities and directly creates value for Bank

KEY RESPONSIBILITIES OF THE ROLE

• Plan and build efficient data platforms, systems and infrastructure for intelligent engines which empower business teams
• Lead a team of data engineers to design, develop, and maintain data systems & pipelines independently to enable faster business analysis
• Assess common patterns and problems to create standardized solutions by centrally building services and technology frameworks for MIS & analytics
• Build algorithms and develop optimal queries to combine raw information from different sources
• Partner with service / backend engineers to integrate data provided by legacy IT solutions into our designed databases and make it accessible to the internal verticals consuming those data.
• Focus on the design and setup of databases, data models, data transformations (ETL), critical online banking business processes
• Contribute to data harmonization as well as data cleansing
• Conduct complex data analysis to ensure data quality & reliability
• Solve complex SQL and Big Data performance challenges
• Utilize Python / PySpark / Scala or Java to analyze large-scale datasets in batch and real-time
• Build reports and data visualizations, using data from the data warehouse and other sources
• Produce scalable, replicable code and engineering solutions that help automate repetitive data management tasks
• Define and create project plans for BI delivery
• Evaluate and balance trade-offs between project size and complexity, cost, urgency, risk, and stakeholder value
• Coordinate the project resources to ensure that projects are delivered on time and within budget
• Partner with business partners to elicit, analyse, and document business requirements and translate business requirements into BI requirements
• Work with the data scientists to understand the data requirements and create re-usable data assets to enable data scientists to build and deploy machine learning models faster
• Partner with business partners to conduct user acceptance testing
• Based on engagement, conduct meetings with vendor for technology evaluation, cost negotiations, assess vendor capability & project progress reviews
• Manage on-site staff of vendor

MINIMUM REQUIREMENTS OF KNOWLEDGE & SKILLS

Educational

Qualifications

• Degree in Computer Science, Software Engineering, Information Systems, Mathematics or other STEM majors
• Strong professional experience and expert knowledge of database systems and database structures (structured and unstructured data)
• Broad knowledge on mathematical models / statistics
• Knowledge on design principles of ETL processes in the DWH environment / data lake and how to automate them
• Professional software development experience (one or more) with Scala, Spark, Hadoop, Java, Linux, SQL other BI related languages like Python
• Experience in the big data ecosystem, with Hadoop (Pig, Hive, HDFS), Apache Spark and NoSQL/SQL databases
• Experience using ETL big data pipelines (Apache Airflow)

Experience(Years and Core Experience Type)

• 8+ years of experience in processing of the large amount of data, big data (Hadoop, Hive, Sqoop & big data technologies)
• Hands-on experience with Python/Pyspark/Scala and basic libraries for machine learning is required
• Experience with containerization and related technologies (e.g. Docker, Kubernetes)
• Excellent analytical and conceptual thinking and strategic planning skills
• Excellent communication skills, strong independence and initiative, ability to work in agile delivery teams
• Should have knowledge of data profiles, data quality and data lineage
• Be able to perform data analysis and trend analysis and working knowledge of ETL tools, Databases and experience in writing complex queries are required
• Resolving data issues related to different customer journeys and be able to stitch the data across systems including on-premises and cloud solutions
• Proficient in using databases like Teradata, Oracle, SQL Server, Neo4J etc.
• Experience in designing & development of ETL processes using ETL tools / Python or Scala
• Exposure and working knowledge of cloud computing on AWS/Azure/Google Cloud
• Working knowledge of dashboards & reporting platforms like Tableau, Qlik or industry standard reporting platforms
• Use APIs to push and pull data from various data systems and platforms

Certifications

• NA

Functional Skills

• Must have strong banking experience
• Be proficient in evaluating business requirements and converting them into BI requirements
• Strong track record of managing and leading medium to large teams
• Provide functional and technical leadership to ensure execution of BI projects as per the agreed timelines and with accuracy and quality from the teams
• Help build the right data architecture, develop data marts, warehouses and building reporting framework and end user information delivery applications
• Manage projects/sprints and small working groups towards delivery of MIS and data pipelines