Data Engineer

Job Description

Key Accountabilities

•  Working with colleagues to understand and implement requirements
•  Securing the data supply chain, understanding how data is ingested from different sources and combined / transformed into a single data set.
•  Understanding how to analyse, cleanse, join and transform data.
•  Implementing designed / specified solutions into the chosen platform
•  Working with colleagues to ensure that the on-prem/cloud infrastructure available is capable of meeting the solution requirements.
•  Planning, designing and conducting tests of the implementations, correcting errors and retesting to achieve an acceptable result.
•  Appreciate how to manage the data including; security, archiving, structure and storage.

Key Experience and Qualifications

•  Degree level education in Mathematics, Scientific, Computing or Engineering discipline or equivalent experience
•  6+ years of experience at various levels of Data Engineering roles including 3 years in technical lead role managing end to end solution.
•  Experience in designing solutions using databases and data storage technology using RDBMS (MS SQL Server)
•  Experience building and optimizing Big Data data pipelines, architectures and data sets using MS Azure data management and processing components through IaaS/PaaS/SaaS implementation models implemented through custom
•  Proficiency in Python scripting. Experienced in using different python modules used for data munging
•  Be up to date with data processing technology / platforms such as Spark (Databricks)
•  Experience in Data Modelling for optimizing the solution performance
•  Experienced in Azure DevOps
•  Experience with ETL and/or data integration tool  MuleSoft
•  Good understanding of infrastructure components and their fit in different types of data solutions
•  Experience of designing solutions deployed on Microsoft and Linux operating systems
•  Experience of working in an agile environment, within a self-organising team.

Job Category

Posting Date