Data Engineer

Company Description

AutoFlow Technologies (AutoFlowTech) is a leading provider of process automation solutions, catering to industries such as manufacturing, aviation, automotive, and industrial processes. Our team of experts designs open, scalable, and modular solutions to help organizations improve efficiency and drive innovation. Serving a diverse customer base, from SMEs to Fortune 500 companies, AutoFlowTech focuses on fostering growth and long-term success. Based on a foundation of problem-solving expertise, we specialize in addressing complex challenges for our clients.

Experience:

2+ years full-time experience in Data engineering and AWS Cloud Technologies

Mandatory experience in Mid to Expert Level programming capabilities in a large-scale enterprise

Experience with Distributed Versioning Control environments such as GIT

Familiarity with development tools - experience on either IntelliJ / Eclipse / VSCode IDE, Build Tool Maven

Experience on AWS cloud including DMS, Glue , Pyspark

Responsibilities

Responsible to Ingest data from files, streams and databases. Process the data with Hadoop, Scala, SQL Database, Spark, ML, IoT

Develop programs in Python & Pyspark and AQLas part of data cleaning and processing

Responsible to design and develop distributed, high volume, high velocity multi-threaded Ingestion pipeline

Provide high operational excellence guaranteeing high availability and platform stability

Implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies Pyspark, Glue, and AWS Cloud computing etc.

Anticipate and resolve complex problems that impact data design

Assist in critical technical and design decisions during the development process

Key Performance indicators

Ensure Data Pipeline is robust and is monitored with respect to schemas, distributions and data completeness

Evaluate the time required for data to be available for analysis to baseline by monitoring the data velocity against baseline

Maintain and Monitor the Query to report conversion rate

Maintain and monitor the optimal time needed for data to be reported in form of KPIs

Identify issues in codes or bugs that have entered the production stage

Drive performance optimization by developing optimized codes (Pipelines) for efficient resource utilization

Qualifications:

Bachelor's degree in Computer Science, Software Engineering or a related technical discipline

Knowledge of any of the programming languages: Python, Scala or Java

Primary Skill:AWS , AWS Glue , AWS DMS , Cloudwatch , Pyspark , SQL

Domain Knowledge: Life Insurance domain is preferred. Familiarity with Finance data and subject is highly preferred