Data Engineer - AWS

Position: Cloud Data Engineer- AWS+Big Data

Purpose of the Position: You will be a critical member of the InfoCepts Cloud Data Architect Team. This position requires a sound knowledge on Big Data using Cloud technologies, EMR, Athena, Spark-Scala, PySpark, S3, AWS Lambda, etc. Strong foundation on database concepts and SQL is also required.

Location: Nagpur/Pune/Chennai/Bangalore

Type of Employment: Full-time

Key Result Areas and Activities:

Study existing technology landscape and understand current data integration framework and

do impact assessment for the requirements

Develop/Enhance spark jobs using Scala for new/existing project requirements

Create new data pipelines for developed/enhanced spark jobs using AWS Lambda or Apache

Airflow

Analyze data processing, integration, modelling and reporting requirements & Define data

loading strategies considering volume, data types, frequency and analytics specifications

Manage build phase and quality assure code to ensure fulfilling requirements and adhering to

Cloud architecture. Resolve difficult design and develop issues

Essential Skills:

In depth knowledge of the following AWS Services required: S3, EC2, EMR, Athena, AWS Glue,

Lambda

Big Data - Hadoop, Hive, Spark, YARN

Programming Language Scala, Python, Shell Scripts

Operating System - Any flavor of Linux, Windows

Strong SQL skills

Expertise in developing ETL workflows comprising complex transformations like SCD,

deduplications, aggregations etc.

Should have conceptual understanding of AWS VPC, Subnets, Security Groups, & Route Tables

Desirable Skills:

Associate Level and Professional Level AWS Certification

Cloud Databases - AWS Aurora, Singlestore, RedShift, Snowflake

Orchestration Tools - Apache Airflow

Qualifications:

Bachelors degree in computer science, engineering, or related field (Masters degree is a plus)

Demonstrated continued learning through one or more technical certifications or related

methods

At least 5+ years of IT experience; minimum 3 years of experience on Cloud related projects

3+ years of AWS related project