Senior Data Engineer

As a Senior Data Engineer, you will be responsible for:

• Collaborating with Technical Product Managers, Architects and Analytics team to define the scope, capabilities and roadmap of Data Applications.

• Delivering End-to-end ownership of business features, meeting functional & non-functional requirements.

• Owning the design and development of ETL data pipelines, orchestration and process maturity for data lakehouse using medallion architecture.

• Researching and evaluating new open-source technologies and framework to solve problems & improve existing solutions.

• Integrating domain data knowledge into development of data requirements.

• Articulating and documenting best practices and design principles to mentor other developers.

• Looking across multiple systems, understands the purpose of each system and defines data requirements by systems.

• Identifying downstream implications of data loads/migration (e.g., data quality, regulatory, etc.).

Required Qualifications

• Bachelor's degree in computer science or “STEM” Majors (Science, Technology, Engineering and Math). A minimum 7+ years of professional experience.

Desired Characteristics

Technical Expertise: 

•  Must have hands-on expertise in designing cost effective Big data ETL pipelines and orchestration, preferred streaming or near real time data flow.

• Have Proficiency in Lakehouse architecture and distributed data processing aka Big data engineering skills.

• Have hands-on project experience on Databricks development, troubleshooting and optimization using Pyspark.

• Have Good understanding of Apache Spark and Delta lake core concepts.

• Have Working knowledge of SQL/NoSql data model design.

• Have Working knowledge of Git repo, Git Branching strategy and CICD.

• Have Understanding of Big Data technologies core concepts – Airflow/Oozie, Spark, BigQuery, Hive, NoSql, Object Storage, etc.

• Have Ability to leverage data assets to respond to complex questions that require timely answers.

• Able to conduct exploratory data analysis and generates visual summaries of data. Identifies data quality issues.

• Can Actively identifies needs for novel methods/tools and works with team to invent as necessary. Validates new tools and methodologies.