EY - GDS Consulting - AI and DATA - AWS Data Engineer - Senior

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. 

AWS Senior Data Engineer

Experience Required: 

• Minimum 5+  years

Job Summary: 

We are seeking a skilled Data Engineer with a strong background in data ingestion, processing, and storage. The ideal candidate will have experience working with various data sources and technologies, particularly in a cloud environment. You will be responsible for designing and implementing data pipelines, ensuring data quality, and optimizing data storage solutions.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines for data ingestion and processing using Python, Spark, and AWS services.
• Work with on-prem Oracle databases, batch files, and Confluent Kafka for data sourcing.
• Implement and manage ETL processes using AWS Glue and EMR for batch and streaming data.
• Develop and maintain data storage solutions using Medallion Architecture in S3, Redshift, and Oracle.
• Collaborate with cross-functional teams to understand data requirements and deliver solutions that meet business needs.
• Monitor and optimize data workflows using Airflow and other orchestration tools.
• Ensure data quality and integrity throughout the data lifecycle.
• Implement CI/CD practices for data pipeline deployment using Terraform and other tools.
• Utilize monitoring and logging tools such as CloudWatch, Datadog, and Splunk to ensure system reliability and performance.
• Communicate effectively with stakeholders to gather requirements and provide updates on project status.

Technical Skills Required:

• Proficient in Python for data processing and automation.
• Strong experience with Apache Spark for large-scale data processing.
• Familiarity with AWS S3 for data storage and management.
• Experience with Kafka for real-time data streaming.
• Knowledge of Redshift for data warehousing solutions.
• Proficient in Oracle databases for data management.
• Experience with AWS Glue for ETL processes.
• Familiarity with Apache Airflow for workflow orchestration.
• Experience with EMR for big data processing.

Mandatory: 

• Strong AWS data engineering skills.

Good Additional Skills:

• Familiarity with Terraform for infrastructure as code.
• Experience with messaging services such as SNS and SQS.
• Knowledge of monitoring and logging tools like CloudWatch, Datadog, and Splunk.
• Experience with AWS DataSync, DMS, Athena, and Lake Formation.

Communication Skills:

• Excellent verbal and written communication skills are mandatory for effective collaboration with team members and stakeholders.

EY | Building a better working world 

 
_x000D_ EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.  

 
_x000D_ Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.  

 
_x000D_ Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.