Lead Data Engineer

Join JPMorgan Chase and become an integral part of the ICB Chase Data Platform team, where innovation and excellence converge. We are on the lookout for a talented Data Engineer to enhance our cutting-edge data platform, utilizing state-of-the-art cloud infrastructure to propel the firm's business objectives forward.

As a Software Engineer III in Data Engineering, you will be at the forefront of architecting, building, deploying, and maintaining a cloud-native, web-scale data platform. Collaborate with a dynamic team to deliver robust, scalable, and sustainable data pipelines, ensuring seamless integration and optimal performance.

Key Responsibilities:

• Architect and develop scalable data pipelines on cloud infrastructure.
• Collaborate in an agile, customer-facing environment to deliver high-quality solutions.
• Utilize your expertise in Python and SQL to develop and optimize data processes.
• Design and manage relational databases and complex data structures.
• Deploy and manage containerized applications using Docker and Kubernetes.
• Develop Infrastructure as Code (IaC) with Terraform and Terragrunt.
• Engage in all stages of the software development lifecycle, from design to support.
• Utilize scheduling systems, with a preference for Airflow, to manage workflows.
• Automate deployment, releases, and testing in CI/CD pipelines.
• Write and automate unit, component, integration, and end-to-end tests.

Required Qualifications, Capabilities, and Skills:

• Extensive hands-on experience in Python and SQL (any dialect).
• Experience with distributed systems and cloud technologies (AWS, GCP, Azure, etc.); EMR is a plus.
• Hands-on experience in relational databases, data structures, caching concepts, race conditions, and complex analytical queries.
• Experience with Infrastructure as Code (IaC) using Terraform and Terragrunt.
• Experience with scheduling systems, with Airflow preferred.
• Basic understanding of data streaming and messaging frameworks (Kafka, Spark Structured Streaming, Flink, etc.).
• Good knowledge of the Spark framework and its deployment with cloud services.

Preferred Qualifications, Capabilities and Skills:

• Academic qualification in computer science or a STEM-related field, or foreign equivalent.
• At least 3 years of hands-on experience as a back-end software engineer or data engineer.
• Familiarity with table formats such as Iceberg.
• Experience with EMR and Spark is a plus.
• Understanding of RESTful services.