Technical Lead-Data Engg

Area(s) of responsibility

JD - AWS Data Platform Framework Development Engineer

Key Responsibilities
-
Design and develop reusable and scalable data processing frameworks and libraries for data ingestion, processing, and ETL pipelines on AWS alongside the platform development team.
-
Collaborate closely with framework developers, data engineers, architects, and analysts to standardize data pipelines and processing patterns.
-
Develop and enhance Debezium Kafka CDC pipeline frameworks to enable rapid instantiation of CDC data ingestion workflows.
-
Build and maintain AWS Glue PySpark job frameworks aligned with medallion architecture principles.
-
Implement and maintain ETL frameworks for loading data into Snowflake.
-
Develop Infrastructure as Code using Terraform and Github to automate provisioning and deployment of platform components.
-
Ensure platform reliability, scalability, and observability.
-
Contribute to improving development standards, code reviews, and best practices focused on framework and platform engineering.
Required Skills & Experience
-
Masters degree in software engineering, computer science or equivalent
-
AWS certifications (Solutions Architect Associate, Developer Associate, Data Engineer Associate).
-
Strong software engineering background with expertise in Python, especially PySpark.
-
Experience with and thorough understanding of Kafka and Kafka Connect concepts.
-
Proven track record developing reusable frameworks or libraries focusing on scalability and maintainability.
-
Sound understanding and practical application of OOP and SOLID principles (encapsulation, inheritance, polymorphism, abstraction).
-
Hands-on experience with AWS services including Glue, ECS, S3, Kafka (including Debezium), and Snowflake.
-
Experience building and orchestrating data pipelines using Airflow or similar tools.
-
Proficient in Infrastructure as Code using Terraform.
-
Familiarity with CI/CD workflows using GitHub or similar platforms.
-
Strong problem-solving skills and ability to write clean, modular, and well-documented code.
-
Excellent communication skills and ability to work collaboratively in an international team of highly skilled IT engineers.
Preferred Qualifications
-
Experience with Iceberg or other open table formats in a data lakehouse environment.
-
Prior experience working on CDC (Change Data Capture) pipelines or Kafka streaming frameworks.
-
Experience with big data processing frameworks is considered a plus.
-
Understanding of medallion architecture and data lakehouse design patterns.
-
Multiple years of application development experience with an OOP native programming language like Java, C++, or C#.
-
Experience with open table formats and big data is considered a plus.