PySpark Developer

Position: PySpark Developer Location: Hyderabad Employment Type: Full-Time

Job Summary: We are seeking an experienced PySpark Developer to join our team for a migration project. The primary responsibility will be to migrate Teradata tables and the Data Warehouse to Spark 3.x and Delta Lake. The ideal candidate will have extensive experience with PySpark, Teradata, Control M and Delta Lake, and a strong understanding of data processing and big data technologies. Key Responsibilities: Analyze existing Teradata tables and Data Warehouse structures. Design and implement a strategy to migrate data from Teradata to Spark 3.x and Delta Lake. Develop PySpark scripts to extract, transform, and load (ETL) data from Teradata to Delta Lake. Optimize and refactor existing SQL queries and ETL processes for performance in Spark. Collaborate with data engineers, database administrators, and business stakeholders to ensure data accuracy and integrity. Perform data validation and reconciliation to ensure the integrity of migrated data. Monitor and troubleshoot data migration processes to resolve any issues that arise. Document the migration process, including any changes made to data structures and ETL processes. Stay updated with the latest features and best practices in Spark, Delta Lake, and big data technologies. Design, develop, and implement job scheduling and automation processes using Control-M. Monitor and troubleshoot job schedules and automation workflows to ensure smooth operations. Perform regular maintenance and updates to Control-M job schedules and shell scripts. Stay updated with the latest features and best practices in Control-M and shell scripting.