Job Summary
We are seeking a highly skilled Technical Lead-CDS with 8 to 12 years of experience to join our dynamic team. The ideal candidate will have extensive experience in PySpark Python Spark Streaming Spark Pyspark SparkSQL Spark Optimization Spark Shell scripting Impala and Hive. This role is hybrid with day shifts and does not require travel. The candidate will play a crucial role in driving technical excellence and innovation within the company.
Responsibilities
Lead the design and implementation of complex data processing systems using Spark PySpark and related technologies.Oversee the development and optimization of SparkSQL queries to ensure high performance and scalability.Provide technical guidance and mentorship to junior team members in Shell scripting Impala Hive and Python.Collaborate with cross-functional teams to define and implement data processing workflows and pipelines.Ensure the reliability and efficiency of Spark Streaming applications for real-time data processing.Develop and maintain automated scripts using Shell scripting to streamline data processing tasks.Conduct code reviews and ensure adherence to best practices and coding standards.Optimize Spark jobs for performance and resource utilization to meet business requirements.Troubleshoot and resolve complex technical issues related to data processing and analytics.Implement data quality checks and validation processes to ensure data accuracy and integrity.Stay updated with the latest advancements in Spark and related technologies to drive continuous improvement.Participate in the planning and execution of data migration and integration projects.Collaborate with stakeholders to gather requirements and translate them into technical specifications.
Qualifications
Possess a strong background in Shell scripting Impala Hive and Python.Demonstrate expertise in Spark Streaming Spark Pyspark SparkSQL and Spark Optimization.Have experience in developing and optimizing data processing workflows and pipelines.Show proficiency in troubleshooting and resolving technical issues related to data processing.Exhibit strong problem-solving skills and attention to detail.Have excellent communication and collaboration skills.Be able to mentor and guide junior team members effectively.Stay updated with the latest industry trends and technologies.Be capable of working in a hybrid work model with day shifts.Have a minimum of 8 years and a maximum of 12 years of relevant experience.
Certifications Required
Certified Spark Developer Python Certification Big Data Certification
