PySpark Developer

Key Responsibilities:

• Design, develop, and maintain data processing pipelines using PySpark
• Implement ETL workflows to ingest, transform, and load large datasets
• Optimize Spark jobs for performance, scalability, and reliability
• Work with structured and unstructured data from multiple sources (HDFS, S3, RDBMS, APIs, streaming sources)
• Collaborate with data engineers, analysts, and data scientists
• Ensure data quality, validation, and consistency
• Perform debugging, profiling, and tuning of Spark applications
• Write clean, reusable, and well-documented code
• Support production deployments and troubleshooting
• Follow best practices in data engineering, security, and governance

Required Skills & Qualifications:

• Strong experience with PySpark and Apache Spark
• Proficient in Python
• Experience with SQL and relational databases
• Solid understanding of Spark architecture (RDDs, DataFrames, Spark SQL, DAGs)
• Hands-on experience with big data ecosystems (Hadoop, Hive, HDFS)
• Experience with cloud platforms (AWS / Azure / GCP) and object storage
• Knowledge of data modeling, ETL concepts, and pipeline design
• Familiarity with version control tools (Git)
• Understanding of Linux/Unix environments