Pyspark developer

Role - Pyspark developer

Required Technical Skill Set -Pyspark, Redshift, PostgreSQL

year of experience- 4 to 8 years

Location- Chennai / Hyderabad

Desired Competencies (Technical/Behavioral Competency)

Must-Have

1. 5+ years of experience in data engineering, with strong focus on PySpark/Spark for big data processing.
2. Expertise in building data pipelines and ingestion frameworks from relational, semi-structured (JSON, XML), and unstructured sources (logs, PDFs).
3. Proficiency in Python with strong knowledge of data processing libraries.
4. Strong SQL skills for querying and validating data in platforms like Amazon Redshift, PostgreSQL, or similar.
5. Experience with distributed computing frameworks (e.g., Spark on EMR, Databricks).
6. Familiarity with workflow orchestration tools (e.g., AWS Step Functions, or similar).
7. Solid understanding of data lake / data warehouse architectures and data modeling basics.

Good-to-Have

1. Experience with AWS data services: Glue, S3, Redshift, Lambda, CloudWatch, etc.
2. Familiarity with Delta Lake or similar for large-scale data storage.
3. Exposure to real-time streaming frameworks (e.g., Spark Structured Streaming, Kafka).
4. Knowledge of data governance, lineage, and cataloging tools (e.g., AWS Glue Catalog, Apache Atlas).
5. Understanding of DevOps/CI-CD pipelines for data projects using Git, Jenkins, or similar tools.