Skill: Databricks + Pyspark
Experience: 6 to 9 years
Location: AIA Kolkata
Job description
We are looking for a highly skilled Data Engineer with expertise in PySpark and Databricks to design, build, and optimize scalable data pipelines for processing massive datasets.
Key Responsibilities:
- Build & Optimize Pipelines: Develop high-throughput ETL workflows using PySpark on Databricks.
- Data Architecture & Engineering: Work on distributed computing solutions, optimize Spark jobs, and build efficient data models.
- Performance & Cost Optimization: Fine-tune Spark configurations, optimize Databricks clusters, and reduce compute/storage costs.
- Collaboration: Work closely with Data Scientists, Analysts, and DevOps teams to ensure data reliability.
- ETL & Data Warehousing: Implement scalable ETL processes for structured & unstructured data.
- Monitoring & Automation: Implement logging, monitoring, and alerting mechanisms for data pipeline health and fault tolerance.