Big Data Developer

JD Big Data Engineer

Designation: C1/C2

Experience: 7-11 years

Location: Pune/ Chennai /Bengaluru

Role Overview:

We are seeking a highly skilled Big Data Engineer to design, develop, and maintain scalable data pipelines and back-end systems. The ideal candidate will have strong expertise in PySpark, Python, SQL and Hadoop, along with experience in building robust data workflows and managing large datasets.

Key Responsibilities:

Design and implement data pipelines for processing and transforming large-scale datasets.

Develop and optimize PySpark and Python scripts for data ingestion, transformation, and analysis.

Manage and maintain Hadoop-based ecosystems and distributed data processing frameworks.

Create and maintain Hive tables and ensure efficient query performance.

Collaborate with cross-functional teams to integrate data solutions into business processes.

Monitor, troubleshoot, and optimize data workflows for performance and reliability.

Ensure data quality, security, and compliance with organizational standards.

Required Skills & Experience:

Strong proficiency in PySpark, Python, and SQL.

Hands-on experience in data pipeline development and ETL processes.

Expertise in Hive and working with large-scale relational datasets.

Familiarity with tools such as: PyCharm, Autosys, UDeploy, TeamCity, Hue, Impala, Anaconda Navigator.

Solid understanding of distributed computing and big data architectures.

Good to Have:

Experience with PostgreSQL, Flask, Airflow, Apache NiFi.

Knowledge of CI/CD practices and deployment automation.

Exposure to cloud-based big data platforms (AWS, Azure, GCP).