Big Data Developer

Job Description:

Design, develop, and maintain scalable data processing pipelines using Hadoop and Spark.

Implement data integration and ETL processes to ingest and transform large datasets.

Collaborate with data scientists, analysts, business partners and other stakeholders to understand data requirements and deliver solutions.

Optimize and tune Hadoop and Spark jobs for performance and efficiency.

Manage and maintain data storage solutions, ensuring data integrity and security.

Utilize GitHub for version control and collaboration on code development.

Work with CDP (Cloudera Data Platform) to manage and deploy data applications.

Integrate and manage data solutions on Cloud Azure, and Snowflake ensuring seamless data flow and accessibility.

Monitor and troubleshoot data processing workflows, resolving issues promptly.

Stay updated with the latest industry trends and technologies in big data and cloud computing.

Explore and develop skills in newer technologies according to big data technology roadmap.

Integrate and manage data solutions on Cloud Azure and on-premises infrastructure, ensuring seamless data flow and accessibility.

Qualifications:

Strong experience with Apache Spark, Hadoop ecosystem, Hive, Iceberg, HBase, Apache Kafka, Spark Streaming or Flink, and Oozie (Airflow/NiFi good to have).

Proficient in Scala, Java, and/or Python with solid coding and debugging skills.

Applied experience with Solr for indexing and search workloads.

Familiarity with Git/GitHub, GitHub Actions, Azure DevOps; good to have cloud experience (Azure/AWS).

Knowledge of SQL, data modeling (nice to have), and experience working in UNIX or UNIXlike environments.

Experience with SCRUM or similar Agile frameworks; strong problemsolving skills; able to produce clear technical documentation.

Selfstarter, effective individual contributor, and good to have Oil & Gas domain knowledge.

Shift 2nd 1.00 PM 10.00 PM