Big Data Lead / Architect

Job Title: Big Data Lead/ Architect (Spark, PySpark & SQL)

Experience: 715 Years

Location: Mumbai, India

Department: Big Data, Analytics & Cloud

Company: Smart Analytica Private limited is niche Data and AI company. Based in Pune, we are pioneers in data-driven innovation, transforming enterprise data into strategic insights. Established in 2017, our team has experience in handling large datasets up to 20 PBs in a single implementation, delivering many successful data and AI projects across major industries, including retail, finance, telecom, manufacturing, insurance, and capital markets. We are leaders in Big Data, Cloud and Analytics projects with super specialization in very large Data Platforms.

https://smart-analytica.com

Empowering Your Digital Transformation with Data Modernization and AI

Job Summary

We are looking for an experienced Big Data Architect with strong expertise in Hadoop ecosystem technologies, Spark, PySpark, and SQL to design and lead scalable enterprise data solutions. The ideal candidate should have hands-on experience in distributed data processing, data architecture, performance optimization, and technical leadership.

Key Responsibilities

• Architect and implement scalable Big Data solutions using Hadoop technologies
• Design and optimize Spark/PySpark-based data processing pipelines
• Perform performance tuning of Spark jobs and SQL queries
• Design and manage ETL/data integration workflows
• Ensure scalability, reliability, and security of data platforms
• Collaborate with business stakeholders, data engineers, and analysts for requirement gathering and solution delivery
• Oversee Hadoop cluster architecture and resource optimization
• Maintain technical documentation and architecture standards
• Mentor team members and provide technical leadership
• Drive continuous improvements and best practices across data platforms

Required Skills & Expertise

• Strong experience in Big Data technologies:
• Hadoop Ecosystem: Hive, Impala, HDFS, YARN, Kudu, Ozone
• Apache Spark (Core, SQL, Streaming)
• PySpark development and optimization
• Advanced SQL knowledge with query optimization expertise
• Hands-on experience with large-scale distributed data processing
• Strong understanding of ETL frameworks and data integration
• Knowledge of Hadoop administration and cluster management
• Understanding of data security and governance concepts
• Excellent problem-solving, communication, and stakeholder management skills

Preferred Qualifications

• Bachelors/Masters degree in Computer Science or related field
• Exposure to AWS/Azure/GCP is an added advantage
• Relevant Big Data certifications preferred

Experience Required

• 10+ years of overall IT experience
• 5+ years of relevant experience in Big Data Architecture

If you have a passion for big data technologies and a strong background in Hadoop, we encourage you to apply and be a part of our innovative team at Smart Analytica Private Limited.