Senior Data Engineer

About the Company

We are seeking a highly skilled Data Engineer with strong experience in building scalable data pipelines, distributed data processing systems, and cloud-based data architectures. The ideal candidate will have hands-on expertise with Python/Spark, SQL, Apache Spark, Apache Kafka, Cloud Platforms (AWS/GCP/Azure), and Infrastructure as Code (IaC) tools like Terraform or CloudFormation. Experience working with Data Warehousing (DWH) platforms is essential.

About the Role

Key Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines using Python, SQL, and Apache Spark.
• Develop and manage real-time streaming solutions using Apache Kafka.
• Implement data workflows on cloud platforms (AWS, GCP, or Azure) leveraging managed services.
• Work with data warehousing solutions such as Redshift, BigQuery, Snowflake, Synapse, or onprem DWH tools.
• Collaborate with cross-functional teams to translate business requirements into technical data solutions.
• Optimize data pipelines for performance, reliability, and cost-efficiency.
• Infrastructure provisioning using IaC tools (Terraform, AWS CloudFormation, or equivalent).
• Ensure data quality, governance, and security best practices across all environments.
• Troubleshoot and resolve issues across batch and streaming data systems.
• Participate in code reviews, architecture discussions, and continuous improvement initiatives.

Qualifications

• Strong programming experience in Python and Pyspark/Spark.
• Advanced proficiency in SQL for complex data transformations.
• Hands-on experience with Apache Spark for large-scale data processing.
• Proficiency in Apache Kafka for streaming data pipelines.
• Practical experience with any major cloud provider (AWS, GCP, or Azure).
• Experience working on Data Warehouse platforms.
• Knowledge of Infrastructure as Code (Terraform or AWS CloudFormation).
• Strong understanding of distributed systems, data modeling, and pipeline orchestration.
• Experience with Git, CI/CD, and modern DevOps practices is a plus.

Preferred Skills

• Experience with managed cloud data services like:
• AWS Glue, EMR, RDS, Redshift
• GCP Dataflow, BigQuery, Dataproc
• Azure Databricks, Synapse
• Exposure to Airflow, DBT, or other orchestration tools.
• Knowledge of containerization (Docker, Kubernetes).
• Background in performance tuning of big data workloads.

Soft Skills

• Strong analytical and problemsolving abilities.
• Excellent communication and collaboration skills.
• Ability to work in fast-paced and agile environments.
• Self-driven with a strong learning mindset.