Senior Data Engineer - GCP

Objective of the Role:

We are looking for Data Engineers to take a lead role in developing and delivering innovative data platform solutions. This role demands a proactive approach to ownership, driving initiatives from concept to deployment, and ensuring high-quality outcomes that align with both technical and product goals. The ideal candidate should be comfortable working with modern data architectures such as Data Lakehouse, real-time streaming platforms, and AI-enabled data pipelines.

You Will

Drive architecture decisions for modern data platforms includingData Lakehouse, streaming architectures, and event-driven systems. Define and enforce data modelling standards, schema design, and data contracts across multiple services and teams.

• Optimize large-scale data processing pipelines for performance, cost, and latency across batch and streaming workloads.
• Lead initiatives around platform standardization, reusable frameworks, and shared data infrastructure components.
• Design and implement event-driven data pipelines using Kafka, Flink, or similardistributed streaming frameworks.
• Design and implement Data Access Layers and Data APIs that connect to multiple structured and unstructured data sources and provide fast responses using distributed caching systems like Redis or Memcached.
• Develop cloud-native, scalable data pipelines capable of handling high-volume data ingestion and transformation.
• Implement data transformation and modeling frameworks such as dbt, Spark SQL, or SQL-based transformation layers.
• Build reliable work flow orchestration pipelines using tools like Apache Airflow, Dagster.

• Collaborate with ML teams to design feature stores, ML training pipelines, and real-time inference data pipelines.
• Build scalable pipelines for AI/ML workloads, vector search pipelines, and recommendation systems.

YouMust Have

• Ensure data governance, security,and compliance practices are implemented across data pipelines.
• 3-8 years of experience on data platform automation, including infrastructure-as-code.
• Develop real-time analytics systems capable of processing millions of events per second.
• Experience with NoSQL databases such as MongoDB,Cassandra, or DynamoDB.
• Exposure to modern analytical data warehouses such as Snowflake, Amazon Redshift, Google BigQuery, or ClickHouse.
• Exposure to data mesh implementations or domain-oriented data architectures.

Data Architecture & Platforms

• Deep understanding of batch and streaming data architectures.
• Knowledge of event-driven architectures, microservices, and data mesh concepts.
• Hands-on experience with modern lakehouse data formats like Delta Lake, Apache Iceberg, or Apache Hudi.
• Strong expertise in Big Data architecture best practices, including designing systems that overcome limitations of traditional data warehouses.

Big Data & Data Processing Frameworks

Hands-on experience with one or more of the following:

• Apache Spark / PySpark
• Apache Flink
• Apache Beam
• Kafka Streams

Experience with large-scale distributed data processing is highly preferred.

Data Infrastructure & Cloud

• Experience building cloud-native data platforms.
• Familiarity with cloud data stacks: GCP

Data Engineering Best Practices

• Strong understanding of ETL/ ELT pipeline design and optimization.
• Experience with distributed file systems and storage systems (S3, HDFS, ADLS).
• Knowledge of data quality frameworks, lineage tracking, and observability tools (e.g., Great Expectations, Monte Carlo, OpenLineage).
• Familiarity with CI/CD practices for data pipelines and infrastructure automation.

Strong Interpersonal Skills

• Ability to facilitate complex technical discussions with team members and senior stakeholders.
• Strong ownership mindset and ability to drive projects from ideation to production.
• Resilient and adaptable in fast-changing, high-growth environments.
• Comfortable working in multicultural and distributed teams within highly competitive markets.