Senior Manager - Data Plaform Engineer

A Senior Data Platform Engineer with 12-14 years of experience to own and build the core data platform for a multi-region CPaaS system supporting multiple digital channels.

The platform processes 40,000+ transactions per second, with multiple ( 10) state updates per transaction, and powers real-time metrics, billing, reporting, campaign analytics, and audits.

This role is hands-on and ownership-driven, focused on data correctness, scalability, and reliability in large-scale streaming systems.

Experience & Background

12-14 years of experience in data engineering, platform engineering, or distributed systems

Proven experience building large-scale, streaming-heavy data platforms

Strong hands-on background in real-time analytics and data consistency

Comfortable operating at Principal & Staff Engineer level with full technical ownership

Responsibilities:

Mandatory / Primary Responsibilities

Own the end-to-end data architecture for high-throughput, event-driven CPaaS systems.

Design state-transition and event-sourced data models handling millions of updates per second.

Build and operate streaming data pipelines using Apache Pulsar for ingestion and Apache Flink for stateful processing and real-time aggregation.

Design and maintain a data lake using Apache Hudi to support: Upserts and deletes

Incremental queries

Time-travel and auditability

Build real-time analytics datasets in ClickHouse for dashboards, tenant metrics, and campaign analysis.

Define and enforce processing semantics, including idempotency, deduplication, ordering, and replay safety.

Own data correctness, completeness, and freshness SLAs across streaming and analytics systems.

Design multi-region data ingestion and aggregation, including replication, failover, and reprocessing.

Lead hands-on development, code reviews, performance tuning, and production troubleshooting.

Secondary Responsibilities

Mentor senior engineers and review architectural designs.

Define data standards, schema evolution practices, and platform guidelines.

Participate in release planning and technical prioritization.

Evaluate new data technologies and patterns through POCs.

Collaborate with product and operations teams on data-driven features.

Skills

Mandatory / Primary Skills

Apache Pulsar (topics, partitions, subscriptions, geo-replication)

Apache Flink (stateful stream processing, windows, checkpoints)

Apache Hudi (Merge-On-Read, Copy-On-Write, incremental processing)

ClickHouse (real-time analytics, high-cardinality datasets)

Event-driven and streaming data architectures

Event sourcing and state-transition modeling

Idempotent, replay-safe, and failure-tolerant data pipelines

Designing systems for high TPS and frequent state updates

Multi-region data design and disaster recovery

Secondary Skills

Redis for low-latency caching, counters, and ephemeral state

PostgreSQL for transactional, relational, and reference data

MySQL for operational data stores and legacy integrations

Low-Level Design (LLD) for scalable, fault-tolerant components

High-Level Design (HLD) for distributed, multi-region systems

NoSQL stores for hot state (e.g., Mongo, DynamoDB, Cassandra)

Relational databases for reference/config data

Data lake query engines (Athena, Spark)

Observability for data pipelines (lag, freshness, reconciliation)

Containerized runtimes and cloud-native deployments

Prior experience with CPaaS, telecom, or large-scale messaging platforms.

If interested please share your updated resume to janani.k1@tatacommunications.com