Data Platform Engineer

Yulu; Hyderabad; In loco; INR 1.500.000 - 2.100.000; Tempo pieno

Descrizione del lavoro

Yulu, India’s leading shared micro-mobility platform, is seeking a Data Platform Engineer to design, build, and operate scalable data lakehouse infrastructure. You will own the architecture of our data lake and lakehouse, build robust data ingestion and streaming systems, and ensure high performance, reliability, and cost efficiency across the platform.

The role requires strong hands-on experience with AWS, Spark, Kafka, Python, SQL, and modern data lake technologies.

Competenze

• 4+ years of experience building and operating data platforms.
• Strong hands-on experience with the AWS data stack (S3, Glue, Athena, EMR, IAM).
• Strong experience with Apache Spark/PySpark and performance tuning.
• Production experience with Iceberg, Hudi, or Delta Lake.
• Strong Kafka production experience including design, retention and delivery guarantees.
• Strong Python skills for production-grade code.
• Strong SQL skills with distributed query execution and Parquet/ORC.
• Understanding of dimensional modelling and incremental processing.
• Experience with Airflow or similar workflow orchestration tools.
• Experience debugging production incidents in distributed systems.

Mansioni

• Design and evolve the architecture of the data lake and lakehouse.
• Build and operate data ingestion from OLTP sources and streaming pipelines.
• Own Spark on EMR, cost optimisation, and performance tuning.
• Manage Athena/Glue layer, catalog hygiene, and concurrency controls.
• Ensure reliability with monitoring, on-call response, and guardrails.

Conoscenze

Python

SQL

Kafka

Spark

Airflow

Cost optimisation

Data platform ownership

Data engineering

Strumenti

AWS data stack

Apache Iceberg

Hudi

Delta Lake

Kafka

EMR

Glue

Athena

Descrizione del lavoro

Yulu is India’s leading shared micro-mobility platform, revolutionizing urban transportation through smart, sustainable, and electric-first mobility solutions. With a rapidly growing fleet of tech-enabled electric two-wheelers and a robust battery-swapping infrastructure, Yulu makes last-mile commutes not only efficient but also planet-friendly.

Our IoT-driven platform and smart electric vehicles are helping cities reduce traffic congestion and carbon emissions while empowering millions with affordable and reliable transportation.

Backed by industry giants like Bajaj Auto and Magna International, Yulu operates at the intersection of mobility, technology, and sustainability. Our mission is to reduce congestion, cut emissions, and transform how India moves — one ride at a time.

With millions of rides completed, thousands of EVs on the road, and a rapidly expanding footprint, we’re not just building EVs — we’re building the future of urban mobility in India.

Role Summary

We are looking for a Data Platform Engineer to design, build, and operate scalable and reliable data platform infrastructure at Yulu. The role will involve owning the architecture of our data lake and lakehouse, building robust data ingestion and streaming systems, and ensuring high performance, reliability, and cost efficiency across the platform.

The role requires strong hands-on experience with AWS, Spark, Kafka, Python, SQL, and modern data lake technologies. You will work on large-scale data infrastructure, platform migrations, distributed data processing, production reliability, and self-service tooling that enables data engineers and analysts to work efficiently.

Key Responsibilities

1. Data Platform & Lakehouse Architecture

• Design and evolve the architecture of the data lake, including storage layout, table formats, partitioning, file sizing, and compaction.
• Make and document build-vs-buy and tooling decisions considering cost, operational effort, and migration risk.
• Own the lakehouse table format strategy across Apache Iceberg, Hudi, or Delta Lake.
• Manage table maintenance activities such as compaction, snapshot expiry, orphan file cleanup, and schema evolution.
• Plan and execute large-scale migrations across hundreds of tables with minimal downstream disruption.
• Design appropriate rollback mechanisms and manage concurrency, write conflicts, commit failures, locking, and idempotent writes.

2. Data Ingestion & Streaming

• Build and operate data ingestion from OLTP databases such as MySQL and PostgreSQL into the data lake through batch and CDC pipelines.
• Own Kafka infrastructure and usage patterns, including topic design, partitioning, schema registry, consumer groups, retention, and replay.
• Solve streaming data challenges such as exactly-once/effectively-once delivery, out-of-order events, late-arriving data, and backfills.

3. Compute & Data Processing

• Own Spark on EMR, including cluster configuration, autoscaling, job tuning, and cost optimisation.
• Identify and resolve performance issues such as data skew, shuffle, small files, spill, and memory pressure.
• Own the Athena and Glue layer, including workgroups, query performance, partition management, catalogue hygiene, and concurrency limits.

4. Platform Reliability & Cost Optimisation

• Own the reliability of the data platform through monitoring, alerting, on-call response, and post-incident follow-through.
• Track platform costs by job, team, and table and identify opportunities to reduce unnecessary spend.
• Build cost and usage guardrails to prevent runaway queries and clusters.
• Build self-service tooling and internal libraries that allow data engineers and analysts to work independently.
• Manage infrastructure as code and CI/CD for data infrastructure.

Preferred Qualifications

Must Have

• 4+ years of experience building and operating data platforms, with hands-on ownership of production infrastructure.
• Strong hands-on experience with the AWS data stack, including S3, Glue, Athena, EMR, and IAM.
• Strong experience with Apache Spark/PySpark, including query-plan analysis and performance tuning.
• Production experience with at least one open table format such as Apache Iceberg, Hudi, or Delta Lake.
• Strong production experience with Kafka, including topic and partition design, consumer semantics, offset management, and delivery guarantees.
• Strong Python skills with the ability to write production-grade code.
• Strong SQL skills and understanding of distributed query execution, partition pruning, predicate pushdown, join strategies, and Parquet/ORC.
• Understanding of analytical data modelling, including dimensional modelling, slowly changing dimensions, and incremental processing.
• Experience with Airflow or similar workflow orchestration tools.
• Experience debugging production incidents in distributed systems and identifying root causes.

Good to Have

• Experience with query engines such as Trino, Presto, StarRocks, ClickHouse, Druid, Pinot, or Redshift.
• Experience with Debezium or another CDC tool.
• Experience with Infrastructure as Code tools such as Terraform or CloudFormation.
• Experience with container orchestration platforms such as Kubernetes or ECS.
• Experience managing large-scale platform migrations without disrupting downstream consumers.
• Experience with Redis or similar technologies for caching, distributed locking, or probabilistic data structures.

What We Look For

• Strong ownership of production data infrastructure.
• Ability to make sound platform and architectural decisions and explain the trade-offs involved.
• Strong focus on cost as a first-class design consideration.
• Ability to design for reliability, rollback, and blast-radius containment.
• Strong understanding of data lake/lakehouse internals and the ability to explain how different table formats handle operations such as deletes and updates.