Data Platform Engineer (IC Role - SDE 2 to Staff)

Weekday AI (YC W21); Hyderabad; On-site; INR 1,500,000 - 2,500,000; Full time

Benefits offered by this job

High ownership

Impactful work in AI

Job summary

A leading data solutions provider is looking for a Data Engineer with 3-12 years of experience to take full ownership of the data lakehouse and manage real-time stream processing frameworks. The ideal candidate will have strong expertise in tools like Spark and Kafka, proficient in AWS services, and preferably skilled in Java, Python, or Scala. This role offers the opportunity to directly impact AI-powered supply chain decisions for Fortune 500 clients, working in a team that values high ownership and scalable challenges.

Qualifications

• 3-12 years of experience in data engineering, at least 1-7 years building managing data platforms.
• Deep hands-on expertise with data engineering tools and frameworks.
• Experience managing daily processing of terabytes and billions of events.

Responsibilities

• Take full ownership of the data lakehouse including architecture and reliability.
• Develop and manage real-time stream processing frameworks.
• Design and scale OLAP stores for analytics.
• Implement cost observability measures for expenses.

Skills

Data engineering

AWS data ecosystem

Pipeline management

Real-time stream processing

Java

Python

Scala

Tools

Spark

Kafka

Airflow

Hudi/Delta Lake

Presto/Trino

Debezium

DBT

Airbyte

Job description

This role is for one of the Weekday's clients

Min Experience: 3 years

Location: Bengaluru, Chennai, Hyderabad, Pune

JobType: full-time

This role goes beyond simply maintaining pipelines. You will be responsible for designing and managing the foundational infrastructure upon which everything else is built.

What You'll Do

• Take full ownership of the data lakehouse, including its architecture, ingestion from CDC sources (Postgres, DynamoDB), scalability, and reliability
• Develop and manage real-time stream processing frameworks for applications such as anomaly detection, customer 360 views, and live supply chain signals—ensuring high throughput and low latency
• Design and scale OLAP stores to support both real-time and batch processing for internal analytics and AI/ML pipelines
• Create self-service ETL and query frameworks that enable data consumers to operate quickly without creating bottlenecks for the platform team
• Implement cost observability measures that provide detailed insights into compute, storage, and query expenses by job, user, and source—and then take action to reduce these costs
• Build data movement APIs and reverse-ETL pipelines to efficiently deliver data to downstream consumers at scale
• Establish robust job orchestration layers that remain stable under scale (experience with YARN, Airflow, EMR is a plus)

Who You Are

• Have 3-12 years of experience in data engineering, with at least 1-7 years focused on building or managing a data platform (beyond just pipelines)
• Possess deep hands‑on expertise with tools like Spark, Hudi/Delta Lake, Kafka, Airflow, Debezium, Presto/Trino, DBT, Airbyte
• Are comfortable working with the AWS data ecosystem, including EMR, S3, Athena, Glue, and CloudWatch
• Have managed daily processing of terabytes and billions of events—scale is part of your daily experience
• Have demonstrably reduced infrastructure costs and can provide metrics showing your impact
• Are proficient in Java, Python, or Scala—ideally experienced in all three
• Preferably have experience as a pod lead or tech lead; you're the person others rely on when things break at 2 a.m

Bonus

• Experience with OLAP engines such as Pinot, Druid, or ClickHouse
• Have built or contributed to data movement or reverse-ETL APIs
• Familiarity with feature stores (Feast, Feathr) or data catalog tools like Datahub

What Makes This Different

Our data platform powers AI that drives supply chain decisions for Fortune 500 companies. You'll directly witness the real business impact of your work—not just through dashboards. Join a small team with high ownership and the challenge of working at true scale.