Lead Engineer - Data & AI Platform

Recrew AI; New Delhi; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

The company is building a next-generation media super-app that unifies video, audio, radio, micro-drama, news, and commerce on a single digital platform. This role owns the scalable data and ML infrastructure powering content discovery, personalised recommendations, and user engagement at scale.

The engineer bridges production data systems and AI/ML teams, moving models from experimentation into reliable, high-throughput production environments.

Qualifications

• 7–9 years of hands-on software or data engineering experience.
• Proficiency in Python, Java, or Scala for data and ML pipeline development.
• Production experience with distributed data processing and streaming technologies such as Spark, Kafka, Flink, or Airflow.
• Hands-on experience with cloud-based data infrastructure on AWS, GCP, or Azure.
• Demonstrated experience deploying ML/AI models into production environments, including batch and real-time inference pipelines.
• Proficiency with Docker and Kubernetes for containerising and orchestrating model and pipeline workloads.
• Strong understanding of data architecture, distributed systems, and data modelling for both analytical and ML use cases.

Responsibilities

• Design and build scalable data architecture for a high-volume consumer media platform, covering ingestion, transformation, storage, and serving layers for user activity, content consumption, and commerce data.
• Build and maintain batch and real-time streaming pipelines processing millions of user events and large volumes of multimedia metadata using technologies such as Spark, Kafka, Flink, or Airflow.
• Architect data lakes, warehouses, and analytical datasets supporting product analytics, reporting, and AI/ML use cases; enforce data quality, lineage, observability, and governance across all pipelines.
• Develop feature engineering pipelines and reusable feature stores for recommendation, personalisation, ranking, search, and content intelligence systems.
• Build training and inference pipelines connecting production data with ML models; enable continuous feedback loops for model retraining from live user interactions.
• Deploy ML/AI models into production on AWS, GCP, or Azure; containerise and orchestrate model workloads using Docker and Kubernetes; build APIs/services for consumer-facing AI integration.
• Monitor model and pipeline performance — latency, scalability, reliability — and partner with AI/ML teams to operationalise models from experimentation to production.

Skills

Python/Java/Scala

Distributed systems

Data pipelines

Cloud platforms

Tools

Spark

Kafka

Flink

Airflow

Docker

Kubernetes

AWS

GCP

Azure

Job description

Type: Full-time

About Company

The company is one of India's largest private FM radio networks. It reaches over 40 million listeners weekly across 49 cities, 1,000+ towns, and 50,000 villages.

It is the only private FM station broadcasting from Jammu & Kashmir, with an international presence in Singapore and Bhutan. Its digital platform extends the brand into curated lifestyle, music, and news content.

The company operates at the intersection of legacy broadcast media and digital innovation, with a team of 740.

Position Overview

The company is building a next-generation media super-app that unifies video, audio, radio, micro-drama, news, and commerce on a single digital platform. This role owns the scalable data and ML infrastructure powering content discovery, personalised recommendations, and user engagement at scale. The engineer in this role bridges production data systems and AI/ML teams, moving models from experimentation into reliable, high-throughput production environments.

Role & Responsibilities

• Design and build scalable data architecture for a high-volume consumer media platform, covering ingestion, transformation, storage, and serving layers for user activity, content consumption, and commerce data.
• Build and maintain batch and real-time streaming pipelines processing millions of user events and large volumes of multimedia metadata using technologies such as Spark, Kafka, Flink, or Airflow.
• Architect data lakes, warehouses, and analytical datasets supporting product analytics, reporting, and AI/ML use cases; enforce data quality, lineage, observability, and governance across all pipelines.
• Develop feature engineering pipelines and reusable feature stores for recommendation, personalisation, ranking, search, and content intelligence systems.
• Build training and inference pipelines connecting production data with ML models; enable continuous feedback loops for model retraining from live user interactions.
• Deploy ML/AI models into production on AWS, GCP, or Azure; containerise and orchestrate model workloads using Docker and Kubernetes; build APIs/services for consumer-facing AI integration.
• Monitor model and pipeline performance — latency, scalability, reliability — and partner with AI/ML teams to operationalise models from experimentation to production.

Must Have Criteria

• 7–9 years of hands‑on software or data engineering experience, with a track record of building and operating large‑scale data platforms.
• Strong proficiency in Python, Java, or Scala for data and ML pipeline development.
• Production experience with distributed data processing and streaming technologies such as Spark, Kafka, Flink, or Airflow.
• Hands‑on experience with cloud‑based data infrastructure on AWS, GCP, or Azure, including data lakes, warehouses, and distributed storage.
• Demonstrated experience deploying ML/AI models into production environments, including batch and real‑time inference pipelines.
• Proficiency with Docker and Kubernetes for containerising and orchestrating model and pipeline workloads.
• Strong understanding of data architecture, distributed systems, and data modelling for both analytical and ML use cases.

Nice to Have

• Experience building data infrastructure for recommendation, personalisation, search, or content ranking systems.
• Exposure to feature stores, MLOps platforms, and ML training/inference workload management.
• Prior experience on high‑scale B2C or consumer internet platforms handling behavioural and event‑streaming data.
• Background in media, OTT, video, audio, music, social media, gaming, or news platforms.
• Exposure to Generative AI, embeddings, vector databases, or LLM‑based applications.

What We Offer

• End‑to‑end ownership of data and AI platform infrastructure for a consumer product reaching tens of millions of users.
• Opportunity to work at the convergence of broadcast media and digital innovation on a greenfield super‑app build.
• Collaborative environment with AI/ML, product, and engineering teams working on high‑impact personalisation and content intelligence problems.