Staff Software Development Engineer - Data Platforms

JioStar; Bengaluru; On-site; INR 3,000,000 - 4,200,000; Full time

Job summary

JioHotStar in Bengaluru seeks a Staff Software Development Engineer to lead the design, build, and optimization of our data platform. You will handle hundreds of millions of RPM, process billions of events daily from Kafka, and manage petabytes of data in S3.

You will own end-to-end delivery, mentor engineers, and drive reliability, performance, and cost efficiencies for AI data pipelines and MCP infrastructure.

Qualifications

• 8+ years of progressive professional experience in data engineering.
• Experience with data platforms handling hundreds of millions of events/requests per minute and petabytes of storage.
• Strong programming proficiency in Python, Java, Scala, or Go.
• Deep expertise with distributed data processing frameworks (Spark, Flink).
• Experience building data pipelines for AI/ML model training and feature stores.
• Extensive experience with Kafka, Kinesis, or similar streaming platforms.
• Hands-on cloud experience (AWS/GCP/Azure) and data services (S3, EMR, Redshift, etc.).
• Strong SQL skills and knowledge of relational and NoSQL databases.
• Excellent communication and leadership abilities; able to mentor junior engineers.

Responsibilities

• Architecture & Design: Create architecture designs and evaluate trade-offs for new services and migrations.
• Real-time Data Pipeline: Build scalable, accurate, real-time events collection systems.
• Data Foundation for AI & MCP: Design data pipelines to support AI model training and autonomous AI Agents.
• Data Quality Assurance: Detect and fix data quality issues to maintain trust.
• System Enhancement: Improve existing data systems with latest tooling and standards.
• Cost Optimization: Improve unit economics across ingestion, storage, and processing components.

Skills

Distributed data processing

Programming (Python/Java/Scala/Go)

Kafka / streaming

Cloud platforms (AWS/GCP/Azure)

Data modeling

ETL/Orchestration

SQL / NoSQL databases

Mentorship / leadership

Communication

Tools

Apache Spark

Apache Flink

Airflow / Prefect

Snowflake / Redshift / BigQuery

Delta Lake / Iceberg / Hudi

S3 / HDFS

Kinesis

DynamoDB

PostgreSQL / MySQL

Job description

Job Summary

As a seasoned Staff Software Development Engineer in Data Platform team you will be at the forefront of designing, building, and optimizing the critical data infrastructure that handles hundreds of millions of requests per minute (RPM) for our ingestion services, processes billions of events daily from Kafka, and manages petabytes of data stored in S3. You will leverage your deep expertise to architect and deliver highly scalable, reliable, and performant data solutions that directly impact our global user base and drive our strategic initiatives.

About the team

Data platforms team at JioHotStar is building the next generation of AI-powered experiences and autonomous agents, leveraging our vast data to create truly innovative products. Beyond traditional data engineering, you’ll be instrumental in developing the data foundations for our AI initiatives, including the development of our own MCP server and the infrastructure supporting our cutting edge AI Agents.

Key Responsibilities

• Architecture & Design: Build POCs, create architecture designs, and evaluate trade-offs for new services and migrations
• Real-time Data Pipeline: Build and maintain highly scalable, accurate, clean and real-time events collection systems
• Data Foundation for AI & MCP: Specifically design and build data pipelines to support the development and training of AI models and autonomous AI Agents.
• This includes creating robust data infrastructure for our MCP, ensuring data quality and availability for personalized learning paths and skill validation.
• Data Quality Assurance: Continuously work towards detecting, alerting, and fixing quality issues to build trust in our data
• System Enhancement: Improve existing systems and services using latest tooling and engineering standards
• Cost Optimization: Improve unit economics across ingestion, storage, and processing components

Technical Leadership & Ownership

• Complete Development Lifecycle: Own the entire development process from ideation to deployment
• Cross-functional Collaboration: Work closely with different teams to achieve common goals through efficient coordination
• Technical Documentation: Maintain up-to-date documentation with architecture diagrams for entire projects
• Service Ownership: Take complete ownership of services within your workstream and drive continuous improvement
• Operational Excellence: Establish and maintain robust monitoring, alerting, and operational procedures for mission-critical data systems.

Team Leadership & Growth

• Knowledge Sharing: Write blogs and articles to help team members and the community
• Architecture Breakdown: Break down high-level architecture into manageable services and tasks
• Technology Trends: Learn and share new trends in technology, engineering, and productivity
• Team Productivity: Actively work towards improving team productivity and knowledge
• Mentorship: Guide and mentor junior team members

Business Impact

• Data Governance: Govern data assets across the entire company and audit data posture
• Time to Insight: Reduce time to insight for data team stakeholders
• Business Decision Support: Assist and challenge business decisions using data insights
• Stakeholder Service: Make data available to all stakeholders in an efficient, quick, quarriable, and intuitive manner

Skills and attributes for success

• Experience: 8+ years of progressive professional experience in data engineering, with a proven track record of designing, building, and operating large-scale, high-throughput data platforms, ideally in a consumer-facing or streaming industry.
• Extreme Scale Experience: Demonstrated experience working with data systems that handle hundreds of millions of events/requests per minute, billions of daily events, and petabytes of storage (e.g., S3, HDFS).
• Programming: Strong proficiency in at least one major programming language such as Python, Java, Scala, or Go, with a focus on building robust, production-grade data applications.
• Distributed Data Processing: Deep expertise and practical experience with distributed data processing frameworks like Apache Spark, Apache Flink, or equivalent technologies for batch and streaming workloads.
• AI/ML Data Experience: Specific experience in building data pipelines for Machine Learning model training, inference, and feature stores. Familiarity with data needs for AI Agents or similar intelligent systems.
• Messaging & Streaming: Extensive experience with high-throughput messaging queues and stream processing platforms like Apache Kafka, Kinesis, or equivalent.
• Cloud Platforms: Hands-on expertise with at least one major cloud platform (AWS, GCP, or Azure) and their data-related services, with a strong preference for AWS (S3, EMR, Kinesis, Glue, Redshift, etc.).
• Data Warehousing/Lakes: Strong understanding and hands-on experience with modern data warehousing concepts and data lake architectures (e.g., Snowflake, BigQuery, Redshift, Delta Lake, Iceberg, Hudi).
• ETL/Orchestration: Expertise in building, optimizing, and orchestrating complex ETL/ELT pipelines using tools like Apache Airflow, Prefect, or other workflow management systems.
• Databases: Strong SQL skills and experience with both relational (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., Cassandra, DynamoDB).
• Data Modeling: Advanced data modeling skills (dimensional, Kimball, data vault) for large, complex datasets.
• Problem-Solving: Exceptional analytical and problem-solving abilities, with a proven track record of debugging and optimizing complex distributed systems.
• Communication: Excellent communication, collaboration, and interpersonal skills, with the ability to articulate complex technical concepts to diverse audiences.
• Mentorship: Proven ability to mentor, guide, and technically lead senior and junior engineers.