Senior Data Engineer

Rapido; Bengaluru; Sur place; INR 400 000 - 700 000; Plein temps

Résumé du poste

Rapido in Bengaluru seeks a Senior Data Engineer to design and implement robust data pipelines, ensuring accuracy and SLA adherence. You will own end-to-end data infrastructure, collaborating with data producers and consumers to optimize ingestion, processing and serving while promoting best practices.

You will work with petabytes of data and a suite of tools including Hadoop, Spark, Flink, Hive, Kafka, and Airflow, building scalable pipelines and observability into the data platform to support

Qualifications

• Around 5+ years of experience in data engineering.
• Experience with HDFS, S3, NoSQL and distributed platforms like Hadoop, Spark, Flink, Hive, Kafka, Oozie, Airflow and Elasticsearch.
• Experience with MapR, Cloudera, Hortonworks and cloud Hadoop distributions (GCP preferred).
• Strong data modeling and ETL skills; ML model experience.
• Experience with scheduling tools like Airflow and monitoring stacks (Prometheus/Grafana).
• Solid software engineering fundamentals and data structures.

Responsabilités

• Create and improve complex data processing pipelines with CD practices.
• Deploy models in real-time applications via microservices or real-time pipelines.
• Maintain infrastructure as code with Terraform and Kubernetes.
• Build observability and alerting around data pipelines.
• Identify architectural shortcomings and propose solutions in forums.
• Mentor junior engineers on best practices.

Connaissances

Data engineering

ETL concepts

Distributed systems

Outils

HDFS

S3

NoSQL databases

Spark

Flink

Hive

Kafka

Oozie

Airflow

Elasticsearch

Terraform

Kubernetes

Description du poste

Founded in 2015, Rapido is India’s leading multi-modal mobility platform. What began with bike taxis has rapidly evolved to include auto-rickshaws, cab-hailing, and peer-to-peer deliveries. Today, Rapido commands a dominant 70% market share in bike taxis, 40% in auto rides, and holds the second-largest position in the cab segment with a 22% share—establishing us as the overall market leader in ride-hailing. With over 4 million rides completed daily and more than 2 billion rides served to date, Rapido operates in 200+ cities and is aggressively expanding toward a presence in 500 cities. In 2024, we proudly achieved unicorn status with a valuation of $1.1 billion.Driven by technology and innovation, we’re expanding into new categories while remaining committed to improving last-mile connectivity and empowering livelihoods at scale.

Opportunities don't happen, you create them!

Job Summary:

As a Senior Data Engineer, you will design, create and implement optimal data pipelines. And also improve existing data pipelines and processes around it. Data accuracy, SLA adherence being core to our data product, it is expected to build & maintain robust & reliable data & infra pipelines. You are expected to communicate with producers and consumers of data to better understand the patterns of ingestion,processing & serving and thus bring goodness of best practices to our data pipelines. We believe in complete ownership of streams so as subject matter expert of certain streams you will be looking after the necessary infrastructure requirements for your pipelines and are expected to own end-to-end deployment of the same. Being in this role means that you will own multiple processes which directly impact business and are critical for daily operations. Also work with petabytes of data and hundreds of complex data pipelines.

Job Responsibilities:

• Create and improve complex data processing pipelines, while keeping accuracy, reliability and cost in mind and deploy the same using best Continuous Delivery practices.
• Deploy models in real-time applications either as part of a micro service(HTTP-HyperText Transfer Protocol or RPC-Remote Procedure Call) with bounded context or as real-time pipelines producing events in response to user actions on ground.
• Maintain infrastructure as code using tools like terraform and comfortable with working in kubernetes.
• Build observability & alerting systems around data-pipelines.
• Identify, analyze tech / architecture shortcomings in existing systems and proactively discuss in architecture forums with proposed solutions.
• Being able to mentor, guide junior engineers for best engineering practices. Junior Developers will look up to you as mentors.

Job Requirements:

• Around 5+ years of experience. For transitioned data engineers over all experience of 5+ years in preferred.
• Experience in Hadoop Distributed File System(HDFS), Simple storage Service (S3), NoSql Databases and distributed platforms like Hadoop, Spark, Flink, Hive, Kafka, Oozie, Airflow, Elastic search etc.
• Experience in any of MapR, Cloudera, and Horton Works and/ or cloud based Hadoop Distributions (Google Cloud Platform preferred).
• Have a good understanding of data modeling, Extract, Transform and Load(ETL), distributed systems.
• Experience creating and building data centric applications involving Machine Learning(ML) models.
• Hands on experience with scheduling tools like Airflow, monitoring stack prometheus-grafana.
• Fundamental software engineering principle & data structure knowledge are must.

What’s there for you?

In the Data team at Rapido you will get exposure to every stack possible – big data, software engineering(java/scala), ML-ops,data-ops and more.

• Be a part of a platform which serves hundreds of users with their data needs with state of the art Trino clusters.
• Get to experience real time applications deployed in Flink.
• Create and manage tools and frameworks built on top of open source technologies.
• Work with mammoth kubernetes clusters which host plethora of applications and tools.
• Hands on experience on open source big data tools and its best practices. Excited to solve challenges?