Senior Data Engineer

Mployee.me; Bengaluru; On-site; INR 3,000,000 - 6,000,000; Full time

Job summary

Rapido is seeking a Senior Data Engineer to design, build, and optimize large-scale data pipelines across multi-modal mobility data. You will ensure accuracy and SLAs while collaborating with data producers and consumers to optimize ingestion, processing, and serving patterns.

You will own end-to-end deployment, include infrastructure as code, and work with massive data and many pipelines. You will work with big data tech stacks, real-time processing, and a variety of open-source tools on a

Qualifications

• Around 5+ years of experience in data engineering.
• Experience with HDFS, S3, NoSQL and distributed platforms like Hadoop, Spark, Flink, Hive, Kafka, Oozie, Airflow, Elasticsearch.
• Experience with MapR, Cloudera and Hortonworks and/or cloud-based Hadoop distributions (GCP preferred).
• Strong data modeling, ETL and distributed systems understanding.
• Experience building data-centric applications involving ML models.
• Hands-on with scheduling tools like Airflow and monitoring with Prometheus/Grafana.
• Solid software engineering fundamentals and data structures knowledge.

Responsibilities

• Create and improve complex data processing pipelines with accuracy and reliability.
• Deploy models in real-time applications as microservices or real-time pipelines.
• Maintain infrastructure as code using Terraform and operate in Kubernetes environments.
• Build observability and alerting around data pipelines.
• Identify architecture shortcomings and discuss solutions in forums.
• Mentor junior engineers and promote best engineering practices.

Skills

Hadoop

Spark

Flink

Airflow

Kafka

HDFS

NoSQL

Data modeling

ETL

Kubernetes

Terraform

Prometheus

Grafana

Tools

Terraform

Kubernetes

Airflow

Prometheus

Grafana

ElasticSearch

Job description

About Rapido

Founded in 2015, Rapido is India's leading multi-modal mobility platform. What began with bike taxis has rapidly evolved to include auto-rickshaws, cab-hailing, and peer-to-peer deliveries. Today, Rapido commands a dominant 70% market share in bike taxis, 40% in auto rides, and holds the second-largest position in the cab segment with a 22% share-establishing us as the overall market leader in ride-hailing. With over 4 million rides completed daily and more than 2 billion rides served to date, Rapido operates in 200+ cities and is aggressively expanding toward a presence in 500 cities. In 2024, we proudly achieved unicorn status with a valuation of $1.1 billion.Fueled by technology and innovation, we're expanding into new categories while remaining committed to improving last-mile connectivity and empowering livelihoods at scale.

Opportunities don't happen, you create them!

Job Summary

As a Senior Data Engineer, you will design, create and implement optimal data pipelines. And also improve existing data pipelines and processes around it. Data accuracy, SLA adherence being core to our data product, it is expected to build & maintain robust & reliable data & infra pipelines. You are expected to communicate with producers and consumers of data to better understand the patterns of ingestion,processing & serving and thus bring goodness of best practices to our data pipelines. We believe in complete ownership of streams so as subject matter expert of certain streams you will be looking after the necessary infrastructure requirements for your pipelines and are expected to own end-to-end deployment of the same. Being in this role means that you will own multiple processes which directly impact business and are critical for daily operations. Also work with petabytes of data and hundreds of complex data pipelines.

Job Responsibilities:

• Create and improve complex data processing pipelines, while keeping accuracy, reliability and cost in mind and deploy the same using best Continuous Delivery practices.
• Deploy models in real-time applications either as part of a micro service(HTTP-HyperText Transfer Protocol or RPC-Remote Procedure Call) with bounded context or as real-time pipelines producing events in response to user actions on ground.
• Maintain infrastructure as code using tools like terraform and comfortable with working in kubernetes.
• Build observability & alerting systems around data-pipelines.
• Identify, analyze tech / architecture shortcomings in existing systems and proactively discuss in architecture forums with proposed solutions.
• Being able to mentor, guide junior engineers for best engineering practices. Junior Developers will look up to you as mentors.

Job Requirements:

• Around 5+ years of experience. For transitioned data engineers over all experience of 5+ years in preferred.
• Experience in Hadoop Distributed File System(HDFS), Simple storage Service (S3), NoSql Databases and distributed platforms like Hadoop, Spark, Flink, Hive, Kafka, Oozie, Airflow, Elastic search etc.
• Experience in any of MapR, Cloudera, and Horton Works and/ or cloud based Hadoop Distributions (Google Cloud Platform preferred).
• Have a good understanding of data modeling, Extract, Transform and Load(ETL), distributed systems.
• Experience creating and building data centric applications involving Machine Learning(ML) models.
• Hands on experience with scheduling tools like Airflow, monitoring stack prometheus-grafana.
• Fundamental software engineering principle & data structure knowledge are must.

What's there for you?

In the Data team at Rapido you will get exposure to every stack possible – big data, software engineering(java/scala), ML-ops,data-ops and more.

• Be a part of a platform which serves hundreds of users with their data needs with state of the art Trino clusters.
• Get to experience real time applications deployed in Flink.
• Create and manage tools and frameworks built on top of open source technologies.
• Work with mammoth kubernetes clusters which host plethora of applications and tools.
• Hands on experience on open source big data tools and its best practices. Excited to solve changes?

Join Rapido & chase bigger milestones too!