Data Engineering - Senior / Lead Engineer

Paytm; Dadri; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Paytm is looking for a Data Engineering – Technical Lead to work on complex technical projects in an innovative and fast-paced environment. This role requires strong product design sense and expertise in Hadoop and Spark technologies.

The ideal candidate will have 4 to 8 years of experience in Big Data, hands-on with tools like Hive and Spark, and will contribute by designing a real-time data ingestion pipeline.

Qualifications

• 4 to 8 years of experience in Big Data technologies.
• Strong hands-on experience with Hadoop, MapReduce, Hive, and Spark.
• Experience with scripting languages such as Python and Bash.

Responsibilities

• Work on complex technical projects and collaborate with peers.
• Grow analytics capabilities with reliable tools for big data.
• Design and develop a real-time events pipeline for data ingestion.

Skills

Hadoop

Spark

Python

Java

Scala

NoSQL databases (HBase, Cassandra)

SQL

AWS

Kafka

Redis

Job description

Data Engineering – Technical Lead

About the Role:

This position requires someone to work on complex technical projects and closely collaborate with peers in an innovative and fast-paced environment. The role expects a strong product design sense and specialized expertise in Hadoop and Spark technologies.

Requirements:

• 4 to 8 years of experience in Big Data technologies.
• Grow our analytics capabilities with faster, more reliable tools for handling petabytes of data every day.
• Brainstorm and create new platforms that can help make services available to cluster users in all shapes and forms, with low latency and horizontal scalability.
• Make changes to diagnose problems across the entire technical stack.
• Design and develop a real‑time events pipeline for data ingestion to support real‑time dash‑boarding, and develop complex, efficient functions to transform raw data sources into powerful, reliable components of our data lake.
• Design and implement new components and explore emerging technologies in the Hadoop ecosystem, ensuring successful execution of various projects.

Skills:

• Fluent with strong hands‑on experience with Hadoop, MapReduce, Hive, Spark, PySpark, etc.
• Excellent programming/debugging skills in Python, Java, and Scala.
• Experience with scripting languages such as Python, Bash, etc.
• Good to have experience working with NoSQL databases like HBase, Cassandra; hands‑on programming experience with multithreaded applications.
• Good to have experience with databases, SQL, and messaging queues such as Kafka.
• Good to have experience developing streaming applications such as Spark Streaming, Flink, Storm, etc.
• Good to have experience with AWS and cloud technologies such as S3.
• Experience with caching architectures such as Redis.