Data Engineering - Senior / Lead Engineer

One97 Communications Limited; Dadri; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

One97 Communications Limited is seeking a professional experienced in Big Data technologies to work on complex projects in a fast-paced environment. Ideal candidates will have 4 to 8 years of experience with Hadoop, Spark, and strong programming skills in Python and Java.

This role involves growing analytics capabilities, designing real-time data pipelines, and implementing new technologies in the Hadoop ecosystem.

Join us in democratizing credit for users across India!

Qualifications

• 4 to 8 years of experience in Big Data technologies.
• Strong hands-on experience with Hadoop, MapReduce, Hive, Spark, PySpark.
• Excellent programming/debugging skills in Python, Java, and Scala.

Responsibilities

• Grow analytics capabilities with faster, reliable tools handling petabytes of data.
• Create new platforms enabling low latency and horizontal scalability.
• Diagnose problems across the entire technical stack.
• Design and develop a real-time events pipeline for data ingestion.
• Implement new components in Hadoop Eco-System.

Skills

Hadoop

MapReduce

Hive

Spark

PySpark

Python

Java

Scala

Bash

HBase

Cassandra

Kafka

Redis

Job description

About Us: Paytm

Paytm is India’s leading digital payments and financial services company, which is focused on driving consumers and merchants to its platform by offering them a variety of payment use cases. To merchants, Paytm offers acquiring devices like Soundbox, EDC, QR and Payment Gateway where payment aggregation is done through PPI and also other banks’ financial instruments. To further enhance merchants’ business, Paytm offers merchants commerce services through advertising and Paytm Mini app store. Operating on this platform, the company then offers credit services such as merchant loans, personal loans and BNPL, sourced by its financial partners.

About the Role

This position requires someone to work on complex technical projects and closely work with peers in an innovative and fast-paced environment. For this role, we require someone with a strong product design sense & specialized in Hadoop and Spark technologies.

Requirements

• 4 to 8 years of experience in Big Data technologies.

Responsibilities

• Grow our analytics capabilities with faster, more reliable tools, handling petabytes of data every day.
• Brainstorm and create new platforms that can help make available to cluster users in all shapes and forms, with low latency and horizontal scalability.
• Make changes to our diagnosing any problems across the entire technical stack.
• Design and develop a real-time events pipeline for data ingestion for real-time dash-boarding. Develop complex and efficient functions to transform raw data sources into powerful, reliable components of our data lake.
• Design & implement new components and various emerging technologies in Hadoop Eco-System, and successful execution of various projects.

Skills

• Fluent with strong hands-on experience with Hadoop, MapReduce, Hive, Spark, PySpark, etc.
• Excellent programming/debugging skills in Python, Java, and Scala.
• Experience with any scripting language such as Python, Bash, etc.
• Good to have experience of working with noSQL databases like HBase, Cassandra.
• Hands-on programming experience with multithreaded applications.
• Good to have experience in Database, SQL, messaging queues like Kafka.
• Good to have experience in developing streaming applications e.g., Spark Streaming, Flink, Storm, etc.
• Good to have experience with AWS and cloud technologies such as S3.
• Experience with caching architectures like Redis, etc.

Compensation

If you are the right fit, we believe in creating wealth for you with enviable 500 mn+ registered users, 21 mn+ merchants and depth of data in our ecosystem, we are in a unique position to democratize credit for deserving consumers & merchants – and we are committed to it. India’s largest digital lending story is brewing here. It’s your opportunity to be a part of the story!