Data Engineer (Real-time Streaming)

NPCI International

Hyderabad

On-site

INR 1,800,000 - 2,400,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Medical insurance
Accidental insurance
Life insurance
Employee assistance
Wellness program
Parental leave
Relocation support
Mobile reimbursement
Car lease
Salary advances
Meal subsidy
Rewards program
IPR policy
Provident fund
Gratuity

Job summary

National Payments Corporation of India (NPCI) is seeking a data analytics professional to design and run real-time data pipelines. You will work with Apache Kafka, Spark or Flink, and data lake technologies to ensure 24x7 availability and scalable analytics across multiple sources.

Collaboration with Data Analytics, Product, and Technology teams is essential. You will develop and optimize ETL/ITL patterns, manage table formats, and implement orchestration using DBT, Dagster or Airflow, writing

Qualifications

  • Mandatory technical skills include Kafka and stream processing.
  • Experience with data lakes/lakehouses and table formats.
  • Strong SQL and data transformation expertise.
  • Experience in 24x7 data processing environments.

Responsibilities

  • Design and develop real-time data pipelines using Kafka and stream processing frameworks.
  • Ensure 24x7 data availability with fault-tolerant systems.
  • Implement ingestion, transformation, and load patterns for data lakes.
  • Work with table formats and SQL engines for analytics.
  • Develop orchestration workflows with DBT, Dagster, or Airflow.
  • Write code in Python, Scala, and Java for data processing.
  • Collaborate with cross-functional teams to deliver data solutions.
  • Monitor pipelines and ensure security, governance, and audit compliance.

Skills

Analytical thinking
Problem solving
24x7 availability

Education

BE/B.Tech in CS/IT

Tools

Apache Kafka
Spark Structured Streaming
Apache Flink
Iceberg
Hudi
Paimon
Trino
Hive
DBT
Dagster
Airflow
Python
Scala
Java
MongoDB
Cassandra
Redis
Jenkins
GitHub Actions
Linux
Superset
Tableau

Job description

National Payments Corporation of India (NPCI) | Full Time


The National Payments Corporation of India (NPCI) is a pivotal institution in India's digital payments ecosystem, established by the Reserve Bank of India (RBI) and the Indian Banks’ Association (IBA). It operates under the Payment and Settlement Systems Act, 2007, and is incorporated as a “Not for Profit” company under Section 25 of the Companies Act 1956 (now Section 8 of the Companies Act 2013). NPCI is dedicated to building world‑class digital payment infrastructure through innovative and efficient retail payment platforms. As an Equal Opportunity Employer, NPCI is committed to fostering an inclusive workplace culture, with zero tolerance for discrimination based on race, ethnicity, disability, gender identity, or sexual orientation, including support for the LGBTQ+ community.


To learn more about our company please click on link AboutNPCI


Our Culture

TheNPCIWAY –OurGuidingPrinciples


At NPCI, we foster a culture of Inclusion, Innovation, and a High- Performance Workplace .


The NPCIWAY isnotjustaframework - it’sasharedcommitmentbyevery individual to align with our evolving business needs, dynamic market conditions, and workforce expectations.


FiveTenetsoftheNPCIWAY


  • ForwardThinkingMind-Set

  • CustomerFirst

  • LeadWith Purpose

  • PassiontoDeliver

  • AllSucceedTogether


Employee Perks and Benefits

Insurance&Wellness:



  • ComprehensiveCoverage :Medical,accidental,andlifeinsurance.

  • Employee Assistance Program : Onsite medical center, financial/legal counselling, mental wellness.

  • AnnualWellnessBouquet :Holisticwellbeingsupport


Family &Childcare


  • ParentalLeave :Inclusivematernityandpaternityleave.

  • Nanny at home Support: Assistance for Working Parents


Mobility&Lifestyle


  • MobilityBenefits :Relocation,transfers,andtravelsupport.

  • HomeLoanInterestSupport :Upto50%reimbursement*

  • MobileReimbursement :Newhandsetevery2years.

  • CarLease,SalaryAdvance,SubsidizedMeals.

  • Reward&RecognitionProgram.

  • TechnologyInnovation: PatentsandIPRightspolicy.


RetirementPlanning


  • ProvidentFund(PF) and Gratuity Contributions .


Why Join Us

At NPCI, you’ll be part of a purpose-driven organization shaping the future of digital payments in India and beyond. NPCI offers a unique opportunity to work on cutting‑edge projects that directly impact millions. We foster a culture of innovation, inclusion, and high performance, where every individual is empowered to lead with purpose and deliver with passion. With a strong focus on employee wellbeing, continuous learning, and collaborative success, NPCI is more than just a workplace - it’s a platform to grow, contribute, and make a meaningful difference.


To design and implement real-time data streaming pipelines ensuring 24×7 availability of high-volume transactional data. Your role includes building fault‑tolerant, scalable systems using modern big data technologies, integrating data from multiple sources into data lakes/lakehouses, and enabling downstream analytics. You will collaborate with internal teams (Data Science, Product, and Technology) to deliver reliable data solutions, optimize performance, and maintain compliance with NPCI standards.



  • Division: Data Analytics

  • Years of Experience: 3–8 years

  • Education: Graduation in Computer Science/IT (preferably BE/B.Tech) or equivalent; advanced degrees are a plus

  • Employment Type: Full-time, Permanent


Key Responsibilities


  • Design and develop real-time data pipelines using Apache Kafka and stream processing frameworks (Spark Structured Streaming / Apache Flink).

  • Ensure 24×7 data availability with fault‑tolerant, highly reliable systems.

  • Implement ingestion, transformation, and load (ITL) patterns for data lakes/lakehouses (S3/MinIO, HDFS).

  • Work with table formats like Iceberg, Hudi, or Paimon for ACID transactions and schema evolution.

  • Optimize SQL queries on Trino/Hive for large‑scale analytics.

  • Develop orchestration workflows using DBT, Dagster, or Airflow for data transformations.

  • Write efficient code in Python, Scala, and Java for data processing and automation.

  • Collaborate with cross‑functional teams to understand requirements and deliver high‑quality data solutions.

  • Monitor pipeline health, manage checkpoints, and implement observability for streaming jobs.

  • Ensure compliance with security, governance, and audit standards.


Requirements

Key Skills and Experience Required


  • Mandatory Technical Skills:

    • Apache Kafka (topics, partitions, offsets, reliability)

    • Stream processing (Spark Structured Streaming or Apache Flink)



  • Preferred Technical Skills:

    • Data Lake / Lakehouse (S3, MinIO, HDFS)

    • Table formats: Iceberg, Hudi, Paimon

    • SQL engines: Trino, Hive

    • Orchestration tools: DBT, Dagster, Airflow

    • Programming: Python, Scala, Java

    • NoSQL databases (MongoDB, Cassandra, Redis)

    • CI/CD (Jenkins, GitHub Actions), Linux basics

    • BI tools: Superset, Tableau

    • Data Quality frameworks and observability practices



  • Other Requirements:

    • Strong analytical and problem-solving skills

    • Ability to work in 24×7 environments

    • Quick adaptability to new technologies



Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Associate Application Development, NBBL
Senior Associate Application Development, NBBL

National Payments Corporation of India • Chennai District

On-site
INR 1,200,000 - 2,100,000
Comprehensive Coverage
Employee Assistance Program
Annual Wellness Bouquet
+4
Data Engineer (Apache Spark or Flink)
Data Engineer (Apache Spark or Flink)

National Payments Corporation of India (NPCI) • Hyderabad

On-site
INR 1,500,000 - 3,000,000
Senior Associate Database
Senior Associate Database

National Payments Corporation of India • Hyderabad

On-site
INR 700,000 - 1,000,000
Comprehensive Medical Coverage
Employee Assistance Program
Annual Wellness Bouquet
+5
Data Engineer (Apache Spark or Flink)
Data Engineer (Apache Spark or Flink)

Mployee.me • Hyderabad

On-site
INR 1,200,000 - 2,400,000
Backend Engineer (NodeJS)
Backend Engineer (NodeJS)

NPCI International • Mumbai

On-site
INR 1,200,000 - 1,800,000
Comprehensive Medical Insurance
Employee Assistance Program
Parental Leave
+2
Backend Engineer (NodeJS)
Backend Engineer (NodeJS)

NPCI • Mumbai

On-site
INR 1,500,000 - 2,100,000
Senior Associate DBA
Senior Associate DBA

National Payments Corporation of India • Chennai District

On-site
INR 1,200,000 - 1,800,000
Comprehensive medical, accidental, and life insurance
Employee Assistance Program
Parental Leave
+3
Senior Associate Customer Success
Senior Associate Customer Success

NPCI • Mumbai

Hybrid
INR 1,800,000 - 2,400,000
Insurance & Wellness
Annual Wellness Bouquet
Mobility & Relocation support
+1
Lead Data Scientist
Lead Data Scientist

NPCI International • Hyderabad

On-site
INR 1,200,000 - 2,000,000
Senior Lead - Application Development (AePS, IMPS & NETC)
Senior Lead - Application Development (AePS, IMPS & NETC)

National Payments Corporation of India • Mumbai

Hybrid
INR 1,200,000 - 1,800,000
Comprehensive medical insurance
Family and childcare support
Mobility and lifestyle benefits