Senior Associate Data Engineering

NPCI; Hyderabad; On-site; INR 1,800,000 - 3,200,000; Full time

Benefits offered by this job

Comprehensive Coverage

Employee Assistance Program

Parental Leave

Nanny at home Support

Mobility Benefits

Home Loan Interest Support

Mobile Reimbursement

Car Lease

Salary Advance

Subsidized Meals

Reward & Recognition Program

IPR policy

Job summary

NPCI is seeking a Data Analytics professional to design and implement real-time data pipelines using Apache Kafka and Spark/Flink. You will collaborate with Data Science, Product, and Technology teams to deliver scalable data solutions.

The role emphasizes 24x7 data availability, ingestion and processing into data lakes/lakehouses, and using Iceberg/Hudi/Paimon, Trino/Hive, and DBT/Airflow for orchestration. 3–8 years of experience and BE/B.Tech are preferred.

Qualifications

• Bachelor's degree in Computer Science/IT or equivalent; BE/B.Tech preferred.
• 3–8 years of experience in data engineering or analytics.
• Experience with real-time data pipelines and big data tech is required.
• Strong knowledge of Kafka, Spark/Flink, SQL, and data lakes.

Responsibilities

• Design and develop real-time data pipelines using Apache Kafka and streaming frameworks (Spark/Flink).
• Ensure 24×7 data availability with fault-tolerant, reliable systems.
• Implement ingestion, transformation, and load patterns for data lakes/lakehouses (S3/MinIO/HDFS).
• Work with table formats like Iceberg, Hudi, or Paimon for ACID and schema evolution.
• Optimize SQL queries on Trino/Hive for large-scale analytics.
• Develop orchestration workflows with DBT, Dagster, or Airflow for data transformations.
• Write Python, Scala, and Java for data processing and automation.
• Collaborate with cross-functional teams to deliver high-quality data solutions.
• Monitor pipeline health, checkpoints, and observability; ensure security and governance.

Skills

Apache Kafka

Spark Streaming

Apache Flink

Python

Java

SQL

NoSQL

Linux basics

Git/Jenkins

Education

BE/B.Tech in CS/IT

Tools

DBT

Dagster

Airflow

Hive

Trino

Job description

National Payments Corporation of India (NPCI) | Full Time

The National Payments Corporation of India (NPCI) is a pivotal institution in India's digital payments ecosystem, established by the Reserve Bank of India (RBI) and the Indian Banks’ Association (IBA). It operates under the Payment and Settlement Systems Act, 2007, and is incorporated as a “Not for Profit” company under Section 25 of the Companies Act 1956 (now Section 8 of the Companies Act 2013). NPCI is dedicated to building world-class digital payment infrastructure through innovative and efficient retail payment platforms. As an Equal Opportunity Employer, NPCI is committed to fostering an inclusive workplace culture, with zero tolerance for discrimination based on race, ethnicity, disability, gender identity, or sexual orientation, including support for the LGBTQ+ community.

To learn more about our company please click on link AboutNPCI

Our Culture

TheNPCIWAY –OurGuidingPrinciples

At NPCI, we foster a culture of Inclusion, Innovation, and a High- Performance Workplace .

The NPCIWAY isnotjustaframework - it’sasharedcommitmentbyevery individual to align with our evolving business needs, dynamic market conditions, and workforce expectations.

FiveTenetsoftheNPCIWAY

• ForwardThinkingMind-Set
• CustomerFirst
• LeadWith Purpose
• PassiontoDeliver
• AllSucceedTogether

Employee Perks and Benefits

Insurance & Wellness:

• ComprehensiveCoverage :Medical,accidental,andlifeinsurance.
• Employee Assistance Program : Onsite medical center, financial/legal counselling, mental wellness.
• AnnualWellnessBouquet :Holisticwellbeingsupport

Family & Childcare:

• ParentalLeave :Inclusivematernityandpaternityleave.
• Nanny at home Support: Assistance for Working Parents

Mobility & Lifestyle:

• MobilityBenefits :Relocation,transfers,andtravelsupport.
• HomeLoanInterestSupport :Upto50%reimbursement*
• MobileReimbursement :Newhandsetevery2years.
• CarLease,SalaryAdvance,SubsidizedMeals.
• Reward&RecognitionProgram.
• TechnologyInnovation: PatentsandIPRightspolicy.

Retirement Planning:

• ProvidentFund(PF) and Gratuity Contributions .

Why Join Us

At NPCI, you’ll be part of a purpose-driven organization shaping the future of digital payments in India and beyond. NPCI offers a unique opportunity to work on cutting-edge projects that directly impact millions. We foster a culture of innovation, inclusion, and high performance, where every individual is empowered to lead with purpose and deliver with passion. With a strong focus on employee wellbeing, continuous learning, and collaborative success, NPCI is more than just a workplace - it’s a platform to grow, contribute, and make a meaningful difference.

To design and implement real-time data streaming pipelines ensuring 24×7 availability of high-volume transactional data. Your role includes building fault-tolerant, scalable systems using modern big data technologies, integrating data from multiple sources into data lakes/lakehouses, and enabling downstream analytics. You will collaborate with internal teams (Data Science, Product, and Technology) to deliver reliable data solutions, optimize performance, and maintain compliance with NPCI standards.

• Division: Data Analytics
• Years of Experience: 3–8 years
• Education: Graduation in Computer Science/IT (preferably BE/B.Tech) or equivalent; advanced degrees are a plus
• Employment Type: Full-time, Permanent

Key Responsibilities

• Design and develop real-time data pipelines using Apache Kafka and stream processing frameworks (Spark Structured Streaming / Apache Flink).
• Ensure 24×7 data availability with fault-tolerant, highly reliable systems.
• Implement ingestion, transformation, and load (ITL) patterns for data lakes/lakehouses (S3/MinIO, HDFS).
• Work with table formats like Iceberg, Hudi, or Paimon for ACID transactions and schema evolution.
• Optimize SQL queries on Trino/Hive for large-scale analytics.
• Develop orchestration workflows using DBT, Dagster, or Airflow for data transformations.
• Write efficient code in Python, Scala, and Java for data processing and automation.
• Collaborate with cross-functional teams to understand requirements and deliver high-quality data solutions.
• Monitor pipeline health, manage checkpoints, and implement observability for streaming jobs.
• Ensure compliance with security, governance, and audit standards.

Key Skills and Experience Required

• Mandatory Technical Skills:
  • Apache Kafka (topics, partitions, offsets, reliability)
  • Stream processing (Spark Structured Streaming or Apache Flink)
• Preferred Technical Skills:
  • Data Lake / Lakehouse (S3, MinIO, HDFS)
  • Table formats: Iceberg, Hudi, Paimon
  • SQL engines: Trino, Hive
  • Orchestration tools: DBT, Dagster, Airflow
  • Programming: Python, Scala, Java
  • NoSQL databases (MongoDB, Cassandra, Redis)
  • CI/CD (Jenkins, GitHub Actions), Linux basics
  • BI tools: Superset, Tableau
  • Data Quality frameworks and observability practices
• Other Requirements:
  • Strong analytical and problem-solving skills
  • Ability to work in 24×7 environments
  • Quick adaptability to new technologies