Data Engineer (Apache Spark or Flink)

Mployee.me

Hyderabad

On-site

INR 1,200,000 - 2,400,000

Full time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Mployee.me is hiring a Data Engineer (Real Time Streaming) in Hyderabad to design and implement real-time data pipelines. You will build fault-tolerant systems, ingest and transform data into data lakes or lakehouses, and enable downstream analytics.

Collaboration with Data Science, Product and Tech teams is essential to ensure compliant, scalable data solutions. You will work with technologies like Apache Kafka, Spark/Flink, Iceberg, Hudi, and Trino/Hive, while scripting in Python/Scala/Java.

Qualifications

  • 3–8 years of experience in data engineering or similar field.
  • Hands-on with real-time streaming data pipelines and big data stacks.
  • Experience designing fault-tolerant, scalable data systems in 24/7 environments.

Responsibilities

  • Design and implement real-time data pipelines using Kafka and stream frameworks.
  • Ensure continuous data availability with fault-tolerant systems.
  • Build data lake/lakehouse ingestion, transformation, and load patterns.
  • Work with Iceberg/Hudi/Paimon for ACID transactions and schema evolution.
  • Optimize SQL on Trino/Hive for large-scale analytics.
  • Develop orchestration workflows with DBT, Dagster, or Airflow.
  • Write code in Python, Scala, and Java for data processing and automation.
  • Collaborate with cross-functional teams to deliver reliable data solutions.
  • Monitor pipelines, checkpoints, and observability for streaming jobs.
  • Ensure security, governance, and audit compliance.

Skills

Kafka
Spark
Flink
SQL
Python
Scala
Java
CI/CD
Linux
Observability
Data modeling

Education

Bachelor's degree in CS/IT or BE/B.Tech

Tools

S3/MinIO
HDFS
Iceberg
Hudi
Paimon
Trino
Hive
Airflow
DBT
Dagster
Jenkins
GitHub Actions
Linux

Job description

The Opportunity

To design and implement real-time data streaming pipelines ensuring 247 availability of high-volume transactional data. Your role includes building fault-tolerant, scalable systems using modern big data technologies, integrating data from multiple sources into data lakes/lakehouses, and enabling downstream analytics. You will collaborate with internal teams (Data Science, Product, and Technology) to deliver reliable data solutions, optimize performance, and maintain compliance with NPCI standards.

Job Details
  • Job Title: Data Engineer (Real Time Streaming)
  • Division: Data Analytics
  • Years of Experience: 3 to 8 years
  • Education: Graduation in Computer Science/IT (preferably BE/B.Tech) or equivalent; advanced degrees are a plus
  • Employment Type: Full-time, Permanent
  • Location: Hyderabad(or NPCI offices as applicable)
Key Responsibilities
  • Strong Design and develop real-time data pipelines using Apache Kafka and stream processing frameworks (Spark Structured Streaming / Apache Flink).
  • Ensure 247 data availability with fault-tolerant, highly reliable systems.
  • Implement ingestion, transformation, and load (ITL) patterns for data lakes/lakehouses (S3/MinIO, HDFS).
  • Work with table formats like Iceberg, Hudi, or Paimon for ACID transactions and schema evolution.
  • Optimize SQL queries on Trino/Hive for large-scale analytics.
  • Develop orchestration workflows using DBT, Dagster, or Airflow for data transformations.
  • Write efficient code in Python, Scala, and Java for data processing and automation.
  • Collaborate with cross-functional teams to understand requirements and deliver high-quality data solutions.
  • Monitor pipeline health, manage checkpoints, and implement observability for streaming jobs.
  • Ensure compliance with security, governance, and audit standards.
Key Skills and Experience Required
  • Mandatory Technical Skills:
    • Apache Kafka (topics, partitions, offsets, reliability)
    • Stream processing (Spark Structured Streaming or Apache Flink)
  • Preferred Technical Skills:
    • Data Lake / Lakehouse (S3, MinIO, HDFS)
    • Table formats: Iceberg, Hudi, Paimon
    • SQL engines: Trino, Hive
    • Orchestration tools: DBT, Dagster, Airflow
    • Programming: Python, Scala, Java
    • NoSQL databases (MongoDB, Cassandra, Redis)
    • CI/CD (Jenkins, GitHub Actions), Linux basics
    • BI tools: Superset, Tableau
    • Data Quality frameworks and observability practices
  • Other Requirements:
    • Strong analytical and problem-solving skills
    • Ability to work in 247 environments
    • Quick adaptability to new technologies
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer (Apache Spark or Flink)
Data Engineer (Apache Spark or Flink)

National Payments Corporation of India (NPCI) • Hyderabad

On-site
INR 1,500,000 - 3,000,000
Data Engineer (Real-time Streaming)
Data Engineer (Real-time Streaming)

NPCI International • Hyderabad

On-site
INR 1,800,000 - 2,400,000
Medical insurance
Accidental insurance
Life insurance
+12
Data Engineer – Real-Time Streaming (Flink / Java / Kafka)
Data Engineer – Real-Time Streaming (Flink / Java / Kafka)

D4 Insight • Bengaluru

On-site
INR 900,000 - 1,300,000
Lead Data Engineer – Streaming Data
Lead Data Engineer – Streaming Data

Navikenz • Bengaluru

On-site
INR 1,800,000 - 3,200,000
Data Engineering Lead
Data Engineering Lead

Altimetrik • Mumbai

On-site
INR 4,000,000 - 7,000,000
Data Streaming Specialist
Data Streaming Specialist

Navikenz India • Bengaluru

Hybrid
INR 1,400,000 - 2,800,000
Lead Data Engineer
Lead Data Engineer

Navikenz • Bengaluru

On-site
INR 1,200,000 - 1,800,000
Senior Data Engineer
Senior Data Engineer

Hiring Hub • Mohali

On-site
INR 2,000,000 - 3,500,000
Data Engineer
Data Engineer

Comviva • Gurugram District

On-site
INR 1,200,000 - 2,000,000
Data Engineer 2/3
Data Engineer 2/3

Manatal • Bengaluru

On-site
INR 1,200,000 - 1,800,000