Senior Data Engineer - (Real-Time Streaming)

GSS Group; Chennai District; On-site; INR 1,200,000 - 1,900,000; Full time

Job summary

GSS Group in Chennai, India seeks an experienced Senior Data Engineer to design and operate high‑performance real‑time streaming platforms. You will build and maintain data pipelines with Java, PySpark, Kafka, and Flink to support enterprise Business Platforms and Risk & Compliance initiatives.

The role focuses on event‑driven architectures, distributed data processing, and collaboration with architects, platform teams, and business stakeholders to ensure data quality and operational excellence.

Qualifications

• Designing and implementing real‑time data pipelines and event‑driven architectures.

Responsibilities

• Design, develop, and maintain scalable real-time data pipelines using Apache Kafka, Apache Flink, Java, and PySpark.
• Build high-throughput streaming applications for enterprise workloads and regulatory use cases.
• Collaborate with architects and stakeholders to deliver scalable data solutions.
• Monitor, troubleshoot, and optimize streaming jobs for performance and reliability.
• Create technical documentation and runbooks.

Skills

Java

Python

Real-Time Streaming

Data Engineering

Event-Driven Architecture

SQL

NoSQL databases

Education

Bachelor's or Master's Degree in CS/IT/Data Engineering or related field

Tools

Apache Kafka

Apache Flink

Kafka Streams

PySpark

Apache Spark

Job description

We are looking for an experiencedSenior Data Engineerwith strong expertise inReal-Time Streaming, Java, Apache Kafka, Apache Flink, and PySparkto build and operate high-performance, scalable streaming data platforms supporting enterprise Business Platforms and Risk & Compliance initiatives.

The ideal candidate will have extensive experience designing and implementing real-time data pipelines, event-driven architectures, and distributed data processing systems capable of handling high-volume, low-latency data workloads. This role requires strong engineering expertise in modern data platforms, stream processing, and Big Data technologies while collaborating closely with architects, platform teams, and business stakeholders.

Key Responsibilities

• Design, develop, and maintain scalable real-time data pipelines using Apache Kafka, Apache Flink, Java, and PySpark.
• Build high-throughput, low-latency streaming applications supporting enterprise-scale business and regulatory workloads.
• Develop event-driven data processing solutions using Kafka and modern streaming technologies.
• Design and implement distributed data processing pipelines for real-time analytics and operational reporting.
• Collaborate with Data Architects, Platform Engineering teams, Analytics teams, and Business stakeholders to understand requirements and deliver scalable data solutions.
• Develop reusable, secure, and maintainable data engineering components following enterprise engineering standards.
• Optimize streaming jobs for performance, scalability, fault tolerance, and reliability.
• Build and maintain batch and streaming ETL/ELT pipelines using PySpark.
• Monitor, troubleshoot, and resolve production issues within streaming data platforms.
• Participate in architecture discussions and contribute to the design of modern enterprise data platforms.
• Ensure data quality, governance, monitoring, logging, and operational excellence across streaming pipelines.
• Participate in Agile/Scrum ceremonies including sprint planning, backlog refinement, stand-ups, and retrospectives.
• Create technical documentation, implementation guides, and operational runbooks.

Required Technical Skills

Programming

• Strong hands-on experience withJava (Mandatory)
• Strong proficiency inPython
• Excellent understanding of Object-Oriented Programming (OOP) principles

Real-Time Streaming

• Apache Kafka
• Apache Flink
• Kafka Streams
• Event-Driven Architecture
• Stream Processing
• Real-Time Data Engineering

Big Data

• PySpark
• Apache Spark
• Distributed Data Processing
• Data Pipeline Development

Databases

• SQL
• Relational Databases
• NoSQL databases (preferred)

Data Engineering

• ETL/ELT Development
• Data Ingestion
• Data Transformation
• Data Integration
• Pipeline Orchestration
• Data Quality
• Performance Optimization

Development & DevOps

• Git
• CI/CD pipelines
• Linux environment
• API Integration
• Monitoring & Logging tools

Required Competencies

• Strong experience designing scalable distributed data systems.
• Excellent understanding of real-time streaming architectures.
• Ability to build secure, resilient, and fault-tolerant data pipelines.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management skills.
• Ability to work effectively in Agile delivery environments.
• Strong ownership mindset with focus on delivery quality and operational excellence.
• Ability to collaborate across cross-functional engineering and business teams.

Preferred Experience

• Experience working on enterprise-scale data platforms.
• Exposure to Risk & Compliance or Business Platform data ecosystems.
• Experience supporting mission-critical, high-volume production environments.
• Knowledge of cloud-based data engineering platforms is an advantage.
• Experience with modern data architecture and distributed systems.

Education

Bachelor's or Master's Degree in Computer Science, Information Technology, Data Engineering, or a related field (or equivalent industry experience).

Boolean Keywords (for better profile quality on job portals)

Mandatory Skills:

• Java
• Apache Kafka
• Apache Flink
• PySpark
• Real-Time Streaming
• Data Engineering
• Event-Driven Architecture