Sr. Data Engineer - (RealTime Streaming)

Global Software Solutions Group; Bengaluru; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Global Software Solutions Group in Bengaluru is seeking an experienced Senior Data Engineer to design and manage real-time streaming platforms using Kafka, Flink, Java and PySpark. You will build high-throughput pipelines and ensure scalable, fault-tolerant data infrastructure across enterprise environments.

The role focuses on cloud-native data solutions, security, DR readiness, and IaC automation with Terraform, GitOps practices, and advanced monitoring.

Qualifications

• Experience designing, building, and managing scalable real-time streaming platforms using Kafka, Flink, Java, and PySpark.
• Develop high-throughput, low-latency data pipelines and manage Kafka clusters.
• Implement cloud-native data solutions with a focus on reliability, security, and scalability.
• Set up Kafka clusters, perform tuning, monitoring, and DR strategies.
• Follow IaC with Terraform and GitOps for deployment and infrastructure management.
• Apply strong security practices including SSL/mTLS, SASL, and ACL management.

Responsibilities

• Design, develop, and manage Kafka-based real-time streaming architectures for high-volume workloads.
• Build scalable streaming pipelines using Kafka Connect, ksqlDB, and Schema Registry.
• Develop and optimize real-time processing with Flink, Java, and PySpark.
• Administer Kafka clusters and ensure high availability with DR strategies.
• Collaborate with cross-functional teams to deliver real-time data solutions.

Skills

Apache Kafka

Apache Flink

Java

PySpark

Real-Time Streaming Architecture

Kafka Connect

ksqlDB

Schema Registry

Streaming Data Pipelines

Cloud-Native Applications

Terraform

Infrastructure-as-Code (IaC)

GitOps

High Availability & Disaster Recovery

Performance Tuning & Optimization

Fault Tolerance & Scalability

SSL / mTLS

SASL

ACLs

Job description

We are looking for an experienced Senior Data Engineer with strong expertise in real-time streaming technologies and large-scale data engineering solutions. The ideal candidate will have hands-on experience designing, building, and managing highly scalable and fault-tolerant streaming data platforms using Kafka, Flink, Java, and PySpark.

The candidate will be responsible for developing high-throughput, low-latency data pipelines, managing Kafka clusters, implementing cloud-native data solutions, and ensuring system reliability, security, and scalability in enterprise environments.

Requirements

Key Responsibilities

• Design, develop, implement, and manage Kafka-based real-time streaming architectures capable of handling high-volume and low-latency workloads


• Build and maintain scalable streaming data pipelines using Kafka ecosystem components, including:
  
  
  
  • Kafka Connect
  
  
  • ksqlDB
  
  
  • Schema Registry
  
  
  
  


• Develop and optimize real-time data processing applications using:
  
  
  
  • Apache Flink
  
  
  • Java
  
  
  • PySpark
  
  
  
  


• Perform Kafka cluster setup, administration, configuration, tuning, monitoring, and performance optimization


• Ensure Kafka clusters are highly available and implement disaster recovery strategies and best practices


• Design and implement fault-tolerant, scalable, and resilient streaming systems for enterprise-grade applications


• Work with cloud-native applications and modern data engineering frameworks to build scalable solutions


• Automate infrastructure provisioning and deployment using Infrastructure-as-Code (IaC) tools such as Terraform


• Implement and follow GitOps practices for deployment automation and infrastructure management


• Apply robust security standards and best practices, including:
  
  
  
  • SSL / mTLS
  
  
  • SASL authentication
  
  
  • ACL management
  
  
  
  


• Collaborate with cross-functional teams to deliver real-time data engineering solutions aligned with business and analytics requirements

Required Skills & Technologies

Mandatory Skills

• Apache Kafka


• Apache Flink


• Java


• PySpark


• Real-Time Streaming Architecture


• Kafka Cluster Management


• Kafka Connect


• ksqlDB


• Schema Registry


• Streaming Data Pipelines


• Cloud-Native Applications


• Terraform


• Infrastructure-as-Code (IaC)


• GitOps


• High Availability & Disaster Recovery


• Performance Tuning & Optimization


• Fault Tolerance & Scalability


• SSL / mTLS


• SASL


• ACLs

Preferred Experience

• Strong experience in building enterprise-grade real-time data platforms


• Hands-on experience working in high-throughput and low-latency environments


• Experience with large-scale distributed data processing systems


• Exposure to cloud platforms and modern DevOps practices will be an added advantage