Data Engineer – Real-Time Streaming (Flink / Java / Kafka)

D4 Insight; Bengaluru; On-site; INR 900,000 - 1,300,000; Full time

Job summary

D4 Insight in Bengaluru is seeking an experienced Data Engineer to design, implement, and manage real-time streaming solutions using Kafka, Flink, and Java. You will build scalable, event-driven architectures and manage cloud-native infrastructure to support enterprise-scale digital banking applications.

Collaborate with Data Engineering, DevOps, Cloud, and Product teams; troubleshoot production issues; promote best practices; and participate in Agile delivery and continuous improvement

Qualifications

• Experience with real-time streaming systems and modern data pipelines.
• Hands-on use of Kafka, Flink, and Java in production environments.
• Knowledge of cloud-native infrastructure, security, and governance.

Responsibilities

• Design, develop, and manage Kafka-based real-time streaming applications for high-throughput, low-latency environments.
• Build scalable streaming data pipelines using Kafka Connect, ksqlDB, and Schema Registry.
• Develop distributed streaming applications using Apache Flink and Java.
• Optimize streaming architecture for performance, reliability, and scalability.

Skills

Java

Kafka Connect

ksqlDB

Schema Registry

Terraform

GitOps

SSL/mTLS, SASL & ACLs

Tools

Docker

Job description

Role Overview:

We are looking for experienced Data Engineer to design, implement, and manage high-performance real-time streaming solutions using Apache Kafka, Apache Flink, and Java. The role focuses on building scalable event-driven architectures, managing cloud-native infrastructure, and ensuring secure, fault-tolerant streaming platforms that support enterprise-scale digital banking applications.

Key Responsibilities:

• Design, develop, and manage Kafka-based real-time streaming applications for high-throughput, low-latency environments.
• Build scalable streaming data pipelines using Kafka Connect, ksqlDB, and Schema Registry.
• Develop distributed streaming applications using Apache Flink and Java.
• Optimize streaming architecture for performance, reliability, and scalability.

Platform & Infrastructure

• Configure, administer, and optimize Kafka clusters for high availability and disaster recovery.
• Implement GitOps practices for deployment and configuration management.
• Ensure robust platform security using SSL/mTLS, SASL, and ACLs.
• Work closely with Data Engineering, DevOps, Cloud, and Product teams.
• Troubleshoot production issues and optimize streaming performance.
• Maintain documentation and promote engineering best practices.
• Participate in Agile delivery, sprint planning, and continuous improvement initiatives.

Required Skills:

• Java
• Kafka Connect
• ksqlDB
• Schema Registry
• Terraform
• GitOps
• SSL/mTLS, SASL & ACLs

Preferred Skills:

• Docker
• Banking or Financial Services Experience

We craft, deploy, and manage bespoke services in CRM, data and AI, cybersecurity and consulting.