Kafka Engineer

3i Infotech; Thane; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

3i Infotech is seeking a Kafka Engineer to design, develop, and optimize high-throughput, low-latency data platforms. The role requires hands-on expertise with Apache Kafka, Redis, and MongoDB, and a solid understanding of real-time event streaming and distributed caching.

You will collaborate with developers and DevOps to build scalable, resilient data solutions, optimize data flows, and ensure reliable message delivery and schema evolution across systems.

Qualifications

• 25 years of backend or data platform engineering experience
• Hands-on experience with Apache Kafka, Redis and MongoDB
• Strong understanding of distributed systems, event-driven architecture and data streaming
• Knowledge of caching strategies and NoSQL database design
• Experience with Kafka consumer groups, partitions, schema registry and message delivery semantics
• Familiarity with Redis cluster, Sentinel, Pub/Sub and advanced Redis data structures
• Understanding of Linux environments, REST APIs and monitoring tools
• Experience with Git and Agile development methodologies

Responsibilities

• Design, implement and maintain high-throughput Kafka topics, partitions and consumer groups.
• Configure Kafka producers and consumers for optimal performance and reliability.
• Ensure message delivery guarantees and manage schema evolution using Confluent Schema Registry.
• Monitor and troubleshoot Kafka clusters, consumer lag and broker performance.
• Design and implement efficient caching strategies using Redis to reduce DB load.
• Utilize Redis data structures (Hashes, Lists, Sets, Sorted Sets, HyperLogLogs, Geospatial indexes).
• Configure Redis clusters, Sentinel deployments, Pub/Sub and persistence mechanisms.
• Optimize cache performance and resolve cache-related issues.
• Design, implement and optimize MongoDB databases including replica sets and sharded clusters.
• Develop efficient BSON document models and create optimized indexes.
• Develop complex aggregation pipelines and optimize queries.
• Profile and optimize end-to-end data flows across Kafka, Redis and MongoDB.
• Identify and resolve bottlenecks including slow queries and storage inefficiencies.
• Participate in modernization and performance improvement initiatives.

Skills

Distributed Systems

Event-Driven Architecture

Data Streaming

Caching Strategies

NoSQL Database Design

Git

Agile development methodologies

Tools

Apache Kafka

Redis

MongoDB

Job description

Job Summary

We are looking for a Kafka Engineer with 2 to 5 years of experience to design, develop, and optimize high-throughput, low-latency data platforms. The ideal candidate will have hands‑on expertise in Apache Kafka, Redis, and MongoDB, with a strong understanding of real‑time event streaming, distributed caching, NoSQL databases, and performance optimization. You will work closely with application developers, DevOps engineers, and cross‑functional teams to build scalable and resilient data solutions.

Key Responsibilities

Apache Kafka (Event Streaming & Messaging)

• Design, implement, and maintain high-throughput Kafka topics, partitions, and consumer groups.
• Configure Kafka producers and consumers for optimal performance and reliability.
• Ensure message delivery guarantees and manage schema evolution using Confluent Schema Registry.
• Monitor and troubleshoot Kafka clusters, consumer lag, and broker performance.

Redis (Distributed Caching & In-Memory Data Store)

• Design and implement efficient caching strategies to improve application performance and reduce database load.
• Utilize Redis data structures such as Hashes, Lists, Sets, Sorted Sets, HyperLogLogs, and Geospatial indexes.
• Configure and manage Redis Clusters, Sentinel deployments, Pub/Sub messaging, and persistence mechanisms.
• Optimize cache performance and resolve issues such as cache stampedes and memory fragmentation.

MongoDB (NoSQL Database)

• Design, implement, and optimize MongoDB databases, including Replica Sets and Sharded Clusters.
• Develop efficient BSON document models while following MongoDB best practices.
• Build optimized indexes (Compound, Partial, and Text Indexes) to improve query performance.
• Develop complex aggregation pipelines and perform query optimization.

Performance Optimization & Modernization

• Profile and optimize end‑to‑end data flows across Kafka, Redis, and MongoDB.
• Identify and resolve system bottlenecks including slow queries, consumer lag, caching issues, and storage inefficiencies.
• Implement modern data engineering best practices for scalable and reliable distributed systems.
• Participate in database modernization and system optimization initiatives.

Required Skills

Technical Skills

• 25 years of experience in backend or data platform engineering.
• Strong hands‑on experience with:
• Apache Kafka
• Redis
• MongoDB
• Good understanding of:
• Distributed Systems
• Event‑Driven Architecture
• Data Streaming
• Caching Strategies
• NoSQL Database Design
• Experience with Kafka Consumer Groups, Partitions, Schema Registry, and Message Delivery semantics.
• Knowledge of MongoDB Aggregation Framework, Indexing, Replication, and Sharding.
• Familiarity with Redis Cluster, Sentinel, Pub/Sub, and advanced Redis data structures.
• Understanding of Linux environments, REST APIs, and monitoring tools.
• Experience with Git and Agile development methodologies.

Preferred Qualifications

• Exposure to enterprise‑scale database modernization or migration projects.
• Knowledge of Docker, Kubernetes, CI/CD pipelines, or cloud platforms (AWS, Azure, or GCP) is an added advantage.
• Familiarity with monitoring and observability tools such as Prometheus, Grafana, or ELK Stack.