Kafka + Java + Data Engineer

Cognizant; Bengaluru Urban; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Cognizant is seeking an experienced data streaming engineer to design and build real-time pipelines using Kafka and the Hadoop ecosystem. The role emphasizes scalable, fault-tolerant architectures and strong debugging of production issues.

Expected involvement includes design leadership, hands-on coding, and mentoring. Candidates should be proficient with Databricks, Azure, Kubernetes, Docker, and CI/CD workflows, and comfortable in Agile environments.

Qualifications

• Extensive experience designing and developing streaming pipelines for high-volume data processing.
• Strong expertise in Java, Kafka and Hadoop ecosystem (Spark, Hive, YARN, HDFS).
• Experience with Linux on-prem, CI/CD tools (Jenkins, GitHub).
• Proficient in Kafka-based event-driven architectures and real-time processing.
• Ability to debug and fix post-production issues in complex data pipelines.

Responsibilities

• Design and implement real-time streaming pipelines using Kafka.
• Develop producers and consumers for high-throughput, fault-tolerant systems.
• Debug and fix post-production issues for developed code.
• Lead design/architecture and provide coding guidance to team (70/30 split).
• Work with Databricks, Azure Cloud services, Kubernetes, GitHub, Docker; CI/CD workflows.

Skills

Java

Kafka

Hadoop ecosystem

Scala

CI/CD

Linux

Tools

Spark

Hive

YARN

HDFS

Databricks

Azure

Kubernetes

GitHub

Docker

PySpark

Polars

Pandas

Job description

• Extensive experience designing and developing streaming pipelines, data platform to handle high volume data processing, distributed system for both batch and realtime
• Strong Expertise in– Java, Kafka and Hadoop ecosystem (Spark, Hive, YARN, HDFS), Scala, Cassandra, Linux (on prem), CI-CD (Jenkins, Github), Obs (Dynatrace).
• Design and implement real-time streaming pipelines using Kafka
• Develop producers and consumers for high-throughput, fault-tolerant systems
• Responsible for both development and strong expertise to debug and fix any post production issues
• Implement Kafka-based event-driven architectures
• Experience with Databricks, Azure Cloud PaaS Services, Azure Data Factory, ADLS, Kubernetes, GitHub, Docker, and CI/CD workflows
• Experience in scripting, ETL (Pandas, Pola.rs, PySpark, Ibis), SQL, and NoSQL databases (Cassandra, MongoDB)
• Understanding of system architecture concepts: distributed systems, caching, replication, data modeling
• Leads : Role is 70% of time in Design, Architect, Coding and 30% of time in team mentoring, guidance, coordination
• Responsible for both development and strong expertise to debug and fix any post production issues for their developed code
• Interview ask : To write optimal code (in Java/Kafka/Python/Pyspark) in timeboxed period for scenario given (covering data quality, transformation) without AI tool and code to be accurate. To resolve code issue for a bug introduced in sample code
• Excellent written and verbal communication skills
• Experience working in Agile environments; familiarity with cloud platforms
• Experience with orchestration tools (e.g. NiFi, Griffin, Hamilton, Airflow)
• Functional experience of Retail Banking, capital markets, securities processing will be advantage