Kafka Developer

Innova Solutions

Pune City

On-site

INR 1,200,000 - 1,800,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

A global technology services partner is seeking a Kafka Developer to create real-time data ingestion pipelines in Pune, India. Responsibilities include developing Kafka producers and consumers, handling data transformation with Apache Spark, and ensuring data integrity throughout the pipeline. Ideal candidates will have strong experience with Apache Kafka as well as skills in Scala, Java, or Python. This is a full-time mid-senior level position offering competitive compensation.

Qualifications

Proven experience in developing Kafka producers and consumers.
Strong expertise in Apache Kafka and related technologies.
Experience with real-time data ingestion pipelines.

Responsibilities

Develop Kafka producers for data ingestion.
Build consumers to stream data into processing jobs.
Implement serialization, transformation, and validation logic.

Skills

Kafka producers

Kafka consumers

Apache Kafka

Apache Kafka Streams

Apache Spark

Scala

Java

Python

IPFIX

Avro

JSON

Protobuf

Docker

Kubernetes

Tools

Prometheus

Grafana

Splunk

HDFS

Hive

YARN

Knox

Job description

Get AI-powered advice on this job and more exclusive features.

Direct message the job poster from Innova Solutions

Senior Technical IT Recruiter @Innova Solutions -Hiring in IT Infrastructure, Software & Data Engineer in Belgium and Netherlands …

Keywords: Kafka, Apache Kafka, Kafka Connect, Kafka Streams, and Schema Registry.

Position Overview: We are looking for a Kafka Developer to design and implement real-time data ingestion pipelines using Apache Kafka. The role involves integrating with upstream flow record sources, transforming and validating data, and streaming it into a centralized data lake for analytics and operational intelligence

What you will Have:-

Responsibilities:

Key Responsibilities:

Develop Kafka producers to ingest flow records from upstream systems such as flow record exporters (e.g., IPFIX-compatible probes).
Build Kafka consumers to stream data into Spark Structured Streaming jobs and downstream data lakes.
Define and manage Kafka topic schemas using Avro and Schema Registry for schema evolution.
Implement message serialization, transformation, enrichment, and validation logic within the streaming pipeline.
Ensure exactly once processing, checkpointing, and fault tolerance in streaming jobs.
Integrate with downstream systems such as HDFS or Parquet-based data lakes, ensuring compatibility with ingestion standards.
Collaborate with Kafka administrators to align topic configurations, retention policies, and security protocols.
Participate in code reviews, unit testing, and performance tuning to ensure high-quality deliverables.
Document pipeline architecture, data flow logic, and operational procedures for handover and support.

Required Skills & Qualifications:

Proven experience in developing Kafka producers and consumers for real-time data ingestion pipelines.
Strong hands-on expertise in Apache Kafka, Kafka Connect, Kafka Streams, and Schema Registry.
Proficiency in Apache Spark (Structured Streaming) for real-time data transformation and enrichment.
Solid understanding of IPFIX, NetFlow, and network flow data formats; experience integrating with nProbe Cento is a plus.
Experience with Avro, JSON, or Protobuf for message serialization and schema evolution.
Familiarity with Cloudera Data Platform components such as HDFS, Hive, YARN, and Knox.
Experience integrating Kafka pipelines with data lakes or warehouses using Parquet or Delta formats.
Strong programming skills in Scala, Java, or Python for stream processing and data engineering tasks.
Knowledge of Kafka security protocols including TLS/SSL, Kerberos, and access control via Apache Ranger.
Experience with monitoring and logging tools such as Prometheus, Grafana, and Splunk.
Understanding of CI/CD pipelines, Git-based workflows, and containerization (Docker/Kubernetes)

A little about us:

Innova Solutions is a diverse and award-winning global technology services partner. We provide our clients with strategic technology, talent, and business transformation solutions, enabling them to be leaders in their field.

Founded in 1998, headquartered in Atlanta (Duluth), Georgia.
Employs over 50,000 professionals worldwide, with annual revenue approaching $3.0B.
Delivers strategic technology and business transformation solutions globally.
Operates through global delivery centers across North America, Asia, and Europe.
Provides services for data center migration and workload development for cloud service providers.

Awardee of prestigious recognitions including:

Women’s Choice Awards - Best Companies to Work for Women & Millennials, 2024
Forbes, America’s Best Temporary Staffing and Best Professional Recruiting Firms, 2023
American Best in Business, Globee Awards, Healthcare Vulnerability Technology Solutions, 2023
Global Health & Pharma, Best Full Service Workforce Lifecycle Management Enterprise, 2023
Received 3 SBU Leadership in Business Awards
Stevie International Business Awards, Denials Remediation Healthcare Technology Solutions, 2023

Seniority level