Real-Time Kafka Engineer — Scalable Data Pipelines

Staffing Spot, Inc.; New York (NY); On-site; USD 120,000 - 150,000; Full time

Job summary

Staffing Spot, Inc. is seeking a Kafka Engineer in New York, NY to design, develop, implement, and support scalable event-streaming pipelines using Apache Kafka and related technologies.

You will build producers/consumers, manage topics and partitions, and ensure high availability across distributed systems. The role requires deep knowledge of Kafka, Connect, Streams, and cloud platforms, plus strong scripting and software development in Java, Python, or Scala.

Qualifications

• Strong knowledge of Apache Kafka.
• Kafka Producers and Consumers.
• Kafka Topics, Partitions, Offsets, Consumer Groups, and Replication.
• Kafka Connect for source and sink integrations.
• Kafka Streams applications where required.
• Schema management and serialization formats such as Avro, JSON, and Protobuf.
• REST APIs and microservices.
• Linux/Unix.
• SQL and database fundamentals.
• Git and CI/CD.
• Docker and Kubernetes.
• Cloud platforms such as AWS, Azure, or GCP.

Responsibilities

• Design and implement scalable Kafka-based event-streaming solutions.
• Develop and maintain Kafka producers and consumers.
• Create and manage Kafka topics, partitions, consumer groups, and replication.
• Configure and monitor Kafka brokers and clusters.
• Implement high availability, fault tolerance, and disaster-recovery strategies.
• Develop real-time data pipelines and integrations between applications and data platforms.
• Work with Kafka Connect for source and sink integrations.
• Implement and maintain Kafka Streams applications where required.
• Monitor Kafka performance, throughput, latency, lag, and cluster health.
• Troubleshoot producer, consumer, broker, partition, and replication issues.
• Implement security using SSL/TLS, SASL, ACLs, and authentication/authorization.
• Optimize Kafka configurations for performance and scalability.
• Work with development and data engineering teams to define event schemas and integration patterns.
• Implement monitoring and alerting using tools such as Prometheus, Grafana, or equivalent platforms.
• Participate in production support, incident management, and root-cause analysis.
• Automate deployments and operational tasks using CI/CD and scripting.

Skills

Kafka expertise

Event-streaming design

CI/CD scripting

Linux/Unix

REST APIs

SQL fundamentals

Git

Tools

Apache Kafka

Kafka Connect

Kafka Streams

Docker

Kubernetes

AWS

Azure

GCP

Java

Python

Scala

Job description

Staffing Spot, Inc. is seeking a Kafka Engineer in New York, NY to design, develop, implement, and support scalable event-streaming pipelines using Apache Kafka and related technologies.

You will build producers/consumers, manage topics and partitions, and ensure high availability across distributed systems. The role requires deep knowledge of Kafka, Connect, Streams, and cloud platforms, plus strong scripting and software development in Java, Python, or Scala.