Kafka Engineer

Staffing Spot, Inc.; San Francisco (CA); On-site; USD 150,000 - 230,000; Full time

Job summary

Staffing Spot, Inc. is seeking a Kafka Engineer to design, develop, and operate scalable event-streaming pipelines using Apache Kafka and related technologies in a cloud-native environment.

You will collaborate with data engineers to implement high-availability architectures, monitor performance with Prometheus and Grafana, and automate deployments using CI/CD within a Kubernetes/Docker stack to ensure robust, scalable data platforms.

Qualifications

• Experience with Apache Kafka, including producers and consumers.
• Strong knowledge of Kafka topics, partitions, offsets and replication.
• Experience with Kafka Connect and Kafka Streams.
• Hands-on experience with Docker and Kubernetes in production.
• Familiarity with monitoring: Prometheus and Grafana.

Responsibilities

• Design and implement scalable Kafka-based event-streaming solutions.
• Develop and maintain Kafka producers and consumers.
• Create and manage Kafka topics, partitions, and consumer groups.
• Configure and monitor Kafka brokers and clusters.
• Implement high availability, fault tolerance, and disaster recovery.

Skills

Kafka expertise

Real-time pipelines

CI/CD scripting

Troubleshooting

Team collaboration

Tools

Kafka Connect

Kafka Streams

Docker

Kubernetes

Prometheus

Grafana

Job description

We are looking for a Kafka Engineer responsible for designing, developing, implementing, and supporting highly scalable event-streaming and real-time data pipelines using Apache Kafka and related technologies.

Key Responsibilities

• Design and implement scalable Kafka-based event-streaming solutions.
• Develop and maintain Kafka producers and consumers.
• Create and manage Kafka topics, partitions, consumer groups, and replication.
• Configure and monitor Kafka brokers and clusters.
• Implement high availability, fault tolerance, and disaster-recovery strategies.
• Develop real-time data pipelines and integrations between applications and data platforms.
• Work with Kafka Connect for source and sink integrations.
• Implement and maintain Kafka Streams applications where required.
• Monitor Kafka performance, throughput, latency, lag, and cluster health.
• Troubleshoot producer, consumer, broker, partition, and replication issues.
• Implement security using SSL/TLS, SASL, ACLs, and authentication/authorization.
• Optimize Kafka configurations for performance and scalability.
• Work with development and data engineering teams to define event schemas and integration patterns.
• Implement monitoring and alerting using tools such as Prometheus, Grafana, or equivalent platforms.
• Participate in production support, incident management, and root-cause analysis.
• Automate deployments and operational tasks using CI/CD and scripting.

Required Technical Skills

• Strong knowledge of Apache Kafka.
• Kafka Producers and Consumers.
• Kafka Topics, Partitions, Offsets, Consumer Groups, and Replication.
• Kafka Connect.
• Kafka Streams.
• Schema management and serialization formats such as Avro, JSON, and Protobuf.
• Strong programming experience in Java, Python, or Scala.
• REST APIs and microservices.
• Linux/Unix.
• SQL and database fundamentals.
• Git and CI/CD.
• Docker and Kubernetes.
• Cloud platforms such as AWS, Azure, or GCP.

Skills: data,apache kafka,kafka,platforms,apache