Lead Kafka Engineer

Golden Technology, Inc.; United States; On-site; USD 140,000 - 210,000; Full time

Job summary

Golden Technology, Inc. seeks a Principal Kafka Engineer to lead the management, scaling, and optimization of its enterprise event-streaming ecosystem. The role emphasizes deep Kafka expertise, OOP automation, and ITSM-driven service delivery across on-prem and cloud environments.

The ideal candidate will design and maintain highly available clusters, drive architecture decisions, and collaborate with globally distributed teams to ensure reliability and high throughput in production.

Qualifications

• 5+ years of Kafka administration experience and 10+ years in software development or systems engineering.
• Expertise in Apache Kafka/Confluent Platform including Schema Registry, Kafka Connect, Avro, and KSQL.
• Proficient in Cluster Management - Zookeeper, KRaft; strong Java, Go, Python OO programming.

Responsibilities

• Kafka Administration: Deploy, configure, and maintain highly available Kafka clusters across on-prem and cloud.
• OOP-based development and automation using Java for cluster provisioning, monitoring, and self-service onboarding.
• ITSM integration: manage lifecycle with incident, problem, and change management to reduce downtime.
• Security and compliance: implement security controls and authentication mechanisms.
• Performance optimization: monitor cluster health with Prometheus, Grafana, dynaTrace; plan capacity and tuning.
• Collaboration: work with DevOps and application teams on event-driven architecture and producer/consumer best practices.

Skills

Kafka

Java

Python

Go

Spring Boot

Kubernetes

Terraform

Ansible

GitHub Actions

ITSM/ITIL

OOP

dynaTrace

Architecture

Networking

Tools

Docker

Kubernetes

Kafka Connect

Schema Registry

Avro

KRaft

Zookeeper

Prometheus

Grafana

Terraform

Job description

We are seeking a Principal Kafka Engineer to lead the management, scaling, and optimization of our enterprise event-streaming ecosystem. The ideal candidate brings deep expertise in Apache/Confluent Kafka, applies object-oriented programming principles to advanced automation, and uses ITSM practices to support reliable, high-quality service delivery.

Required Qualifications:

• Experience: 5+ years of Kafka administration experience and 10+ years in software development or systems engineering.
• Technical Skills:
• Expertise in Apache Kafka or Confluent Platform, including Schema Registry, Kafka Connect, Avro, and KSQL.
• Proficient in Cluster Management - Zookeeper, KRaft
• Strong object-oriented programming skills; Java, Go, Python
• Hands-on experience with Infrastructure as Code tools such as Terraform, Ansible, or Salt.
• Java backend development experience with Spring Boot.
• Solid understanding of software design patterns.
• Strong knowledge of architecture and design concepts, with the ability to create and maintain architecture artifacts.
• Proficient in containerization technology such as Docker, Kubernetes etc.
• Expertise in DevOps, CI/CD, and GitHub Actions.
• Expertise in Domain Driven Design and Enterprise Application Integration.
• Experience with observability, telemetry, and APM tools such as dynaTrace.
• Good understanding of networking, storage, virtualization, and other infrastructure architecture.
• Self-starter with a goal-oriented mindset.
• Excellent communication, negotiation, and problem-solving skills.
• Expertise in supporting a Highly Available Low Latent Enterprise Application in Production environment(s).
• Experience working with globally distributed team(s).
• Frameworks: Solid understanding of ITSM/ITIL frameworks and Agile methodologies.

Desired Qualifications:

• Front-end development experience with Angular, React (preferred), or other TypeScript-based frameworks.
• Expertise in FinOps.
• Experience working in platform teams.

Preferred Certifications:

• Cloud provider certifications, such as Azure Certified Solutions Architect.
• Confluent Certified Administrator for Apache Kafka (CCAAK).
• TOGAF

Key Responsibilities:

• Kafka Administration: Deploy, configure, and maintain highly available Kafka clusters across on-premises and cloud environments, including Azure and GCP. Manage topics, partitions, and replication to ensure platform reliability.
• OOP-Based Development and Automation: Design and build reusable frameworks, automation tools, and APIs using Java to simplify cluster provisioning, monitoring, and self-service onboarding.
• ITSM Integration: Manage the platform lifecycle through ITSM processes, including incident, problem, and change management, to reduce downtime and maintain SLA compliance.
• Security and Compliance: Implement and maintain security controls and authentication mechanisms.
• Performance Optimization: Monitor cluster health with tools such as dynaTrace, Prometheus, and Grafana. Perform capacity planning and tuning to support high-throughput data pipelines.
• Collaboration: Partner with DevOps and application teams to provide guidance on event-driven architecture and producer/consumer performance best practices.

NOTES:

• Top 3 skills for each req : Java (Object Oriented Programming), Kafka, Software Architecture, Vulnerability Management
• Soft Skills needed : Self Starter, Driven, Negotiation, Effective Communication
• Are there any automatic disqualifiers? - Someone who can’t drive things, understand what is being asked, unable to prioritize, support production issues, mitigate security risks
• Team dynamics & make-up (size, locations) - right now it is a 12-member team, with a mix of onsite, offshore team.