Senior Kafka/Confluent Platform Engineer (On-Prem)

Machine Learning Technologies, LLC; Phoenix (AZ); On-site; USD 120,000 - 180,000; Full time

Job summary

Machine Learning Technologies, LLC seeks a senior contract Kafka/Confluent administrator to own and evolve our on-prem event streaming platform, focusing on Confluent Platform. You will lead hardware refresh planning for on-prem clusters, drive reliability and performance, and embed DevOps and automation across provisioning, deployment, observability, and incident response.

Experience with Apache Kafka and AWS MSK is desirable for cross-environment alignment.

Qualifications

• 5+ years in systems/platform engineering, SRE, or DevOps; 4+ years operating Kafka in production at scale.
• Deep knowledge of Kafka internals: partitions, replication, retention/compaction, rebalance strategies.
• Hands-on with Kafka Connect, Schema Registry, Mirror Maker/Confluent Replicator.
• Strong Linux fundamentals; networking (TCP, DNS, load balancing), and performance analysis.
• Proficiency in automation/scripting.
• Monitoring/observability: Data Dog, Grafana, JMX exporters, and log aggregation.
• Experience with DR, multi-region design, and incident management.
• Proven ability to produce clear, comprehensive documentation

Responsibilities

• Design, deploy, and operate highly available Kafka clusters (on-prem, cloud, and/or managed services such as Confluent Cloud or AWS MSK).
• Manage topics, partitions, quotas, retention policies, and consumer group strategies for performance and cost.
• Own upgrades, patches, and migrations.
• Implement and manage Kafka components: Kafka Connect, Schema Registry, Mirror Maker/Confluent Replicator, REST Proxy; familiarity with Kafka Streams and ksqlDB is a plus.
• Performance tuning (producers/consumers, batching, compression, acks, ISR, controller health), throughput testing, and benchmarking.
• Capacity planning, partitioning strategy, and cluster right-sizing.
• Hardware refresh plan: capacity model, sizing, architecture diagrams, migration/cutover strategy, risk register
• Implement and validated on-prem clusters on refreshed hardware with performance benchmarks
• Operational documentation: standards, runbooks, monitoring/alerts configuration, backup/restore and DR playbooks.
• Knowledge transfer sessions and documentation handoff at milestones and project close.

Skills

Kafka administration

SRE

DevOps

Automation scripting

Linux

Networking

Monitoring

Tools

Kafka Connect

Schema Registry

MirrorMaker

REST Proxy

Confluent Replicator

Grafana

Datadog

JMX exporters

Job description

Machine Learning Technologies, LLC seeks a senior contract Kafka/Confluent administrator to own and evolve our on-prem event streaming platform, focusing on Confluent Platform. You will lead hardware refresh planning for on-prem clusters, drive reliability and performance, and embed DevOps and automation across provisioning, deployment, observability, and incident response.

Experience with Apache Kafka and AWS MSK is desirable for cross-environment alignment.