AWS Kafka Engineer

Machine Learning Technologies, LLC; Maryland Heights (MO); Remoto; USD 77.000 - 110.000; Tempo pieno

Descrizione del lavoro

Machine Learning Technologies, LLC is seeking a Kafka Platform Engineer to design, deploy, and operate scalable Kafka environments for real-time data processing. You will monitor health, performance, and security, and implement Connect integrations with S3, Trino, and OpenSearch.

The role requires strong scripting, automation, and DR planning. Ideal candidates have hands-on Kafka/Confluent experience, Terraform and Ansible skills, and proven ability to lead cross-team incident resolution in

Competenze

• 3+ years administering enterprise Apache Kafka/Confluent Platform environments, including production cluster design and upgrades.
• 3+ years troubleshooting Kafka issues including broker availability, replication, rebalance, lag, and Connect failures.
• 3+ years automating infra with Terraform/Ansible, with Git-based CI/CD and Python or scripting.
• Experience designing highly available Kafka architectures with cross-cluster replication and disaster recovery.
• Deep understanding of Kafka security: TLS, SASL, ACLs, RBAC, and audit controls.
• Experience designing observability using Prometheus, Grafana, OpenSearch, or similar tooling.
• Ability to define engineering standards and guide producers/consumers.
• Strong client-facing communication and incident leadership in English.
• Office presence, on-call rotation, collaboration with vendor support.

Mansioni

• Design, deploy, and maintain scalable Kafka clusters for real-time data processing.
• Monitor broker health, throughput, lag, and storage; tune for reliability and performance.
• Build topics, replication policies, schemas, and Connect integrations (S3, Trino, OpenSearch).
• Implement security controls: TLS, SASL, ACLs, and credential management.
• Automate provisioning, configuration, scaling, upgrades, and patching using Terraform, Ansible, and Python.
• Plan backup/recovery, cross-cluster replication, and DR testing for defined SLAs.
• Participate in on-call rotation, lead troubleshooting, and coordinate with teams and vendors.
• Provide Kafka architecture guidance and maintain runbooks and capacity plans.

Conoscenze

Apache Kafka

Confluent Platform

Terraform

Ansible

Python scripting

CI/CD

Git workflows

Kafka Connect

OpenSearch

Monitoring dashboards

Strumenti

AWS

Kubernetes

CI/CD tools

Descrizione del lavoro

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Location: Maryland Heights, Missouri – Remote

Duration: 24 Months + Extension

Bill Rate: $68/hour (C2C)

Job Type: W-2, 1099, C2C

Client: To Be Discussed Later

Work Authorization:US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Key Responsibilities

• Design, deploy, configure, upgrade, and maintain highly available, scalable Apache Kafka and Confluent Platform clusters for real-time data processing across production and non-production environments.
• Monitor platform health, broker performance, replication, partition balance, throughput, storage utilization, consumer lag, and connector status; troubleshoot incidents and tune the platform for reliability, performance, and minimal downtime.
• Build and support Kafka topics, partitions, replication policies, schemas, service accounts, and Kafka Connect integrations, including connectors for Amazon S3, Trino, and OpenSearch; assist application and data teams with producer and consumer integration.
• Implement and maintain platform security and compliance controls, including TLS encryption, SASL authentication, access control lists, role-based access control, secrets management, auditability, and secure certificate lifecycle practices.
• Automate cluster provisioning, configuration, scaling, upgrades, patching, and operational tasks using Terraform, Ansible, CI/CD pipelines, and Python or scripting.
• Plan and test backup, recovery, failover, cross-cluster replication, and disaster recovery procedures to support defined service-level objectives and business continuity requirements.
• Participate in production support and on-call rotation, lead technical troubleshooting, document root cause analyses, and coordinate resolution with application teams, infrastructure teams, incident managers, vendors, and cloud support personnel.
• Provide technical guidance on Kafka architecture and operational best practices; maintain runbooks, standards, diagrams, capacity plans, and support documentation; and remain current with Apache Kafka and Confluent capabilities.

Required Qualifications

• 3+ years of hands‑on experience administering enterprise Apache Kafka or Confluent Platform environments, including production cluster design, capacity planning, upgrades, patching, performance tuning, and lifecycle management.
• 3+ years of experience troubleshooting complex Kafka issues involving broker availability, replication, partitioning, rebalance behavior, consumer lag, throughput, latency, storage, and Kafka Connect task failures.
• 3+ years of experience automating infrastructure and platform operations with Terraform and Ansible, supported by Git-based CI/CD pipelines and Python or scripting.
• Demonstrated experience designing and supporting highly available Kafka architectures, including cross-cluster replication, failover, backup, recovery, and disaster recovery testing.
• Deep understanding of Kafka security and governance, including TLS, SASL, ACLs, RBAC, Schema Registry compatibility, service accounts, credential management, and audit controls.
• Experience designing and operating observability capabilities using Prometheus, Grafana, Confluent Control Center, OpenSearch, or equivalent solutions, including actionable dashboards and alert thresholds.
• Proven ability to define engineering standards, advise application teams on producer and consumer patterns, review platform designs, and lead technical resolution across multiple teams.
• Proven excellence in client‑facing communication, technical documentation, incident leadership, root cause analysis, and consulting skills in English.
• Ability to work in office, participate in an on‑call rotation, and collaborate with senior client resources and vendor support teams.

Preferred Qualifications

• Experience with Confluent Platform or Confluent Cloud components such as KRaft or ZooKeeper, Schema Registry, Kafka Connect, Control Center, Cluster Linking, and stream‑processing capabilities.
• Experience operating Kafka on AWS, Kubernetes, Red Hat OpenShift, or another cloud or container platform; AWS Managed Streaming for Apache Kafka (Amazon MSK) experience is beneficial.
• Experience with GitLab, GitHub Actions, Jenkins, Argo CD, or comparable CI/CD and GitOps tooling.
• Experience integrating Kafka with Amazon S3, Trino, OpenSearch, relational databases, IBM MQ, data warehouses, or other enterprise source and sink systems.
• Familiarity with data governance, schema evolution, topic naming and retention standards, capacity forecasting, service‑level objectives, and platform cost optimization.
• Relevant certification or formal training in Apache Kafka, Confluent, AWS, Kubernetes, Terraform, Ansible, or related technologies.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.