Lead Kafka Middleware Engineer

Full time; Dallas (TX); On-site; USD 130,000 - 170,000; Full time

Job summary

A prestigious financial institution is seeking a Lead Kafka Middleware Engineer to design and manage Kafka clusters and automate cloud infrastructure. This role will involve extensive use of IaC, cloud automation, and CI/CD tools to ensure high availability and performance of Middleware solutions.

Qualifications

• 10+ years of experience in infrastructure Middleware administration.
• In-depth expertise in Kafka administration, including cluster management.
• Experience with monitoring tools like Splunk, Prometheus, and Grafana.

Responsibilities

• Design, implement, and manage highly available Kafka clusters.
• Automate cloud resource provisioning and management.
• Integrate IaC and container technologies into CI/CD pipelines.

Skills

IaC tools

Cloud automation

CI/CD tools

Scripting (Python, Bash)

Troubleshooting

Networking principles

Education

Bachelors degree in Computer Science, Engineering or a related field

Tools

Ansible

Terraform

AWS

Azure

Kubernetes

GitHub

Jenkins

Job description

Prestigious Financial Institution is currently seeking a Lead Kafka Middleware Engineer with deep expertise in Kafka administration, infrastructure as code (IaC), cloud automation, container orchestration and CI/CD pipelines. The ideal candidate will be responsible for designing, implementing, and maintaining robust and scalable Middleware solutions, ensuring high availability, performance, and security. Candidate will play a crucial role in automating infrastructure provisioning, deployments, and operations, enabling our organization to rapidly deliver and scale applications.

Responsibilities:

• Design, implement and manage highly available and scalable Kafka clusters.
• Monitor Kafka performance, troubleshoot issues and optimize configurations.
• Develop and maintain IaC using Ansible and Terraform for infrastructure provisioning and configuration Management.
• Create and maintain reusable IaC modules.
• Design and implement cloud-based infrastructure solutions on AWS and Azure.
• Automate cloud resource provisioning, scaling and management using cloud-native tools and services.
• Deploy and Manage containerized applications using Kubernetes and Rancher
• Troubleshoot container-related issues and optimize container performance.
• Design, implement and maintain CI/CD pipelines using tools like GitHub, Artifactory, Harness and Jenkins.
• Automated the build, test and deployment of Middleware components.
• Integrate IaC and container technologies into CI/CD pipelines.
• Document all processes and procedures.
• Work with development teams to ensure smooth deployments.

Qualifications:

• Strong proficiency in IaC tools, specifically Ansible, Terraform and bash Scripting.
• Extensive experience with cloud automation and provisioning on AWS and Azure.
• Proficiency in CI/CD tools, including GitHub, Artifactory, Harness and Jenkins.
• Strong Scripting skills in languages like Python and Bash.
• Excellent troubleshooting and problem-solving skills
• Understanding of networking principles.
• Experience with monitoring tools like Splunk, Splunk OTEL, Prometheus and Grafana.
• Bachelors degree in Computer Science, Engineering or a related field (or equivalent experience)
• 10+ years of experience in infrastructure Middleware administration.
• In-depth expertise in Kafka administration, including cluster management, performance tuning, and security.
• AWS Solutions Architect, CKAD or CKA certifications preferred.