Platform Architect

BayOne Solutions; United States; On-site; USD 190,000 - 230,000; Full time

Job summary

BayOne Solutions seeks an Architect-level Kafka expert to own end-to-end Confluent Kafka platforms for a major retail client. You will optimize throughput, latency, and reliability while defining enterprise standards and leading migration from on- prem to cloud.

You will mentor a distributed team and collaborate with application teams on event-driven patterns, implementing robust DR and cross-cluster replication. Strong SDLC, security and monitoring skills are essential.

Qualifications

• Architect-level, hands-on experience with Confluent Kafka (Confluent Platform and/or Confluent Cloud). Apache Kafka alone is not enough.
• Has designed, built or remediated Kafka at enterprise scale, ideally rescuing an underperforming setup.
• Retail enterprise experience is effectively mandatory. They should know how Kafka is typically set up in a large retailer.
• Experience with both on-prem and cloud Kafka deployments.
• Proven Kafka performance tuning and optimization in production.
• Strong SDLC background.

Responsibilities

• Own the architecture of the Kafka platform end to end for the client.
• Assess current setup and fix configuration, performance and reliability gaps.
• Tune brokers, topics, partitions, producers and consumers for throughput and latency.
• Define enterprise standards for topic design, schema management, security, monitoring and DR.
• Design and lead migration of Kafka components from on-prem to cloud.
• Guide and upskill the Kafka team (1 onshore, 2 offshore) and review their work.
• Collaborate with application and platform teams on event-driven integration patterns for retail workloads.
• Follow SDLC practices: documentation, change management, testing and production support.
• Design, implement, and manage highly available Kafka/Confluent platforms on Azure, including security and upgrades.
• Develop and maintain Kafka connectors, streams, producers/consumers using Java, Python, or similar.
• Implement cross-cluster replication and DR solutions with failover/failback strategies.

Skills

Confluent Kafka

Enterprise-scale Kafka

Retail experience

On-prem & cloud Kafka

Kafka performance tuning

SDLC

Tools

Terraform

Ansible

Kubernetes

Docker

Azure DevOps

Prometheus

Grafana

Job description

• Own the architecture of Client’s Confluent Kafka platform end to end, as the team's subject-matter expert.


• Assess the current setup, then find and fix its configuration, performance and reliability gaps.


• Tune brokers, topics, partitions, producers and consumers for throughput, latency and stability.


• Define enterprise standards for topic design, schema management, security, monitoring and disaster recovery.


• Design and lead the migration of Kafka components from on-prem to cloud.


• Guide and upskill the existing Kafka team (1 onshore, 2 offshore) and review their work.


• Work with application and platform teams on event-driven integration patterns for retail workloads.


• Follow SDLC practices: documentation, change management, testing and production support.


• Design, implement, and manage highly available and scalable Apache Kafka/Confluent platforms on Microsoft Azure, including brokers, topics, partitions, replication, security, and platform upgrades.


• Develop and maintain Kafka producers, consumers, Kafka Connect, Kafka Streams, and event-driven applications, using Java, Python, or similar technologies.


• Implement and manage Confluent Cluster Linking, cross-cluster replication, and Disaster Recovery (DR) solutions, including failover/failback strategies and business continuity requirements.


• Monitor, troubleshoot, and optimize Kafka environments for performance, reliability, scalability, and consumer-lag management, across development and production environments.


• Automate Kafka platform operations using Terraform, Ansible, Kubernetes, Docker, and Azure DevOps/CI/CD, while establishing platform standards, security controls, monitoring, documentation, and production support

Required qualifications

• Architect-level, hands-on experience with Confluent Kafka (Confluent Platform and/or Confluent Cloud). Apache Kafka alone is not enough.


• Has designed, built or remediated Kafka at enterprise scale, ideally rescuing an underperforming setup.


• Retail enterprise experience is effectively mandatory. They should know how Kafka is typically set up in a large retailer.


• Experience with both on-prem and cloud Kafka deployments.


• Proven Kafka performance tuning and optimization in production.


• Strong SDLC background.

Nice to have

These are typical for the role but were not raised on the call. Confirm them against the client's detailed JD.

• Has led an on-prem to cloud Kafka migration (AWS, Azure or GCP).


• Confluent certification (Administrator or Developer).


• Monitoring and automation tools such as Prometheus, Grafana, Terraform or Ansible.


• Mentoring offshore or distributed teams.