Kafka to AWS MSK Migration Engineer

N Consulting Limited; Hyderabad; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

N Consulting Limited is seeking a Kafka to AWS MSK Migration Engineer to drive end-to-end migration projects from on‑premise Kafka to Amazon MSK in a hybrid cloud setup.

Ideal candidates will show deep expertise in Kafka architecture, MSK provisioning, security, and automation scripting with Python or Java, and have strong CI/CD experience with Jenkins and Git. Collaboration with cross-functional teams is essential.

Qualifications

• 5+ years of experience in Kafka, AWS, and data migration.
• Hands-on with MSK, MirrorMaker 2, and Confluent Replicator.
• Familiar with CI/CD, Jenkins, Git, and monitoring tools.

Responsibilities

• Lead migration using AWS MSK and Kafka tooling.
• Implement automation scripts in Python/Java for topic creation and ACL migration.
• Ensure security, monitoring, and reliability across migration pipelines.

Skills

AWS

Kafka

AWS MSK

Migration tooling

Cloud migration

Kafka ecosystem

Networking & security

Python/Java

CI/CD pipelines

APIs & integration

GitLab/Jenkins

Docker/Kubernetes

Observability tooling

Automation excellence

Tools

GitLab

Jenkins

Maven

Docker

Kubernetes

Job description

Role: Kafka to AWS MSK Migration Engineer

Experience: 5+ years

Location: Bengaluru, Hyderabad

Roles and Responsibilities

Candidates are not expected to be experts in every tool, but should bring strong hands‑on experience across several of the following areas:

• AWS Services: Hands‑on experience with AWS compute and migration patterns, including ECS, EKS, Lambda, CloudWatch, and related cloud services.
• Apache Kafka: Deep hands‑on experience with Kafka architecture including brokers, topics, partitions, consumer groups, replication, and cluster operations.
• AWS MSK: Practical experience with Amazon Managed Streaming for Apache Kafka including cluster provisioning, configuration, monitoring, and security (IAM, mTLS, SASL/SCRAM).
• Migration Tooling: Experience with MirrorMaker 2, Confluent Replicator, or similar cross‑cluster replication tools for data migration and offset synchronization.
• Cloud Migration: Practical understanding of on‑prem to cloud migration strategies, workload assessment, testing, cutover, and post‑migration validation.
• Kafka Ecosystem: Familiarity with Schema Registry, Kafka Connect, Kafka Streams, and related ecosystem components.
• Networking and Security: Understanding of VPC peering, PrivateLink, TLS encryption, IAM policies, and network connectivity patterns for hybrid Kafka architectures.
• Programming and Automation: Experience with Python OR Java, Shell, or similar languages to build scripts, utilities, and automation frameworks for topic creation, ACL migration, and consumer group management.
• Work experience CI/CD pipelines, Jenkins , Docker, Kuberneties
• APIs and Integration: Working knowledge of RESTful APIs, API design, and API Gateway patterns.
• CI/CD and Version Control: Strong experience with GitLab, Jenkins, Maven, or similar tools supporting automated build, test, and deployment pipelines.
• Observability: Familiarity with monitoring, logging, alerting, and troubleshooting using tools such as CloudWatch, Prometheus, Grafana, Splunk, or equivalent platforms for Kafka/MSK cluster health.
• Automation Excellence: Ability to design repeatable automation and apply AI‑assisted engineering approaches to improve migration scale, quality, and efficiency.