Kafka Senior Engineer - Officer

State Street Corporation; Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

State Street Corporation in Bengaluru seeks a Senior Middleware Platform Engineer with deep Kafka experience to design, implement, and operate enterprise messaging infrastructure. The role focuses on building a first Kafka cluster on public cloud, automating tasks, and guiding junior engineers while collaborating with architecture and infrastructure teams.

The position requires strong coding skills, Linux proficiency, and hands-on programming across languages.

Qualifications

• Programming and automation experience.
• 8+ years of experience preferred but not required depending on candidate.
• Computer science degree preferred but not required depending on candidate.
• Experience with high availability (HA) and disaster recovery (DR) solutions for middleware or other platforms.

Responsibilities

• Design, engineer, and evangelize Kafka messaging across the enterprise.
• Lead design of our first Kafka cluster on a public cloud using Ansible, Terraform, and Python.
• Mentor junior engineers and provide technical leadership.
• Collaborate with Enterprise Architecture to identify best Kafka use cases.
• Automate tasks and health checks, monitoring, and reporting.
• Develop Kafka producer/consumer code and onboard apps across languages.
• Configure and test Kafka Connectors for system integration.
• Design monitoring solutions and baseline statistics for the platform.
• Support cross-team efforts to ensure resiliency and scalability.

Skills

Programming

Troubleshooting

Team collaboration

Problem solving

Education

Computer science degree preferred

Tools

Kafka

Ansible

Terraform

Python

Shell

Puppet

Red Hat Linux

Job description

Who we are looking for

As a Senior Middleware Platform Engineer, you will be working in a fast paced environment in a team of highly skilled engineers responsible for leading platform engineering, design, and implementation of next generation messaging infrastructure. You will also be expected to assist and act as final level of support for the production services supported by the team.

What you will be responsible for

As Kafka Senior Engineer you will:

• Design, Engineer, and Evangelize Kafka Messaging across the enterprise
• Lead in the design and engineering of our first Kafka Cluster on a Public Cloud (Azure, AWS, other) using Ansible, Terraform, Python
• Lead and assist in providing direction for Junior Engineers
• Work with Enterprise Architecture team in identifying and implementing use cases best suited for Kafka
• Write code to automate tasks ( request processing, product installations, health checks, proactive monitoring, alerting, reporting, predictive analytics )
• Write Kafka producer code and Kafka consumer code within consumer groups to test the Kafka client APIs, develop Kafka application best practices, and help safely onboard Kafka applications to our Kafka platforms (across programming languages/platforms)
• Configure and test pre-built Kafka Connectors for integrating Kakfa with other systems
• Design, code, and test new Kafka Connectors where needed for integrating Kakfa with other systems
• Perform optimization and tuning of Kafka
• Design monitoring solutions for the platform and baseline statistics reporting
• Work on creating best practices, standards, Integration patterns and on-boarding process automation
• Ability to work with application development teams to understand usage pattern and provide solutions
• Lead in collaborating with other infrastructure teams to achieve high resiliency and scalability of our Kafka Messaging platforms
• Provide technical expertise in scoping end to end business requirements and assist in reflecting them in implementation for different Kafka connectors
• Conduct technical reviews of integration implementation for all SDLC phases to assess the quality of integration solutions built on Kafka
• Strive to provide highly resilient Kafka Messaging systems using the latest high availability (HA) and disaster recovery (DR) technology solution

What we value

These skills will help you succeed in this role

• Love of writing code/programming
• Solid coding ability to write robust, fault tolerant, well tested code. Any scripting/programming language is fine but with willingness to learn Python, Ansible, Terraform, Puppet, Shell. Experience with source code control system
• Experience working on Red Hat Linux
• Deep level of understanding of Kafka Messaging or another Messaging technology
• Experience with finding root cause for problems with Kafka or Applications using Kafka (or another Messaging technology)
• Experience with tuning Kafka systems (or other Messaging systems)
• Experience with supporting Application architectures SEDA, Kappa, and Lambda or similar

Education & Preferred Qualifications

• Programming and automation experience.
• 8+ years of experience preferred but not required depending on candidate
• Computer science degree preferred but not required depending on candidate
• Experience with high availability (HA) and disaster recovery (DR) solutions for middleware or other platforms

Additional requirements

• This is a “hands on keyboard” engineering position
• L4 ability to drive a project and collaborate within our team and with other application and infrastructure teams
• L4 ability to troubleshoot complex issues
• Occasional weekend work may be required for Prod deployments/changes
• Our engineers will need to create new docs and SOPs for use by our support team and perform some cross training of the support team members

Work Schedule

No Hybrid Working. It is fully Working from Office Schedule – APAC or EMEA time Location – Bangalore or Hyderabad (Candidates not willing to relocate need not apply)

Why this role is important to us

The team you will be joining is a part of Platform Engineering Middleware, a function that is vital to the company as we execute over 2 million job runs per day in Autosys, support 3,500 web sites on over 3,000 servers (both internal and external facing), 5 Billion messages per day traverse Kafka and MQ across more than 150,000 queues, and we handle over 2 million file transfers per day. State Street is one of 6 banks designated by the federal reserve as GSIB (Global Systemically Important Bank) with nearly 4 trillion USD in AUM. Our Middleware platforms touch all areas of the company and are the vital plumbing serving all the banks activities. As a GSIB we operate under a strict oversight program from the Federal Reserve and must ensure our systems are operating at full capacity 24/7.

About State Street

What we do. State Street is one of the largest custodian banks, asset managers and asset intelligence companies in the world. From technology to product innovation we’re making our mark on the financial services industry. For more than two centuries, we’ve been helping our clients safeguard and steward the investments of millions of people. We provide investment servicing, data & analytics, investment research & trading and investment management to institutional clients. Work, Live and Grow. We make all efforts to create a great work environment. Our benefits packages are competitive and comprehensive. Details vary in locations, but you may expect generous medical care, insurance and savings plans among other perks. You’ll have access to flexible Work Program to help you match your needs. And our wealth of development programs and educational support will help you reach your full potential. Inclusion, Diversity and Social Responsibility. We truly believe our employees’ diverse backgrounds, experiences and perspective are a powerful contributor to creating an inclusive environment where everyone can thrive and reach their maximum potential while adding value to both our organization and our clients. We warmly welcome the candidates of diverse origin, background, ability, age, sexual orientation, gender identity and personality. Another fundamental value at State Street is active engagement with our communities around the world, both as a partner and a leader. You will have tools to help balance your professional and personal life, paid volunteer days, matching gift program and access to employee networks that help you stay connected to what matters to you. State Street is an equal opportunity and affirmative action employer.

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success. We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future. As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

As an Equal Opportunity Employer

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.