Senior Kafka Infrastructure Engineer — Streaming & Open Source

Bloomberg; New York (NY); On-site; Full time

Job description

The Bloomberg Managed Services organization (BMS) is responsible for creating and curating Bloomberg Engineering's "as a Service" software catalog and infrastructure. We develop and integrate products into a simple, elegant user interface and provide a concise way for engineers to create data-store and middleware instances.

The BMS Streaming team focuses on providing streaming solutions on the BMS Platform powered by Apache Kafka and its ecosystem. Apache Kafka has quickly become the backbone of event driven architecture at Bloomberg and the main data bus for data distribution, live streaming, and general purpose messaging. Given that the majority of Engineering teams in the company already leverage our product in production, you’ll have the chance to positively impact virtually any product under the Bloomberg umbrella by joining our team.

The easiest way to learn more about what our team does and some of the challenges we face is to check out our talk at Kafka Summit: https://www.confluent.io/resources/kafka-summit-2020/fully-managed-multi-tenant-kafka-clusters-tips-tricks-and-tools/

We are extremely proud of our open and inclusive culture, and value diversity of thought and perspective in every form. We're looking for engineers with a real passion for writing reusable, efficient solutions to complex problems, who can collaborate and work effectively on small teams to develop software that impacts thousands of power users around the world.

We work with:

• Open-source frameworks and tools (Apache Kafka, Docker, Kubernetes, Node.js, OpenStack, etc)
• Java, Scala, C++, Python, Javascript, and Go as required to provide these frameworks in an easy to use manner for developers
• All projects follow the team's SDLC practices, including Docker-based unit testing, Jenkins integration, API documentation standards, etc.

We'll trust you to:

• Collaborate within an Agile, multi-disciplinary team
• Develop code around the Apache Kafka broker, its client libraries (C/C++, Java, Python), its ecosystem frameworks (Kafka Connect, Kafka Streams, Cruise Control, Burrow, etc.)
• Work on Open Source projects within the Kafka ecosystem and contribute changes upstream
• Develop an end-to-end full-stack user experience
• Pick up on new frameworks, experiment, and build software that works seamlessly on various environments
• Anticipate engineering needs and expectations from Bloomberg engineers, and build solutions that work for them
• Collaborate directly with product owners and application teams to understand requirements, conduct feature demos, and provide system design consultations
• Identify feature gaps in the technologies and work with the open source community and our vendors to co-design and co-develop features and bug fixes

You'll need to have:

• 4+ years experience programming in Java, C++, Python, Go, Scala or similar in a production environment
• A Degree in Computer Science, Engineering or similar field of study or equivalent work experience
• Strong communication and collaboration skills
• Strong understanding of distributed systems, data structures, algorithms and system design
• Experience designing and implementing low-latency, high-volume, and high-availability distributed systems
• Commitment to writing meaningful unit tests and integration tests

We'd love to see:

• Passion for distributed systems and large scale data systems
• Familiarity with Apache Kafka
• Familiarity with Kubernetes and Docker
• Java programming experience
• Experience maintaining and contributing to large Open Source projects
• Experience evolving and monitoring systems at scale
• Deep understanding of TCP/IP and Unix networking, Linux kernel performance