Senior AI Inference Platform Engineer

Bloomberg; New York (NY); On-site; USD 160,000 - 240,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Disability benefits (short & long term

401(k) + match

Life insurance

Wellness programs

Job summary

Bloomberg is building core AI infrastructure to host, deploy, and serve ML models with guaranteed scalability, performance, and governance. The AI Inference platform leverages KServe and open-source tools to empower teams while abstracting infrastructure complexity.

You will design scalable online/offline inference systems, optimize performance, and drive architecture decisions across the platform, mentoring engineers and collaborating with cross-functional teams in New York.

Qualifications

• 5+ years of professional software engineering experience.
• Experience designing, building, and operating production distributed systems.
• Strong systems intuition and debugging and performance optimization skills.
• Ability to own problems end-to-end and ramp up in unfamiliar areas.
• 4+ years of demonstrated experience with an object-oriented programming language.
• A degree in Computer Science, Electrical Engineering, or equivalent practical experience.

Responsibilities

• Design and build scalable infrastructure for online and offline inference workloads.
• Lead integration of high-performance inference runtimes and serving frameworks.
• Drive architecture decisions balancing latency, throughput, reliability, and cost.
• Partner with engineering teams to improve model deployment, observability, and performance.
• Mentor junior engineers on system design, debugging, and performance optimization.

Skills

Distributed systems design

Performance optimization

Object-oriented programming

Kubernetes

PyTorch

CUDA

NCCL

TensorRT

ONNX

Triton

Education

Degree in Computer Science or Electrical Engineering

Tools

KServe

TensorRT

ONNX

Triton

Kubernetes

CUDA

NCCL

PyTorch

Job description

Bloomberg is building core AI infrastructure to host, deploy, and serve ML models with guaranteed scalability, performance, and governance. The AI Inference platform leverages KServe and open-source tools to empower teams while abstracting infrastructure complexity.

You will design scalable online/offline inference systems, optimize performance, and drive architecture decisions across the platform, mentoring engineers and collaborating with cross-functional teams in New York.