Senior ML Platform Engineer - Remote AI Serving

Bright Vision Technologies; New Jersey; Remote; USD 100,000 - 160,000; Full time

Job summary

Bright Vision Technologies is seeking an ML Platform Engineer to design, build, and operate high-performance inference platforms for production ML models. You will focus on systems engineering for AI deployment, including routing, batching, caching, autoscaling, GPU utilization, and end-to-end observability across model workloads.

The role requires 10+ years in distributed systems, strong Python and systems language skills, and hands-on experience with LLM inference frameworks such as vcLLM or

Qualifications

• Bachelor’s or Master’s degree in Computer Science or a related field.
• 10+ years of experience in distributed systems, infrastructure, or ML platform engineering.
• Strong proficiency in Python and a systems language such as Go, Rust, or C++.
• Deep experience operating high-throughput, low-latency services in production.
• Hands-on experience with LLM or large model inference frameworks such as vcLLM or TensorRT-LLM.
• Strong understanding of GPU architecture, memory hierarchies, and accelerator utilization.
• Familiarity with Kubernetes, autoscaling, and modern cloud platforms.
• Experience with observability stacks including metrics, tracing, and structured logging.
• Solid grounding in performance engineering and capacity planning.
• Strong communication and incident response skills.

Responsibilities

• Design and operate model serving platforms supporting diverse workloads including LLMs, vision models, and recommendation systems.
• Optimize inference performance using continuous batching, paged attention, speculative decoding, and request multiplexing.
• Implement multi-tenant routing, rate limiting, and quality-of-service policies across model endpoints.
• Build autoscaling and capacity management systems that balance latency, throughput, and cost.
• Tune GPU utilization, memory management, and KV cache strategies for LLM serving workloads.
• Integrate model serving with API gateways, identity systems, and observability platforms.
• Implement caching, prompt deduplication, and response reuse strategies where appropriate.
• Drive end-to-end observability including latency histograms, queue dynamics, GPU utilization, and error tracking.
• Develop deployment workflows including canary releases, shadow testing, and automated rollback.
• Operate incident response for high-availability AI services and drive durable reliability improvements.
• Collaborate with ML and product teams to support new model releases and capability rollouts.
• Implement security controls including request signing, content filtering, and abuse detection at the serving layer.
• Document operational procedures, performance characteristics, and tuning guidance for internal teams.
• Stay current with AI serving research and translate advances into production capabilities.

Skills

Distributed systems

Python

Go/Rust/C++ (systems language)

Observability & monitoring

Incident response

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

vcLLM

TensorRT-LLM

Kubernetes

Cloud platforms

Job description

Bright Vision Technologies is seeking an ML Platform Engineer to design, build, and operate high-performance inference platforms for production ML models. You will focus on systems engineering for AI deployment, including routing, batching, caching, autoscaling, GPU utilization, and end-to-end observability across model workloads.

The role requires 10+ years in distributed systems, strong Python and systems language skills, and hands-on experience with LLM inference frameworks such as vcLLM or