Engineering Manager, Serving/API

Jobgether SRL; United States; Remote; USD 225,000 - 350,000; Full time

Benefits offered by this job

Remote-first environment

Fully paid medical, dental and vision

401(k) matching

Generous PTO

Company-provided computer

Job summary

Jobgether SRL is seeking an Engineering Manager to lead the API serving and AI inference team in a remote-first, United States–based role. You will drive technical leadership across tokenization, structured generation, reasoning, and model integration, while shaping the roadmap and ensuring API fidelity, latency, and reliability.

You will collaborate with compiler, accelerator, and production platform teams to scale serving capabilities and ownership.

Qualifications

• Experience leading ML infra teams or LLM serving.
• Strong C++ systems background and Python proficiency.
• Ability to translate requirements into roadmaps with ownership.
• Experience recruiting and developing engineers across levels.
• Deep understanding of LLM serving concepts and APIs.
• Experience with AI accelerators is valuable.

Responsibilities

• Lead and grow the API serving and model integration teams.
• Define technical and organizational roadmaps covering API compatibility and model support.
• Own API fidelity including endpoints, streaming, and error handling.
• Collaborate with compiler, accelerator, and platform teams.
• Establish ownership boundaries and release gates for new models.
• Translate customer priorities into sequenced engineering initiatives.
• Ensure serving latency and reliability targets are met.

Skills

LLM inference

C++

Python

Roadmap planning

Team leadership

APIs

Communication

Hardware collaboration

Tools

vLLM

TensorRT-LLM

SGLang

XGrammar

Hugging Face tokenizers

Job description

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for an Engineering Manager, Serving/API based in United States.

This is a technical leadership role overseeing the software layer that connects customer API requests to production AI inference. You will lead and grow an engineering team responsible for API serving, tokenization, structured generation, reasoning, speculative decoding, and emerging model capabilities. The role combines deep systems thinking with people leadership, product accountability, and cross‑functional execution. You will shape the technical roadmap while ensuring API compatibility, correctness, latency, reliability, and scalable model integration. You will collaborate closely with compiler, accelerator, production platform, hardware, and customer‑facing teams in a rapidly evolving environment. As the model catalog and customer base expand, you will help establish durable ownership structures and engineering practices that can scale with the organization. This is an opportunity to directly influence both the technology and team responsible for delivering high‑performance AI inference services to customers.

• Lead, coach, hire, and develop engineers working across API serving, tokenization, structured generation, reasoning, and model integration.
• Establish and maintain a clear technical and organizational roadmap covering API compatibility, chat templates, tool calling, reasoning, request budgets, speculative decoding, and new‑model support.
• Own API fidelity, including OpenAI‑compatible behavior, HTTP endpoints, request validation, streaming, usage accounting, and robust error handling.
• Lead support for reasoning models, including reasoning formats, reasoning‑effort controls, token limits, and per‑request accounting.
• Partner with compiler and execution teams on speculative decoding, including draft‑model integration, request‑side infrastructure, acceptance metrics, and policies for evaluating when speculation improves performance.
• Define and lead the roadmap for vision‑language model support and interoperability with modern serving frameworks, making informed decisions about what to adopt, integrate, or build internally.
• Ensure serving‑layer overhead remains off the critical path for time‑to‑first‑token and streaming latency.
• Design the API layer to remain independent of underlying hardware topology as inference platforms evolve and expand across multiple hosts.
• Establish clear ownership boundaries for shared host‑level load balancing, request policies, and system protections in partnership with production platform teams.
• Build robust release gates, including API conformance tests, tool‑calling and reasoning evaluations, and regression suites, so new models and capabilities can ship with confidence.
• Collaborate with production platform and orchestration teams to turn new serving capabilities into reliable, supportable production endpoints.
• Translate customer and business priorities into sequenced engineering initiatives while protecting the team from reactive or unstructured demands.
• Establish measurable outcomes around API fidelity, tool‑call accuracy, serving latency, throughput, reliability, and customer‑visible quality.
• Develop clear ownership boundaries and grow emerging technical leaders as the organization and product surface expand.
• Support longer‑term evolution of the function as it scales into areas such as API and protocol compatibility, structured generation, tool calling, speculative decoding, and multimodal serving.

Requirements

• Demonstrated success managing and growing engineering teams responsible for LLM inference, model serving, ML systems, or closely related technical domains.
• Strong systems engineering background with expertise in C++ and working proficiency in Python, including the ability to review performance‑ and correctness‑critical code.
• Proven ability to turn ambiguous product and customer requirements into coherent roadmaps, clear ownership models, and measurable engineering outcomes.
• Experience recruiting, coaching, developing, and retaining engineers across different levels of experience while maintaining high technical standards.
• Strong understanding of LLM serving concepts, including tokenization, chat templates, sampling, streaming APIs, and OpenAI‑compatible APIs.
• Hands‑on understanding of tool calling, function‑calling formats, structured output, and the practical failure modes associated with these capabilities.
• Experience with serving frameworks or open‑source technologies such as vLLM, SGLang, TensorRT‑LLM, llguidance, XGrammar, or Hugging Face tokenizers is highly valuable.
• Familiarity with multimodal serving, including vision‑language models, image preprocessing, vision encoders, and multimodal token handling.
• Experience supporting reasoning models and associated output formats, such as OpenAI Harmony, is advantageous.
• Experience working with GPU, FPGA, ASIC, or other AI accelerators, particularly for memory‑bandwidth‑bound inference, is valuable.
• Experience developing customer‑facing API products where engineering teams are responsible for compatibility, escalations, and release readiness.
• Ability to make sound technical and organizational trade‑offs and communicate decisions clearly to engineers, cross‑functional partners, and executive stakeholders.
• Hands‑on leadership style with enough technical depth to engage meaningfully in architecture and code discussions without becoming a delivery bottleneck.
• Strong written and verbal communication, prioritization, collaboration, and organizational skills.
• Ability to operate effectively in a fast‑moving environment where models, modalities, serving technologies, and customer requirements evolve rapidly.
• Candidates must currently be authorized to work in the United States. New visa sponsorship is not available, though eligible H‑1B transfers may be supported.

Benefits

• Base salary range of $225,000-$350,000, with final compensation determined by experience, qualifications, internal equity, and expected impact.
• Competitive equity as part of the total compensation package.
• Fully company‑paid medical, dental, and vision insurance for employees and dependents.
• Company‑paid life and disability insurance, with voluntary supplemental coverage options.
• Supplemental hospital, critical illness, and accident insurance options.
• Unlimited paid time off, with an emphasis on taking meaningful time to rest and recharge.
• 13 paid company holidays.
• Remote‑first work environment.
• Company‑provided computer and home office setup.
• 401(k) retirement plan with company matching, eligible from day one.
• Opportunity to shape both advanced AI inference technology and the engineering organization supporting it.
• Opportunity to work on rapidly evolving accelerator, model‑serving, and production AI technologies.