Lead/Senior Inference Engineer

Jobgether

Canada

Remote

CAD 140,000 - 175,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Base salary CAD 140,000–175,000
Annual performance bonus potential
Flexible work arrangements
Mental health days and Headspace app
Tuition reimbursement
Volunteer days
Wellbeing programs

Job summary

Jobgether in Canada seeks a Lead/Senior Inference Engineer to build and operate production model-serving infrastructure for large language models. You will optimize latency, cost, and scalability while collaborating with training, evaluation, and product teams.

The role emphasizes autonomy, technical leadership, and shaping our serving platform as we expand internationally, with focus on reliability and compliance in a high-stakes environment.

Qualifications

  • Hands-on infra engineering with Kubernetes and production systems.
  • Experience operating model services at scale with reliability.
  • Direct model-serving infra experience for large language models.
  • Strong background in distributed systems, availability and reliability.
  • Troubleshoot production incidents under real traffic conditions.
  • End-to-end software delivery from design to operations.
  • Autonomy and ownership; mentor peers as the team grows.
  • Preferred: LLM-serving tech like vLLM/Triton, GPU planning, regulated envs, Rust/Go.

Responsibilities

  • Build and operate production-grade model-serving infrastructure.
  • Optimize inference latency with batching, caching, and quantization.
  • Improve observability and operational visibility for scaling.
  • Support ISO 27001 compliance foundations for production growth.
  • Establish cost tracking and GPU capacity contingency planning.
  • Evaluate cost vs. latency vs. reliability trade-offs for leadership.
  • Collaborate with model training/evaluation teams to deploy releases.
  • Shape engineering practices, on-call processes, and docs.
  • Partner with research, product, and engineering on platform evolution.
  • Mentor engineers and contribute to technical strategy.

Skills

Kubernetes
Distributed systems
Model-serving
Go
Rust
Performance optimization

Tools

vLLM
TensorRT-LLM
Triton
SGLang

Job description

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Lead/Senior Inference Engineer based in Canada.

The Lead/Senior Inference Engineer will build and operate the production inference infrastructure powering advanced large language models and AI-driven professional applications. This hands-on engineering role focuses on developing a reliable, scalable model-serving platform while optimizing performance, latency, and infrastructure costs. Working within a newly established engineering team, you will help shape its technical direction, engineering practices, and operational standards from the ground up. You will collaborate closely with model training, evaluation, research, and product teams to bring new model releases into production. The role offers significant autonomy and the opportunity to influence infrastructure strategy, reliability, and international platform expansion in a high-stakes, compliance-focused environment.


Accountabilities
  • Build, maintain, and operate production-grade model-serving infrastructure supporting large-scale AI workloads.
  • Implement and optimize inference techniques, including batching, caching, quantization, and comparable approaches, to improve latency and reduce operating costs without compromising model quality.
  • Strengthen platform observability, monitoring, and operational visibility as traffic volumes and deployment requirements increase.
  • Develop the operational and compliance foundations required for production growth, including controls and evidence supporting the ISO 27001 compliance path.
  • Establish cost-tracking and attribution mechanisms that support financial planning, as well as GPU-capacity contingency plans that help prevent service interruptions and unnecessary parallel infrastructure costs.
  • Evaluate and document trade-offs between cost, latency, and reliability, communicating technical recommendations to engineering leadership and partner teams.
  • Work directly with model training and evaluation teams to integrate new model releases into the serving infrastructure.
  • Help establish engineering practices and tools, including on-call processes, incident reviews, onboarding, and operational documentation.
  • Collaborate with research, product, and engineering stakeholders to ensure the serving platform evolves alongside model capabilities and product requirements.
  • Contribute to the team's technical strategy and mentor other engineers as the function grows.
Requirements:
  • Hands-on infrastructure engineering experience with Kubernetes or comparable technologies, including ownership of production systems beyond prototyping.
  • Experience operating and maintaining model services across different workloads, including peak traffic, scaling, disaster recovery, and service reliability.
  • Direct experience with model-serving infrastructure and/or inference optimization for large language models, at a scale appropriate to previous roles.
  • Strong background in large-scale distributed systems, particularly production scalability, availability, and reliability.
  • Proven ability to troubleshoot distributed systems and resolve production incidents under real traffic conditions.
  • Demonstrated experience delivering software end to end, from technical design and implementation through deployment and ongoing operations.
  • Ability to establish and uphold engineering best practices while adapting to evolving product and business requirements.
  • Strong autonomy, technical judgment, and ownership, with the ability to operate as an early lead engineer and mentor colleagues as the team expands.
  • Preferred qualifications:
    • Experience with LLM-serving technologies such as vLLM, TensorRT-LLM, Triton, SGLang, or similar frameworks.
    • Exposure to GPU-cluster capacity planning and/or multi-region or multi-cloud deployment environments.
    • Experience working in regulated or compliance-intensive environments, including ISO 27001, SOC 2, or comparable standards.
    • Familiarity with systems programming languages such as Rust or Go.
    • Knowledge of systems-level performance optimization, including rate limiting, request routing, and identifying bottlenecks across networking, hardware, and lower-level software.
Benefits:
  • Competitive compensation: CAD $140,000–$175,000 annual base salary for Ontario, Canada.
  • Potential eligibility for an annual performance bonus based on individual and organizational results.
  • Comprehensive benefits and wellbeing programs, including retirement savings, flexible vacation, and employee incentive programs.
  • Flexible work arrangements, including the possibility of working from anywhere for up to eight weeks per year.
  • Two company-wide mental health days annually and access to the Headspace app.
  • Learning and career development opportunities, including skills development and tuition reimbursement.
  • Resources supporting mental, physical, and financial wellbeing.
  • Two paid volunteer days per year, along with opportunities to participate in community projects and environmental and social impact initiatives.
  • Inclusive workplace culture focused on collaboration, professional growth, flexibility, and work-life balance.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer, Backend (ML Training and Serving)
Senior Software Engineer, Backend (ML Training and Serving)

Jobgether • Canada

Remote
CAD 153,000 - 213,000
Equity
Health coverage
Monthly stipends
+5
Senior Software Engineer, Backend (ML Training and Serving)
Senior Software Engineer, Backend (ML Training and Serving)

AItoolnavio • Canada

Remote
CAD 153,000 - 213,000
Equity rewards
Health coverage
Monthly stipends
+5
Senior LLMOps Engineer -Cloud / AI Infrastructure
Senior LLMOps Engineer -Cloud / AI Infrastructure

Talent To Hire Inc. • Toronto

Hybrid
CAD 120,000 - 160,000
Competitive salary
Meaningful equity
Innovative work culture
Principal AI/ML Engineer
Principal AI/ML Engineer

Jobgether • British Columbia

Hybrid
CAD 142,000 - 284,000
Annual salary with incentives
Remote work arrangement
Health & wellbeing benefits
+1
AI Engineer (Superhuman Mail)
AI Engineer (Superhuman Mail)

Jobgether • Canada

Remote
CAD 128,000 - 210,000
Competitive salary range
Equity
Healthcare coverage
+10
Senior Software Engineer, Verticals (AI Native)
Senior Software Engineer, Verticals (AI Native)

Jobgether • Canada

Remote
CAD 150,000 - 178,000
Equity RSUs
Medical, dental, and supplemental ins.
RRSP & DPSP
+6
Member of Technical Staff, Model Efficiency
Member of Technical Staff, Model Efficiency

Cohere • Montreal

Remote
CAD 100,000 - 130,000
Open and inclusive culture
Cutting-edge AI research collaboration
Weekly lunch stipend
+5
Senior Software Engineer, Data Solutions
Senior Software Engineer, Data Solutions

Jobgether • Toronto

Hybrid
CAD 170,000 - 234,000
Base salary CAD 170k-234k
Remote work stipend
WeWork membership
+3
ML/AI Engineer
ML/AI Engineer

Levio • Toronto

On-site
CAD 110,000 - 150,000
4 weeks vacation starting day one
Flexible working hours
Professional Development Allowance (P<
+7
Machine Learning Engineer
Machine Learning Engineer

Jobgether • Canada

Remote
CAD 90,000 - 120,000
Remote work
Full-time
Hands-on ML