Principal Machine Learning Engineer, Vision AI

Candidate; United States; Hybrid; USD 200,000 - 368,000; Full time

Job summary

Toast seeks a Principal Machine Learning Engineer, Vision AI, to architect and build the platform behind Vision AI products for restaurants. You will design video pipelines, build services to analyze footage, and drive model choices while contributing hands-on code and leading architectural direction.

You will work cross-functionally to deliver vision-powered features, mentor engineers, and champion reliable, privacy-conscious systems.

Qualifications

• 10+ years building software and distributed systems in production.
• Experience designing and shipping video/vision ML platforms.
• Strong cloud and backend experience (AWS, APIs, storage).

Responsibilities

• Architect and build the core Vision AI platform.
• Own services that analyze video and generate observations and metrics for operators.
• Build ML systems, model inference pipelines, evaluation, and monitoring in production.
• Develop customer-facing vision-powered features across Toast products.
• Set technical direction and balance immediate needs with long-term vision.
• Write production code and prototype new ideas hands-on.
• Collaborate with teams on integrations and dependencies.
• Lead with AI-forward engineering and adopt AI tooling.

Skills

Distributed systems

ML systems

Video processing

Cloud AWS

Java/Kotlin

Python

Data pipelines

LLM APIs

Production coding

Mentorship

Education

Bachelor's/Master's CS or related

Tools

Kafka

Pulsar

ffmpeg

GStreamer

Job description

Toast creates technology to help restaurants and local businesses succeed in a digital world, helping business owners operate, increase sales, engage customers, and keep employees happy. Toast powers restaurants of every size, from neighborhood favorites to the largest multi-unit brands in the industry. We're investing in Vision AI to help operators see more of what's happening in their restaurants and act on it, so they can run more efficiently, provide better service, and become even more profitable.

We're looking for a Principal Machine Learning Engineer, Vision AI to architect and build the platform behind these products. You'll have an outsized influence on how Vision AI is built and how it reaches restaurants across Toast. This is a highly hands-on role: you'll spend most of your time designing, building, and shipping, and you'll own the technical direction of the platform.

A day in the life (Responsibilities)

• Architect and build the core platform. Design the video pipeline and other essential components, from how video is captured and ingested to how it's stored, processed, and delivered, so the platform is secure and reliable.
• Own the services that analyze video. Build the services that process footage, run vision models, and turn what they see into observations and metrics restaurants can act on.
• Build ML systems and drive model choices. Select and integrate vision models, including vision-language models, and build the ML systems around them, such as inference pipelines, evaluation, and monitoring of model performance in production. Partner with our ML engineers as models mature.
• Build customer-facing use cases across Toast. Bring vision-powered features to restaurant operators across Toast's products and surfaces.
• Set the technical direction. Make the key architecture decisions for the Vision AI platform, building what's right for today while keeping a clear path to where we're headed.
• Stay hands‑on. Write production code, prototype new ideas, and ship. You'll lead through building, not just reviewing.
• Collaborate across orgs. Partner with teams across Toast on integrations, dependencies, and working models when the work calls for it.
• Lead with AI-forward engineering. Use AI coding tools and agent harnesses as a core part of how you design, build, test, and ship, and help the team adopt workflows that make everyone faster.
• Raise the bar. Mentor engineers, set standards for reliability, security, and privacy, and make the platform easier to build on with every use case.

What you'll need to thrive (Requirements)

• Deep engineering experience. 10+ years building software, including designing and shipping distributed systems and data pipelines in production at scale.
• Backend and cloud expertise. Strong experience with cloud infrastructure (AWS preferred), event-driven systems (e.g. Pulsar or Kafka), storage, and APIs, in Kotlin, Java, or similar languages, plus comfort with Python.
• Experience with ML systems and video. You don't need to be a model researcher, but you understand how models are trained, tuned, and served, and you've built and operated ML systems in production, including inference pipelines, evaluation, and model monitoring. You're also comfortable working with video (streams, codecs, tools like ffmpeg).
• Pragmatic architecture. A track record of designing systems that fit the current stage, avoiding over-engineering while leaving room to grow.
• Zero-to-one experience. Experience taking ambiguous ideas from prototype to production with real customers.
• Hands‑on keyboard. You still write significant production code and enjoy doing it.
• AI-forward ways of working. Daily use of AI coding assistants and agent harnesses (e.g. Claude Code, Cursor), and experience building with LLM or vision‑language model APIs.
• Collaboration and communication. Ability to align engineers, product partners, and other orgs around technical decisions and explain tradeoffs clearly.

This role may not be a fit for you if...

• You want a primarily cross‑org or architecture‑only role. This is a hands‑on keyboard role. You'll spend most of your time building, with cross‑org collaboration when the work requires it.
• You're more attached to the solution than the problem. We want someone obsessed with solving real problems for restaurant operators, who picks the simplest approach that solves them.
• You tend to build for scale before it's needed. We need someone who builds the right solution for where we are today and what customers are telling us, while keeping long‑term direction in mind and knowing when to invest in it.
• You prefer established processes and tooling. This team moves fast and shapes its own ways of working. You'll help create the workflows, tools, and standards rather than inherit them.
• You're uncomfortable with ambiguity. The problems we take on and the best ways to solve them will keep evolving, and priorities will shift as we learn from customers.
• You'd rather be perfect than experiment. We move fast, test ideas with real restaurants, and accept that some of what we build will be thrown away once we learn what works.
• AI‑assisted development isn't part of how you work. We expect AI tools to be central to how we design, build, and ship.

Special Sauce (Nice to Haves)

• Experience building computer vision, video analytics, or camera‑based products.
• Experience with real‑time video pipelines and streaming (e.g. Amazon Kinesis Video Streams, GStreamer, WebRTC).
• Experience with connected device fleets: device provisioning and authentication, remote updates, and constrained networks.
• Experience building privacy‑sensitive systems that handle video or personal data.
• Background in restaurant, retail, hospitality, drive‑thru, or other operationally complex environments.

AI at Toast

At Toast, one of our company values is that we're hungry to build and learn. We believe learning new AI tools empowers us to build for our customers faster, more independently, and with higher quality. We provide these tools across all disciplines, from Engineering and Product to Sales and Support, and are inspired by how our Toasters are already driving real value with them. The people who thrive here are those who embrace changes that let us build more for our customers; it’s a core part of our culture.

Our Total Rewards Philosophy

We strive to provide competitive compensation and benefits programs that help to attract, retain, and motivate the best and brightest people in our industry. Our total rewards package goes beyond great earnings potential and provides the means to a healthy lifestyle with the flexibility to meet Toasters’ changing needs. Learn more about our benefits at https://careers.toasttab.com/toast-benefits.

#LI-REMOTE

The base salary range for this role is listed below. The starting salary will be determined based on skills, experience, and geographic location. In addition to base salary, our total rewards components include cash compensation (overtime, bonus/commissions if eligible), equity, and benefits. You can learn more about how we align pay with local labor markets in our Geographic Pay Zone Philosophy.

Zone A$230,000—$368,000 USDZone B$200,000—$320,000 USDZone C$180,000—$288,000 USD

How Toast Uses AI in its Hiring Process

Throughout the hiring process, our goal is to get to know you. We use AI tools to support our recruiters and interviewers with tasks like note-taking, summarization, and documentation of interviews to ensure they can be fully focused on your conversation. All hiring decisions are made by people. We ask that you complete interviews and assessments on your own, without real‑time AI tools, recording, or transcription bots, unless we tell you otherwise in advance. To learn more: https://careers.toasttab.com/ai-in-hiring

Our Approach to Hybrid Working

We embrace a hybrid work model that fosters in-person collaboration while valuing individual needs. Our goal is to build a strong culture of connection as we work together to empower the hospitality community, regardless of location. Please visit the Locations page on our career site to learn more about our in‑office expectations by region: https://careers.toasttab.com/locations-toast

People Are the Secret Ingredient in Our Recipe for Success

At Toast, our employees are our secret ingredient—when they thrive, we thrive. The restaurant industry brings together people with a wide range of backgrounds, experiences, and perspectives, and we value that same breadth at Toast. We strive to build a culture grounded in authenticity, respect, and humility, where every Toaster has a real opportunity to grow, contribute, and do their best work—raising the bar in delivering exceptional experiences for our customers and each other.

Toast is committed to creating an accessible and inclusive hiring process. As part of this commitment, we strive to provide reasonable accommodations for persons with disabilities to enable them to access the hiring process. If you need an accommodation to access the job application or interview process, please contact [e-mail].

------

For roles in the United States, it is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.