Machine Learning Eval Engineer

Reducto, Inc.; San Francisco (CA); On-site; Full time

Benefits offered by this job

Free daily lunch

Reimbursed transportation

Generous health insurance

Health and wellness budget

Parental leave policies

Job summary

A cutting-edge AI company located in San Francisco is seeking an ML Eval Engineer to enhance model evaluations and ensure quality metrics. This role involves designing benchmarks, collaborating with teams to identify model weaknesses, and developing automated processes. Candidates should possess strong Python skills, a passion for solving complex problems, and a background in AI or data infrastructure. The position is in-person and offers a dynamic work environment focused on growth and impact.

Qualifications

• Strong Python skills to build clean technical solutions.
• Experience in AI/ML, data infrastructure, or document understanding.
• Background in product thinking and building user-facing interfaces.

Responsibilities

• Design and maintain evaluation benchmarks for models.
• Develop metrics and workflows to identify failure modes.
• Collaborate with ML engineers to improve models.

Skills

Python programming

Data infrastructure

Problem solving

Collaboration

Tools

AWS S3

Job description

Reducto helps AI teams ingest real-world enterprise data with state-of-the-art accuracy.

Most enterprise data, from financial statements to health records, is locked in unstructured file formats like PDFs and spreadsheets. We train vision models to read those documents the way a human would, enabling teams to build products, train models, and automate processes at scale.

We’ve grown rapidly, increasing revenue 7x year over year and partnering with hundreds of companies, from leading AI teams like Harvey, Vanta, and Scale, to enterprise customers across FAANG and top trading firms.

Reducto has raised over $100M from world-class investors including a16z, Benchmark, and First Round Capital.

The Opportunity

As an ML Eval Engineer, you’ll play a key role in building the evaluation systems and benchmarks that make Reducto’s models better over time. You’ll collaborate closely with our ML, platform, and GTM teams to identify model weaknesses, design strong benchmarks, and create metrics and tooling that surface new failure modes as we scale. This is a high-impact role where you’ll help define how model quality is measured at Reducto and shape the systems we use to improve it.

What You’ll Do

Design, build, and maintain evaluation benchmarks that reveal where our models perform well and where they fail.

Develop metrics, heuristics, and workflows to automatically identify new failure modes across large and messy real-world datasets.

Partner closely with other ML engineers to turn evaluation insights into model improvements and better training priorities.

Work hands‑on with unstructured enterprise data, including PDFs, spreadsheets, and other difficult document formats, to uncover edge cases and hard examples.

Build lightweight internal and user‑facing tools, including simple interfaces in Python frameworks like Flask, to help teams inspect results, analyze model behavior, and communicate evaluation outcomes.

Collaborate with customers and internal teams to understand real‑world data needs and create bespoke benchmarks that highlight Reducto’s strengths.

You’ll Thrive Here If You:

Hold yourself to a high bar for quality and precision.

Enjoy solving complex problems and building from first principles.

Have strong Python skills and can independently build clean, reliable technical solutions. Bonus points for product and front‑end experience!

Are comfortable working with data infrastructure such as AWS S3 and OLAP or analytics systems like Tinybird.

Love getting your hands dirty with unstructured data and chasing down difficult failure cases.

Operate well in fast‑changing, high‑growth environments.

Collaborate effectively across technical and non‑technical teams.

Take full ownership from strategy through execution.

Have experience at an early‑stage or high‑growth startup.

Have some background in product thinking and can build simple, polished user‑facing interfaces.

Are comfortable working directly with customers to understand their workflows and data needs.

Have experience in AI/ML, data infrastructure, enterprise software, or document understanding systems.

Care deeply about combining technical excellence with business impact.

This is an in person role at our office in SF. We’re an early stage company which means that the role requires working hard and moving quickly. Please only apply if that excites you.

About Reducto

Nearly 80% of enterprise data is in unstructured formats like PDFs

PDFs are the status quo for enterprise knowledge in nearly every industry. Insurance claims, financial statements, invoices, and health records are all stored in a structure that’s simply impractical for use in digital workflows. This isn’t an inconvenience—it’s a critical bottleneck that leads to dozens of wasted hours every week.

Traditional approaches fail at reliably extracting information in complex PDFs. OCR and even more sophisticated ML approaches work for simple text documents but are unreliable for anything more complex. Text from different columns are jumbled together, figures are ignored, and tables are a nightmare to get right. Overcoming this usually requires a large engineering effort dedicated to building specialized pipelines for every document type you work with.

Reducto breaks document layouts into subsections and then contextually parses each depending on the type of content. This is made possible by a combination of vision models, LLMs, and a suite of heuristics we built over time. Put simply, we can help you:

Accurately extract text and tables even with non‑standard layouts

Automatically convert graphs to tabular data and summarize images in documents

Extract important fields from complex forms with simple, natural language instructions

Build powerful retrieval pipelines using Reducto’s document metadata

Intelligently chunk information using the document’s layout data

Benefits at Reducto

• Lunch: Receive a free lunch to eat with your teammates daily at the office
• Reimbursed Transportation: Provide us with your receipts and we’ll take care of the costs
• Insurance: Generous health insurance covering medical, dental, and vision.
• Health and Wellness Budget: We provide up to $150/mo reimbursement for health and wellness spending, such as gym memberships, fitness classes, or similar.
• Parental Leave: Work with us to build a leave schedule that works for you and your family

Reducto is an Equal Opportunity Employer committed to diversity and inclusion in the workplace. All qualified applicants will receive consideration for employment without regard to sex, race, color, age, national origin, religion, physical and mental disability, genetic information, marital status, sexual orientation, gender identity/assignment, citizenship, pregnancy or maternity, protected veteran status, or any other status prohibited by applicable national, federal, state or local law.