AI Evaluation & Benchmark Engineer

Themeframer; San Francisco (CA); On-site; USD 180,000 - 230,000; Full time

Benefits offered by this job

Health & dental covered

Equity ownership

Remote-first stipend

Flexible time off

Job summary

Themeframer, a San Francisco–based AI & ML company, is hiring for a full-time role in AI & Machine Learning Generative Systems. You will build automated sandbox testing frameworks and ensure models meet benchmarks, with a compensation range of $180,000–$230,000 USD plus equity.

You will work on 4+ years of Python, CI/CD, and evaluation tooling, with remote-first setup and a learning budget. The role emphasizes testing across large test suites and collaboration with research teams.

Qualifications

• 4+ years experience in Python, automated testing frameworks, and LLM evaluation tooling.
• Strong knowledge of continuous integration (CI) and containerized execution.

Responsibilities

• Build automated sandbox runners that execute and verify generated code across 10,000+ unit test suites.
• Create synthetic benchmark datasets for Next.js, Tailwind, and full-stack API generation.
• Identify model failure modes and partner with research teams on dataset filtering and RL feedback.

Skills

Python

CI/CD

LLM evaluation tooling

Automated testing frameworks

Tools

Docker

PyTest

PostgreSQL

FastAPI

GitHub Actions

Job description

AI & Machine Learning Generative Systems Hybrid · San Francisco, CA Full-Time

$180,000 – $230,000 USD · 0.15% – 0.3% Equity

Ensure Theme Framer models maintain the highest benchmarked accuracy in the industry by building automated sandbox testing frameworks.

What You Will Do

• Build automated sandbox runners that execute and verify generated code across 10,000+ unit test suites.
• Create synthetic benchmark datasets for Next.js, Tailwind, and full-stack API generation.
• Identify model failure modes and partner with research teams on dataset filtering and RL feedback.

What We Are Looking For

• 4+ years experience in Python, automated testing frameworks, and LLM evaluation tooling.
• Strong knowledge of continuous integration (CI) and containerized execution.

Bonus Qualifications

• Experience with HumanEval, SWE-bench, or custom code evaluation suites.

Primary Technology Stack

Python Docker PyTest PostgreSQL FastAPI GitHub Actions

100% Employer-Covered Health & Dental

Meaningful Equity Ownership

Remote-First Setup Stipend ($2,500)

Flexible Time Off & Mental Health Days

Annual Company Engineering Retreats

$2,000 / Year Continuous Learning Budget

Shape The Future Of Autonomous Software

Explore all open roles across compiler infrastructure, generative AI, and design systems.