Staff Engineer, LLM Benchmark Platform

Vals AI, Inc.; San Francisco (CA); On-site; USD 140,000 - 220,000; Full time

Benefits offered by this job

Relocation support

Health insurance

Lunch/dinner provided

401K plan

Unlimited PTO

Housing stipend

Job summary

Vals AI, Inc. is seeking exceptional engineers to own and operate the benchmarking platform, spanning Python libraries, a web platform, distributed systems, and cloud infra.

You will shape the stack, drive architecture decisions, and guide teammates through reviews and best practices. You will work across frontend React/TypeScript and backend Python/Django, building scalable evaluations for startups, enterprises, and research labs.

Qualifications

• Strong engineering fundamentals: build and ship quickly with high quality.
• Significant Python experience in a professional setting.
• System design concepts including VMs, containers, load balancers and databases.
• Experience with LLM APIs and related concepts.
• Team collaboration in sprints, Git workflows, and PR reviews.
• Clear and effective communication, including design docs.
• Ability to break down fuzzy problems into clear steps.
• Willingness to relocate to San Francisco.

Responsibilities

• Build distributed systems to run evaluations across models, benchmarks, and machines.
• Deploy cloud infrastructure using IaC, deployment pipelines, logging, monitoring.
• Contribute to internal and external libraries, including our public model library.
• Develop full-stack features using React/TypeScript on the frontend and Python/Django on the backend.
• Perform code and architecture reviews for team members.
• Help establish engineering best practices across the organization.
• Collaborate with the research team to ensure infrastructure meets their needs.

Skills

Strong engineering fundamentals

Python expertise

System design

LLM familiarity

Team collaboration

Communication

Ambiguity tolerance

Iteration speed

In-person SF

Job description

Vals AI, Inc. is seeking exceptional engineers to own and operate the benchmarking platform, spanning Python libraries, a web platform, distributed systems, and cloud infra.

You will shape the stack, drive architecture decisions, and guide teammates through reviews and best practices. You will work across frontend React/TypeScript and backend Python/Django, building scalable evaluations for startups, enterprises, and research labs.