Senior AI Engineer: Evaluation & Optimization (Remote)

Vantaca; Palo Alto (CA); Hybrid; USD 200,000 - 260,000; Full time

Benefits offered by this job

Medical, Dental, Vision

401K with company match

Remote Flexible

Unlimited PTO

Job summary

Vantaca is seeking a Senior AI Engineer, Evaluation & Optimization to own the production AI infrastructure, experiments, and feedback loops for millions of tasks. You will design evaluation methods, benchmark models, and optimization loops across prompts, routing, and configurations, primarily using TypeScript.

The role emphasizes rigorous production testing, cross-provider model experimentation, and cost-aware improvements in a high-volume environment.

Qualifications

• Strong software engineering skills in TypeScript and Node.js.
• Experience building production systems that use LLMs or other machine-learning models.
• Experience operating high-volume or high-reliability services.
• Experience designing and running rigorous model evaluations.
• Strong SQL and data-analysis skills.
• Python experience is useful, but TypeScript is the primary language for production systems.

Responsibilities

• Build production AI infrastructure to measure, operate, and improve millions of AI tasks.
• Create TypeScript services and libraries for model execution, routing, retries, and outputs.
• Instrument workflows for quality, latency, reliability, token usage, and cost.
• Define durable schemas for tasks, evaluations, experiments, and results.
• Design safe rollout, rollback, and comparison of model/workflow changes.
• Develop evaluation suites representing production workloads and predict outcomes.
• Build optimization loops for prompts, configurations, and routing policies.
• Produce dashboards for quality, cost, and AI economics by workflow.

Skills

TypeScript

Node.js

SQL

Data analysis

Experiment design

Production systems

LLM APIs

Tools

OpenAI models

Anthropic models

Google models

Open-weight providers

Job description

Vantaca is seeking a Senior AI Engineer, Evaluation & Optimization to own the production AI infrastructure, experiments, and feedback loops for millions of tasks. You will design evaluation methods, benchmark models, and optimization loops across prompts, routing, and configurations, primarily using TypeScript.

The role emphasizes rigorous production testing, cross-provider model experimentation, and cost-aware improvements in a high-volume environment.