Text & Language Model Evaluation Engineer

S27a; San Mateo (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Deccan AI is seeking a Machine Learning Engineer to build and scale text and language-eval capabilities in the Bay Area. You will design evaluation pipelines, create prompt sets and rubrics, and run evals across frontier and open-source models, working with language experts and SMEs to surface linguistically meaningful failures.

You will own model-run infrastructure, scoring, and reproducible analysis, and help shape Deccan’s stance on multilingual evaluation and domain-specific benchmarks, in a

Qualifications

• Build model-evaluation pipelines and deterministic/non-deterministic scoring systems.
• Identify subtle model failures in language, reasoning, and domain contexts.
• Produce fast studies and research reports on hot topics in ML evals.
• Work with SMEs and language experts without outsourcing judgment.
• Explain methodology, limitations, and findings to technical and non-technical audiences.

Responsibilities

• Build and maintain evaluation pipelines for text, language, and domain-specific model failures.
• Design fast benchmark workflows that deliver high-signal eval reports on a weekly cadence.
• Create prompt sets, rubrics, failure taxonomies, judge/verifier logic, and benchmark charts.
• Run evals across frontier and open-source models; analyze failure patterns with nuance for model teams.
• Work with language experts, SMEs, delivery, and operations to collect prompts and surface meaningful failures.
• Build tooling to speed eval production: model-run scripts, scoring pipelines, data validation, and reproducible notebooks.
• Help define Deccan’s perspective on languages, evals, localized reasoning, and model failure modes.
• Partner with GTM/Engagement Managers so findings support customer conversations without shallow marketing.
• Train operators and analysts to run repeatable eval workflows while preserving quality.

Skills

Python programming

LLM evaluation

Benchmark design

NLP / ML engineering

Data pipelines

Communication

Tools

LLM tooling

Eval harnesses

Notebooks

ML tools

Job description

Deccan AI is seeking a Machine Learning Engineer to build and scale text and language-eval capabilities in the Bay Area. You will design evaluation pipelines, create prompt sets and rubrics, and run evals across frontier and open-source models, working with language experts and SMEs to surface linguistically meaningful failures.

You will own model-run infrastructure, scoring, and reproducible analysis, and help shape Deccan’s stance on multilingual evaluation and domain-specific benchmarks, in a