LLM Evaluation Domain Expert · Turing Turing · Varies · remote · 4w ago Varies 4w ago

Benture

San Francisco (CA)

Hybrid

USD 90,000 - 150,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Turing, based in San Francisco, is hiring Domain Experts (SMEs) to support evaluation, testing and improvement of advanced Large Language Models (LLMs). You will use your subject-matter expertise to assess AI-generated responses, design challenging prompts and provide evidence-based feedback to enhance model accuracy and reliability.

Ideal candidates hold a Master’s or higher, have at least three years of professional or academic experience, and excel in precise written English and analytical

Qualifications

  • Master's degree or higher in a relevant field.
  • At least three years of professional, research or teaching experience.
  • Excellent written English communication skills.

Responsibilities

  • Create complex, domain-specific prompts to test LLM capabilities.
  • Evaluate AI-generated responses for factual accuracy, reasoning, relevance and overall quality.
  • Identify factual errors, logical inconsistencies, knowledge gaps and model weaknesses.
  • Develop challenging test cases and edge cases that expose model limitations.
  • Provide clear, detailed, evidence-based feedback to improve model performance.

Skills

Excellent written English
Analytical thinking
Attention to detail

Education

Master's degree or higher in a relevant field

Job description

Domain Expert, LLM Evaluation | Pay: Varies | Remote

About the Role

Turing, based in San Francisco, works with frontier AI labs and enterprises on AI research and deployment. The company is hiring Domain Experts (SMEs) to support the evaluation, testing and improvement of advanced Large Language Models (LLMs). You will apply your subject-matter expertise to assess AI-generated responses, identify knowledge gaps, design challenging prompts, and improve model accuracy, reliability and reasoning.

What the Work Involves
  • Create complex, domain-specific prompts to test LLM capabilities.
  • Evaluate AI-generated responses for factual accuracy, reasoning, relevance and overall quality.
  • Identify factual errors, logical inconsistencies, knowledge gaps and model weaknesses.
  • Develop challenging test cases and edge cases that expose model limitations.
  • Provide clear, detailed, evidence-based feedback to improve model performance.
Minimum Qualifications
  • Master's degree or higher in a relevant field.
  • At least three years of professional, research or teaching experience.
  • Excellent written English communication skills.
  • Strong analytical and critical-thinking abilities.
  • Exceptional attention to detail.
Preferred Qualifications
  • Experience working with LLMs or Generative AI.
  • Familiarity with prompt engineering, AI evaluation or model testing.
  • Experience designing challenging prompts, identifying edge cases or stress-testing AI models.
Ideal Candidate

Curious, detail-oriented and deeply knowledgeable in your field. You enjoy solving complex problems, challenging assumptions, and identifying subtle factual or reasoning errors.

Pay, Schedule and Location

The listing does not specify pay, working hours, commitment or location restrictions.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote LLM Evaluation Domain Expert
Remote LLM Evaluation Domain Expert

Benture • San Francisco (CA)

Hybrid
USD 90,000 - 150,000
Tooling Engineer
Tooling Engineer

Turing • United States

Remote
USD 120,000 - 150,000
LLM Evaluation & Prompt Testing Engineer
LLM Evaluation & Prompt Testing Engineer

HumanitApp • San Francisco (CA)

On-site
USD 120,000 - 180,000
LLM Prompting and Evaluation Lead
LLM Prompting and Evaluation Lead

Turing • United States

Remote
USD 120,000 - 150,000
LLM Evaluation Specialist
LLM Evaluation Specialist

OpenTrain AI, Inc. • United States

Hybrid
USD 45,000 - 65,000
LLM Model Response Evaluation
LLM Model Response Evaluation

Upwork • Maryland City (MD)

Remote
USD 28,000 - 55,000
ML Engineer - LLM & Agentic Evaluations
ML Engineer - LLM & Agentic Evaluations

Turing • United States

Remote
USD 34,000 - 55,000
Fully remote work
LLM Evaluation Scenario Architect
LLM Evaluation Scenario Architect

Mindrift • Mississippi

Remote
USD 90,921 - 129,494
Competitive hourly rates
Flexible work hours
Valuable experience for portfolio
LLM Model Response Evaluation
LLM Model Response Evaluation

Lifted (Upwork) • United States

Remote
USD 34,000 - 83,000
Flexible remote work
Variable workload
No guaranteed hours
Tooling Engineer
Tooling Engineer

HumanitApp • San Francisco (CA)

On-site
USD 120,000 - 180,000