LLM Evaluation & Prompt Testing Engineer

HumanitApp; San Francisco (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Turing, based in San Francisco, seeks Domain Experts to support evaluation and testing of advanced LLMs. You will craft domain-specific prompts, assess responses for accuracy and reasoning, and design edge cases to reveal model weaknesses.

This is a highly technical, detail-oriented role requiring strong analytical skills and experience with AI evaluation. The ideal candidate will have a Master’s degree or higher and 3+ years in research, academia, or industry, with familiarity in prompt

Qualifications

• Master’s degree or higher in a relevant field.
• At least three years of professional, research, or teaching experience.
• Strong analytical and critical-thinking abilities.
• Exceptional attention to detail.
• Experience with LLMs or Generative AI preferred.
• Familiarity with prompt engineering, AI evaluation, or model testing.

Responsibilities

• Create complex, domain-specific prompts to test LLM capabilities.
• Evaluate AI-generated responses for factual accuracy, reasoning, relevance, and overall quality.
• Identify factual errors, logical inconsistencies, knowledge gaps, and model weaknesses.
• Develop challenging test cases and edge cases that expose model limitations.
• Provide clear, detailed, and evidence-based feedback to improve model performance.

Skills

Domain expertise

LLMs / Generative AI

Prompt engineering

AI evaluation

Stress testing

Critical thinking

Attention to detail

Education

Master's degree or higher in a relevant field

Job description

Turing, based in San Francisco, seeks Domain Experts to support evaluation and testing of advanced LLMs. You will craft domain-specific prompts, assess responses for accuracy and reasoning, and design edge cases to reveal model weaknesses.

This is a highly technical, detail-oriented role requiring strong analytical skills and experience with AI evaluation. The ideal candidate will have a Master’s degree or higher and 3+ years in research, academia, or industry, with familiarity in prompt