Senior Software Domain Expert — GenAI QA & Benchmarks

Mercor; San Francisco (CA); Hybrid; USD 180,000 - 240,000; Full time

Job summary

Mercor seeks a senior engineering/domain expert to join a leading AI lab’s GenAI team. You will vet engineering knowledge work tasks, write instruction specs, and build benchmarks to measure model improvement. This is a hybrid Bay Area role with on-site collaboration several days weekly.

You will work with Cincinnatus LLC as the employer of record, contribute to high-impact engineering benchmarks, and help translate tacit judgment into explicit, teachable criteria.

Qualifications

• Bachelor's degree or higher in computer science, software engineering, or closely related engineering discipline.
• MS or PhD strongly preferred.
• Genuine specialization in at least one domain (e.g., distributed systems, ML systems, developer tooling).
• Senior-level progression with ownership of systems and design decisions.
• Hands-on experience with large language models in professional work.

Responsibilities

• Data QA and reviews of engineering knowledge work tasks and model outputs.
• Write instruction specs and golden solutions for engineering problems.
• Design benchmarks and domain-specific evaluation sets.
• Calibration with client researchers to maintain consistent standards.

Skills

Data QA

Instruction specs

Benchmarks design

Calibration

AI fluency

Technical communication

Availability 40h/wk

Education

Bachelor's degree in computer science/engineering or related field

MS or PhD strongly preferred

Job description

Mercor seeks a senior engineering/domain expert to join a leading AI lab’s GenAI team. You will vet engineering knowledge work tasks, write instruction specs, and build benchmarks to measure model improvement. This is a hybrid Bay Area role with on-site collaboration several days weekly.

You will work with Cincinnatus LLC as the employer of record, contribute to high-impact engineering benchmarks, and help translate tacit judgment into explicit, teachable criteria.