Evaluation & Measurement Lead, AI-Assisted Validation

Civilization Labs

United States

Remote

USD 76,000 - 91,000

Part time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Civilization Labs is seeking an Evaluation & Measurement Lead to own AI-assisted validation. This hands-on, high-judgment role starts as a two-month project with a path to ongoing work. Remote work with US time zones preferred; NYC optional.

You’ll design evaluation criteria, build test sets, run blind comparisons, and deliver clear findings to executives. Half-time to start, with potential growth as the product matures.

Qualifications

  • Ability to define constructs and operationalize them
  • Experience designing evaluation protocols from scratch
  • Comfort with both machine learning and classical statistics

Responsibilities

  • Define what good means and how it will be measured
  • Build a small golden case set and rubric that can be used consistently
  • Design a fair baseline comparison with honest evaluation conditions
  • Set how the system will be evaluated, scored, and who can judge
  • Run blind comparisons and report what the data support or do not support
  • Test the methods as well as the system to reveal where assumptions break
  • Identify which components drive good results and where gaps exist
  • Create a repeatable method and a clear specification for engineers
  • Grow with the organization as needs evolve
  • Roll up your sleeves: design, run, and inform product storytelling

Skills

Evaluation design
Rubric design
Inter-rater reliability
Small-sample honesty
Applied AI evaluation
Program evaluation
Psychometrics
Research methods
Communicating to executives
Construct definition

Job description

Evaluation & Measurement Lead, AI-Assisted Validation
Company: Civilization Labs | Location: Remote, US time zones preferred | NYC optional | Time Commitment: Likely half-time to start | Initial two-month project with a path to an ongoing and growing role | Reports to: Managing Director, working closely with the CEO and engineering leads

Civilization Labs

Most important decisions are still made with a mix of instinct, politics, incomplete evidence, and rationalization. Civilization Labs exists to change that. We're building decision-grade validation tools and systems that help teams pressure-test important choices with clarity, evidence, and explicit uncertainty, not narrative or false confidence. Our aim is simple to say and hard to build: turn consequential decisions into auditable outputs people can actually use and stand behind.

The Opportunity

We've built a system that pressure-tests consequential decisions. It surfaces the assumptions inside a plan and says which ones may break under pressure. People who’ve read the memos and reports respond to its insights.

We need to understand the efficacy of the system and of the methods inside it: which parts are carrying the weight, which are not, and how the whole compares against the alternatives a serious buyer would weigh. That work starts with specific cases and real decisions rather than with a general-purpose benchmark, and builds toward something generalizable. You’d design the experiments, run them, and tell us what they support.

This is a hands-on, high-judgment role for someone who thinks measurement is the interesting problem to tackle within new technology.

What you’ll do
  • Define what good means. Turn a construct such as better judgment into criteria two reviewers can score the same way.
  • Build the test set. Assemble a small golden case set and a rubric that can survive disagreement, then calibrate it.
  • Design a comparison. Specify the baseline, keep the conditions honest, and work with us to calibrate the system to its optimal performance so the test is fair.
  • Set the evaluation protocol. How the system gets evaluated, how it gets scored, who is qualified to judge, and what we're setting ourselves up to learn.
    Run it and report it. Execute blind comparisons and tell us what they do and do not support, including where the sample is too small to carry the claim.
    Test the methods, not only the system. Take an individual analytical framework, turn it into claims that could fail, design the most discriminating test available, and report what survives.
  • Find out what is doing the work. Determine which components of the system contribute to a good result, in what combination, and whether anything in the sequence is not earning its place.
  • Leave something repeatable behind. A written method someone else can run without you, a plain statement of what we can legitimately say based on what we found, and a clear spec for what our engineers should build so this runs as a process rather than a one-off.
  • Grow with the organization. We're small and moving fast, and the questions worth measuring will change as we learn. The scope above is where you'd start, and we'd expect it to shift toward whatever we need to know most.
  • This is a roll-up-your-sleeves seat: you design the method, and you run it, and the findings inform how we describe the product to the market.
Who you are
  • Your first instinct on hearing "it works" is to ask compared to what, judged by whom.
  • You have designed an evaluation from scratch.
  • You find the definitional problem interesting. Operationalizing something fuzzy without flattening it into a proxy that misses the point is the core skill we're looking for.
  • You're fluent in construct definition, rubric design, blind protocol, inter-rater reliability, and small-sample honesty. Applied AI evaluation, program evaluation, psychometrics, and research methods are all plausible paths in.
  • You're methodology-agnostic in the right way. Machine learning and classical statistics are both on the table, and you choose between them based on the problem in front of you rather than on what you were trained in.
  • You can specify what engineers should build without needing to build it yourself (though building it yourself is a plus).
  • You would deliver a finding somebody did not want. If a comparison shows no advantage on certain measures, we need to hear it quickly and directly.
  • You write clearly for non-specialists. Your findings will be read by a CEO and referenced in commercial conversations, so precision has to survive translation into plain language.
  • You want a defensible answer in weeks, not a publication in a year.
Our Culture

The biggest calls people make, the strategies, plans, and decisions they can't easily walk back, still often get committed on narrative and gut feeling, and the cost lands on real people. We're here to help pressure-test that thinking before anyone commits. That belief shapes the product, and it shapes our culture: we want to do serious work with people who trust evidence over narrative, chase what's true even when it's inconvenient, and care about the consequences of what they build.

  • Integrity: Trust starts with honesty, transparency, and respect for reality.
  • Accountability: We do what we say we'll do, and we take responsibility when we miss.
  • Collaboration: We solve hard problems together and value clear, generous communication.
  • Intellectual Curiosity: We stay interested, skeptical, and eager to improve how we think.
  • Social Impact: We want our work to leave people, institutions, and society more clear-eyed than we found them, especially on the choices that matter most.
Compensation

This starts as a defined two-month project scoped to the deliverables above, though the scope may continue into the future. We'd rather set that ongoing shape from what the first two months produce than guess at it now.

  • Fixed fee: $7,500 per month, likely half-time.
  • We're looking for someone who wants to grow with this, as we expect both the hours and the scope to grow over time.
  • We'll also consider an hourly structure for candidates who prefer it, against the same defined deliverable set.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Research Scientist, Learning & Evaluation
AI Research Scientist, Learning & Evaluation

Studyfetch • Beverly Hills (CA)

On-site
USD 150,000 - 210,000
Medical, Dental, Vision (100% employer
75% dependent coverage
401(k) with employer matching
+2
Lead Product Engineer
Lead Product Engineer

remotehiro • San Francisco (CA)

Hybrid
USD 135,000 - 165,000
Health/dental/vision insurance
403(b) retirement plan
Generous PTO
+1
Senior Frontend Engineer — Weval & Democratic AI Platforms
Senior Frontend Engineer — Weval & Democratic AI Platforms

remotehiro • San Francisco (CA)

Hybrid
USD 135,000 - 165,000
Health/dental/vision insurance
403(b) retirement plan
Generous PTO
+1
Lead Product Engineer
Lead Product Engineer

Collective Intelligence Project • San Francisco (CA)

Hybrid
USD 128,000 - 173,000
Health benefits
Dental plan
Vision plan
+2
Research Engineer
Research Engineer

Firecrawl • San Francisco (CA)

Hybrid
USD 210,000 - 275,000
Equity
Generous PTO
Parental leave
+6
Research Methods Lead
Research Methods Lead

StudyFetch • Beverly Hills (CA)

On-site
USD 120,000 - 170,000
Medical, Dental & Vision
401(k) with employer matching
Daily team dinner in-office
+1
Conceptual Reasoning Researcher (Contract)
Conceptual Reasoning Researcher (Contract)

Trajectory Labs, PBC • United States

Hybrid
USD 138,000 - 207,000
Software Engineer III — Evaluation
Software Engineer III — Evaluation

Education Advanced, Inc. • Tyler (TX)

Remote
USD 130,000 - 175,000
Forward Deployed AI Engineer
Forward Deployed AI Engineer

Malleable • United States

Remote
USD 150,000 - 180,000
Member of Technical Staff - Evaluations
Member of Technical Staff - Evaluations

Liquid AI • San Francisco (CA)

Hybrid
USD 150,000 - 190,000
Equity
Unlimited PTO
Refill Days
+2