Evaluation & Measurement Lead, AI-Assisted Validation
Company: Civilization Labs | Location: Remote, US time zones preferred | NYC optional | Time Commitment: Likely half-time to start | Initial two-month project with a path to an ongoing and growing role | Reports to: Managing Director, working closely with the CEO and engineering leads
Civilization Labs
Most important decisions are still made with a mix of instinct, politics, incomplete evidence, and rationalization. Civilization Labs exists to change that. We're building decision-grade validation tools and systems that help teams pressure-test important choices with clarity, evidence, and explicit uncertainty, not narrative or false confidence. Our aim is simple to say and hard to build: turn consequential decisions into auditable outputs people can actually use and stand behind.
The Opportunity
We've built a system that pressure-tests consequential decisions. It surfaces the assumptions inside a plan and says which ones may break under pressure. People who’ve read the memos and reports respond to its insights.
We need to understand the efficacy of the system and of the methods inside it: which parts are carrying the weight, which are not, and how the whole compares against the alternatives a serious buyer would weigh. That work starts with specific cases and real decisions rather than with a general-purpose benchmark, and builds toward something generalizable. You’d design the experiments, run them, and tell us what they support.
This is a hands-on, high-judgment role for someone who thinks measurement is the interesting problem to tackle within new technology.
What you’ll do
- Define what good means. Turn a construct such as better judgment into criteria two reviewers can score the same way.
- Build the test set. Assemble a small golden case set and a rubric that can survive disagreement, then calibrate it.
- Design a comparison. Specify the baseline, keep the conditions honest, and work with us to calibrate the system to its optimal performance so the test is fair.
- Set the evaluation protocol. How the system gets evaluated, how it gets scored, who is qualified to judge, and what we're setting ourselves up to learn.
Run it and report it. Execute blind comparisons and tell us what they do and do not support, including where the sample is too small to carry the claim.
Test the methods, not only the system. Take an individual analytical framework, turn it into claims that could fail, design the most discriminating test available, and report what survives. - Find out what is doing the work. Determine which components of the system contribute to a good result, in what combination, and whether anything in the sequence is not earning its place.
- Leave something repeatable behind. A written method someone else can run without you, a plain statement of what we can legitimately say based on what we found, and a clear spec for what our engineers should build so this runs as a process rather than a one-off.
- Grow with the organization. We're small and moving fast, and the questions worth measuring will change as we learn. The scope above is where you'd start, and we'd expect it to shift toward whatever we need to know most.
- This is a roll-up-your-sleeves seat: you design the method, and you run it, and the findings inform how we describe the product to the market.
Who you are
- Your first instinct on hearing "it works" is to ask compared to what, judged by whom.
- You have designed an evaluation from scratch.
- You find the definitional problem interesting. Operationalizing something fuzzy without flattening it into a proxy that misses the point is the core skill we're looking for.
- You're fluent in construct definition, rubric design, blind protocol, inter-rater reliability, and small-sample honesty. Applied AI evaluation, program evaluation, psychometrics, and research methods are all plausible paths in.
- You're methodology-agnostic in the right way. Machine learning and classical statistics are both on the table, and you choose between them based on the problem in front of you rather than on what you were trained in.
- You can specify what engineers should build without needing to build it yourself (though building it yourself is a plus).
- You would deliver a finding somebody did not want. If a comparison shows no advantage on certain measures, we need to hear it quickly and directly.
- You write clearly for non-specialists. Your findings will be read by a CEO and referenced in commercial conversations, so precision has to survive translation into plain language.
- You want a defensible answer in weeks, not a publication in a year.
Our Culture
The biggest calls people make, the strategies, plans, and decisions they can't easily walk back, still often get committed on narrative and gut feeling, and the cost lands on real people. We're here to help pressure-test that thinking before anyone commits. That belief shapes the product, and it shapes our culture: we want to do serious work with people who trust evidence over narrative, chase what's true even when it's inconvenient, and care about the consequences of what they build.
- Integrity: Trust starts with honesty, transparency, and respect for reality.
- Accountability: We do what we say we'll do, and we take responsibility when we miss.
- Collaboration: We solve hard problems together and value clear, generous communication.
- Intellectual Curiosity: We stay interested, skeptical, and eager to improve how we think.
- Social Impact: We want our work to leave people, institutions, and society more clear-eyed than we found them, especially on the choices that matter most.
Compensation
This starts as a defined two-month project scoped to the deliverables above, though the scope may continue into the future. We'd rather set that ongoing shape from what the first two months produce than guess at it now.
- Fixed fee: $7,500 per month, likely half-time.
- We're looking for someone who wants to grow with this, as we expect both the hours and the scope to grow over time.
- We'll also consider an hourly structure for candidates who prefer it, against the same defined deliverable set.