Applied Researcher: Agent Evaluation & Data Pipelines

Human Intuition Inc.; New York (NY); On-site; USD 120,000 - 180,000; Full time

Job summary

Human Intuition Inc. is building the autonomous company and seeks a candidate who can turn workflows, decision histories, and outcomes into datasets and evaluations that distinguish plausible actions from correct ones.

You will design tasks, evaluation environments, and verifiers, build data pipelines with provenance, and create repeatable reports to compare quality, reliability, latency, and cost.

Qualifications

• Strong Python and practical experience building data or ML systems.
• Evidence of designing useful evaluations, experiments, or datasets for models or complex software.
• Sound statistical judgment and the ability to explain what a metric establishes and what it misses.
• Care with data quality, reproducibility, and the handling of business information.
• The ability to learn an unfamiliar domain and communicate clearly with both technical and operational colleagues.

Responsibilities

• Work with domain experts to identify the decisions, constraints, and outcomes that matter in a workflow.
• Design tasks, evaluation environments, rubrics, and verifiers that test complete workflows, including ambiguous inputs and failure recovery.
• Build data pipelines that preserve the relationship between context, decisions, actions, and outcomes, with clear provenance and appropriate access controls.
• Create evaluation splits and review processes that limit leakage, expose distribution shifts, and catch regressions.
• Study agent traces to identify recurring failures and turn the findings into better training data, tools, and reward signals.
• Build repeatable evaluation reports that let researchers and engineers compare quality, reliability, latency, and cost.

Skills

Python

Data/ML systems

Evaluation design

Statistical judgment

Data quality & reproducibility

Domain learning & communication

Job description

Human Intuition Inc. is building the autonomous company and seeks a candidate who can turn workflows, decision histories, and outcomes into datasets and evaluations that distinguish plausible actions from correct ones.

You will design tasks, evaluation environments, and verifiers, build data pipelines with provenance, and create repeatable reports to compare quality, reliability, latency, and cost.