Emmanuel Adaramoye

MeeBoss; New York (NY); On-site; USD 220,000 - 300,000; Full time

Benefits offered by this job

On-site collaboration in New York City

Full-time employment

On-site collaboration in New York City

Job summary

Honcho is seeking a Machine Learning Engineer specializing in AI evaluations to advance its memory and identity platform for AI agents. You will own the end-to-end evaluation process, designing metrics, pipelines, and experiments to measure improvements in AI identity representations over time, in multi-agent systems.

The role requires production-quality Python, research-driven evaluation design, and collaboration to evolve evaluation methods as the product grows.

Qualifications

• At least 3 years of experience designing and owning LLM evaluation systems.
• Experience designing, building, and running LLM evaluations end-to-end.
• Strong Python programming skills, including production code and testing.
• Hands-on experience developing evaluation frameworks for large language models and AI systems.
• Ability to design novel evaluation methods beyond existing benchmarks.
• Strong analytical and research skills to interpret complex results.
• Experience building reliable data pipelines and evaluation infrastructure.
• A track record of owning technical projects from research to implementation.
• Published research or high-quality open-source work in AI evaluation.
• Must be based in the United States, preferably in the New York metropolitan area.

Responsibilities

• Design and develop novel evaluation methods to measure AI identity representations over time.
• Define evaluation metrics and scoring frameworks reflecting meaningful improvements.
• Build and maintain evaluation pipelines, datasets, labeling workflows, versioning, and automated reruns.
• Analyze evaluation results and system traces to identify issues and opportunities.
• Develop simulation agents at scale to support experiments across many modeled entities.
• Conduct rigorous experiments to assess AI behavior and improve reliability.
• Translate evaluation findings into actionable engineering improvements.
• Own the entire evaluation lifecycle from research questions to documentation.
• Collaborate with the team to continuously improve evaluation methods as technology evolves.

Skills

Python

LLM evaluation

ML evaluation methodologies

Production software development

Testing

End-to-end evaluation

Data pipelines

Education

Degree in Computer Science or related quantitative field

Tools

PyTorch

Hugging Face

Weights & Biases

Hydra

Kafka

SQL

Job description

We are seeking a Machine Learning Engineer specializing in AI evaluations to help advance Honcho, its memory and identity platform for AI agents.

In this role, you will own the evaluation process end-to-end, developing the methods and infrastructure needed to measure whether AI identity representations improve over time. You will work with complex, multi-agent systems where behavior is not always predictable and traditional ground-truth measurements may not be available.

The ideal candidate combines strong machine learning engineering skills, production-quality Python expertise, and a research-driven approach to designing new evaluation methods. You should enjoy investigating system behavior, identifying performance gaps, and building solutions that improve the quality and reliability of AI representations.

Key Responsibilities

• Design and develop novel evaluation methods to measure the quality and improvement of AI identity representations over time.
• Define evaluation metrics and scoring frameworks that reflect meaningful improvements in AI performance.
• Build and maintain evaluation pipelines, testing harnesses, datasets, labeling workflows, versioning systems, and automated reruns.
• Analyze evaluation results and system traces to identify underlying issues and opportunities for improvement.
• Develop simulation agents at scale to support experimentation across large numbers of modeled entities.
• Conduct rigorous experiments to assess AI behavior, validate hypotheses, and improve system reliability.
• Translate evaluation findings into actionable engineering improvements.
• Own the entire evaluation lifecycle, from identifying research questions and building pipelines to running experiments and documenting findings.
• Collaborate closely with the team to continuously improve evaluation methods as the product and underlying technology evolve.

Required Qualifications

• At least 3 years of experience designing and owning LLM evaluation systems.
• Demonstrated experience designing, building, and running LLM evaluations end-to-end.
• Strong Python programming skills, including writing tests, maintaining production code, and managing regular releases.
• Hands‑on experience developing evaluation frameworks for large language models and AI systems.
• Ability to design novel evaluation methods rather than relying exclusively on existing benchmarks.
• Strong analytical and research skills, with the ability to interpret complex results and identify meaningful performance improvements.
• Experience developing reliable data pipelines and evaluation infrastructure.
• A track record of taking ownership of technical projects from initial research questions through implementation and analysis.
• Published research or equivalent high‑quality open‑source work demonstrating technical depth in AI evaluation or related fields.
• Must be based in the United States, with candidates in the New York metropolitan area preferred.

Technical Skills

Required:

• Python
• Large Language Model (LLM) evaluation
• Machine learning evaluation methodologies
• Production software development and testing

Preferred:

• PyTorch
• Hugging Face
• Weights & Biases (W&B)
• Hydra
• Kafka
• SQL
• AI agent evaluation and optimization
• Simulation systems and multi-agent architectures

Preferred Qualifications

• Experience with AI memory systems, identity representations, personalization, or agentic AI optimization.
• First-author or co-author research published at NeurIPS, ICML, ICLR, or comparable conferences.
• High-quality open-source contributions demonstrating research and engineering capabilities.
• A degree in Computer Science, Mathematics, Statistics, Machine Learning, or a related quantitative discipline.

Ideal Candidate Profile

The ideal candidate is a research-oriented machine learning engineer who can combine scientific thinking with practical engineering execution.

You should be comfortable developing new evaluation approaches from scratch, examining detailed system traces, identifying issues that conventional benchmarks may miss, and implementing improvements directly.

Candidates whose experience is limited to running existing benchmarks, building AI applications without rigorous evaluation, or conducting research without practical implementation experience may not be a strong fit.

Compensation and Benefits

• Annual salary: $220,000–$300,000
• Competitive equity
• Full-time employment
• On-site collaboration in New York City

Work Authorization

Applicants must have existing authorization to work in the United States. The company is not considering candidates who require visa sponsorship.