ML Evaluation Engineer for AI Identity & Memory NYC On-site

MeeBoss; New York (NY); On-site; USD 220,000 - 300,000; Full time

Benefits offered by this job

On-site collaboration in New York City

Job summary

Honcho is seeking a Machine Learning Engineer specializing in AI evaluations to advance its memory and identity platform for AI agents. You will own the end-to-end evaluation process, designing metrics, pipelines, and experiments to measure improvements in AI identity representations over time, in multi-agent systems.

The role requires production-quality Python, research-driven evaluation design, and collaboration to evolve evaluation methods as the product grows.

Qualifications

• At least 3 years of experience designing and owning LLM evaluation systems.
• Experience designing, building, and running LLM evaluations end-to-end.
• Strong Python programming skills, including production code and testing.
• Hands-on experience developing evaluation frameworks for large language models and AI systems.
• Ability to design novel evaluation methods beyond existing benchmarks.
• Strong analytical and research skills to interpret complex results.
• Experience building reliable data pipelines and evaluation infrastructure.
• A track record of owning technical projects from research to implementation.
• Published research or high-quality open-source work in AI evaluation.
• Must be based in the United States, preferably in the New York metropolitan area.

Responsibilities

• Design and develop novel evaluation methods to measure AI identity representations over time.
• Define evaluation metrics and scoring frameworks reflecting meaningful improvements.
• Build and maintain evaluation pipelines, datasets, labeling workflows, versioning, and automated reruns.
• Analyze evaluation results and system traces to identify issues and opportunities.
• Develop simulation agents at scale to support experiments across many modeled entities.
• Conduct rigorous experiments to assess AI behavior and improve reliability.
• Translate evaluation findings into actionable engineering improvements.
• Own the entire evaluation lifecycle from research questions to documentation.
• Collaborate with the team to continuously improve evaluation methods as technology evolves.

Skills

Python

LLM evaluation

ML evaluation methodologies

Production software development

Testing

End-to-end evaluation

Data pipelines

Education

Degree in Computer Science or related quantitative field

Tools

PyTorch

Hugging Face

Weights & Biases

Hydra

Kafka

SQL

Job description

Honcho is seeking a Machine Learning Engineer specializing in AI evaluations to advance its memory and identity platform for AI agents. You will own the end-to-end evaluation process, designing metrics, pipelines, and experiments to measure improvements in AI identity representations over time, in multi-agent systems.

The role requires production-quality Python, research-driven evaluation design, and collaboration to evolve evaluation methods as the product grows.