AI Quality Engineer — Model Behavior & Evaluation

Monograph; New York (NY); Hybrid; USD 98,000 - 140,000; Full time

Job summary

A leading technology company in New York is seeking a candidate to take ownership of quality for AI products. This role blends strategy and operations, focusing on context engineering and enhancing AI product behavior. You'll work closely with data, product, and engineering teams to define quality metrics, evaluate models, and shape strategies for AI implementation. Ideal candidates are proactive problem-solvers with experience in AI or LLMs. The position offers a hybrid work environment and a competitive salary range of $98,000 - $140,000 per year.

Qualifications

• Drive ownership of AI product quality and initiatives.
• Explore LLM capabilities and AI's real-world applications.
• Analyze data to inform AI product improvements.

Responsibilities

• Design and test criteria for how AI products behave.
• Engage with production data to identify and debug issues.
• Develop evaluation strategies for quality measurement.
• Collaborate with leading AI labs to launch new models.
• Shape quality narratives with engineering and product teams.
• Build tools that enhance AI product development.

Skills

Driver mentality

Curiosity

Analytical instinct

Comfortable working with data

Clear communication

Experience with LLMs

Job description

A leading technology company in New York is seeking a candidate to take ownership of quality for AI products. This role blends strategy and operations, focusing on context engineering and enhancing AI product behavior. You'll work closely with data, product, and engineering teams to define quality metrics, evaluate models, and shape strategies for AI implementation. Ideal candidates are proactive problem-solvers with experience in AI or LLMs. The position offers a hybrid work environment and a competitive salary range of $98,000 - $140,000 per year.