Senior ML Specialist (Fully Remote)

Jobgether SRL; United States; Remote; USD 150,000 - 210,000; Full time

Benefits offered by this job

Remote work with high autonomy

Job summary

Jobgether is seeking an AI Evaluation & Development lead to design and maintain evaluation sets, calibrate judges, and benchmark model performance. You will build classifiers and fine-tuned models for routing and detection, and drive prompt/context engineering strategies in collaboration with product teams.

You will work with a remote, autonomous team to advance agentic AI products, investigate hallucinations and drift, and uphold rigorous coding standards across the project lifecycle.

Qualifications

• Experience evaluating ML models using benchmarks.
• Ability to apply statistical analysis to model decisions.
• Familiarity with prompt/context engineering strategies.

Responsibilities

• Evaluate and select models using benchmarks and LLM-as-judge frameworks.
• Apply statistical analysis to support model decisions.
• Design offline evaluation sets, judge calibration, and regression benchmarks.
• Build classification and fine-tuned models for routing, categorization, and detection.
• Develop agentic AI products and prompt/context engineering strategies.
• Partner with data engineering, software engineering, and product teams.
• Investigate model hallucinations, drift, and prompt sensitivity from first principles.
• Maintain high engineering standards for code quality, testing, and documentation.

Skills

Model evaluation

Statistical analysis

Prompt engineering

Collaboration with engineering teams

Code quality & testing

Tools

LLM benchmarking

Job description

• Evaluate and select models from major and emerging providers using rigorous benchmarks and LLM-as-judge frameworks.
• Apply statistical analysis to make model decisions defensible.

AI Evaluation & Development:

• Design and maintain offline evaluation sets, judge calibration, and regression benchmarks.
• Build classification and fine-tuned models for routing, categorization, and detection.
• Develop agentic AI products and prompt/context engineering strategies.

Technical Leadership & Research:

• Partner with data engineering, software engineering, and product teams.
• Investigate model hallucinations, drift, and prompt sensitivity from first principles.
• Maintain high engineering standards for code quality, testing, and documentation.

Jobgether

A production agentic AI platform that builds and deploys advanced machine learning systems. The team is collaborative and values innovation, offering a remote work environment with high autonomy.