Remote Applied AI Engineer: Evaluation & Cost Optimization

Velixo; Montreal (administrative region); Hybrid; CAD 120,000 - 190,000; Full time

Job summary

Velixo is seeking an engineering role focused on measurement-first development for Velixo Intelligence. You will build evaluation and cost-tracking systems to tell us whether model changes improve the product and reduce action costs without sacrificing quality.

This role emphasizes data-driven evaluation of LLM features, with hands-on work on Python/TypeScript, observed token costs, and performance tradeoffs. Fully remote in Canada, with priority for Quebec candidates.

Qualifications

• Substantial production experience with LLM APIs and evaluation tooling.
• Ability to build and maintain an eval harness and regression suites.
• Familiarity with token accounting, context/window management, and caching strategies.

Responsibilities

• Build and maintain regression suites for GIQL query generation, writeback correctness, and tool selection.
• Define measurable success criteria for each action type and ensure traceability.
• Run structured evaluations before every model upgrade or prompt change; deliver ship/no-ship recommendations.
• Track quality and cost metrics over time to detect degradation before customers.

Skills

LLM APIs

Eval harness

Langfuse

promptfoo

Python

TypeScript

Data analysis

Cost accounting

Tools

Langfuse

promptfoo

Job description

Velixo is seeking an engineering role focused on measurement-first development for Velixo Intelligence. You will build evaluation and cost-tracking systems to tell us whether model changes improve the product and reduce action costs without sacrificing quality.

This role emphasizes data-driven evaluation of LLM features, with hands-on work on Python/TypeScript, observed token costs, and performance tradeoffs. Fully remote in Canada, with priority for Quebec candidates.