Member of Technical Staff, Model Routing

Dipp AI Technologies, Inc.; New York (NY); On-site; USD 180,000 - 230,000; Full time

Benefits offered by this job

Equity

Health benefits

401(k) match

Hardware budget

Compute credits

Publish methodology

Job summary

Dipp AI Technologies seeks a senior ML/engineering leader to own the dynamic model routing layer. You will classify tasks, govern evaluation across frontier, open-weight, and private models, and ensure budgets and hard stops are enforced before inference.

You will also govern model entry, shadow-test new models, and document decisions for auditability. The role demands hands-on production experience with LLM systems, strong ML/systems background, and fluency in Python with a systems language for

Qualifications

• Hands-on production experience with LLM systems beyond prototypes.
• Experience building evaluation infrastructure that teams trust.
• Strong applied ML or systems background with rigorous measurement habits.

Responsibilities

• Build task classification and routing policy across frontier, open-weight, and private models.
• Own the evaluation harness that scores models per task class on quality, latency, and cost.
• Implement pre-inference cost governance: budgets, ceilings, degradation paths, and hard stops.
• Enforce enterprise data-boundary rules in routing so no restricted payload reaches an ineligible provider.
• Build shadow evaluation and staged promotion for new models entering the fleet.
• Instrument routing decisions so each one is explainable in the audit ledger.

Skills

LLM systems

ML engineering

Evaluation infrastructure

Python proficiency

Systems programming

Inference economics

Benchmark skepticism

Education

BS/MS/PhD in CS/ML/Statistics

Equivalent applied experience

Tools

LLM serving platforms

Evaluation tooling

Job description

Own dynamic model routing: the right model for each task, under a hard cost and policy ceiling.

Dipp AI's strategy is decoupled orchestration — no single frontier model, but the right model for each task under governed cost, latency, and data-boundary constraints. You will own the routing layer that makes that real.

You will build task classification, capability profiles, evaluation harnesses, and the cost governor that enforces spend ceilings before inference happens rather than after the invoice arrives. Routing decisions must be explainable and reproducible, because they end up in the audit ledger alongside the action they enabled.

You will also own how new models enter the fleet: how they are evaluated, gated, shadow-tested, and promoted or rejected on evidence.

What you will do

• Build task classification and routing policy across frontier, open-weight, and private models.


• Own the evaluation harness that scores models per task class on quality, latency, and cost.


• Implement pre-inference cost governance: budgets, ceilings, degradation paths, and hard stops.


• Enforce enterprise data-boundary rules in routing so no restricted payload reaches an ineligible provider.


• Build shadow evaluation and staged promotion for new models entering the fleet.


• Instrument routing decisions so each one is explainable in the audit ledger.

What we look for

• Hands-on production experience with LLM systems beyond prototypes: serving, evaluation, and cost control.


• Strong applied ML or systems background with rigorous measurement habits.


• Experience building evaluation infrastructure that teams actually trusted.


• Fluency in Python plus a systems language for the serving path.


• Understanding of inference economics: tokens, context, caching, batching, and provider pricing behaviour.


• Skepticism about benchmarks and the discipline to construct better ones.

Experience

• 7+ years in software or ML engineering, including 2+ years shipping LLM-backed systems in production.


• Has owned a routing, serving, or evaluation platform used by other teams.


• Experience with self-hosted or private-tenant model deployment is a strong plus.

Education

• BS, MS, or PhD in Computer Science, Machine Learning, Statistics, or a related field.


• Equivalent applied experience shipping ML systems is accepted in place of a degree.

Compensation and benefits

• Meaningful equity with a standard four-year vest.


• Full medical, dental, and vision coverage.


• 401(k) with company contribution, hardware budget, and generous compute for evaluation work.


• Support for publishing evaluation methodology and results.