Founding AI Engineer - AI Evaluations and Systems

Applied Retention; Mumbai; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Applied Retention is building a retention product for online merchants. You will design evaluation suites to judge changes to prompts, models, or code, and push validated changes into production safely.

The role covers creating blueprints for measurable AI quality, CI-based regression tests, and LLM-as-a-judge systems. You will optimize for reliability, cost, and maintainability in a fast-moving team.

Qualifications

• 2+ years as a software engineer with hands-on work on an LLM feature that people have used.
• At least one evaluation suite built and run with clear impact.
• Hands-on experience with LLM-as-a-judge including rubrics and calibration.
• Experience with at least one evaluation tool from the list provided (e.g., promptfoo, DeepEval).
• Familiarity with evaluation measures: faithfulness, relevance, task completion, precision/recall, confidence intervals.

Responsibilities

• Make AI quality measurable by building golden sets, behavioural assertions, thresholds, and regression cases.
• Build evaluation pipelines that run in CI (e.g., GitHub Actions), pin model versions, and ensure regressions block merges.
• Engineer LLM-as-a-Judge systems with structured rubrics, calibration, and reporting statistics like precision and Cohens kappa.
• Keep evaluation affordable by deterministic checks and batching; monitor cost and latency.
• Ship and run it reliably with Dockerized services, reversible deployments, and runbooks.

Skills

Analytical thinking

Problem solving

Team collaboration

Communication

Education

Bachelor's degree in CS

Tools

promptfoo

DeepEval

Ragas

TruLens

LangSmith

Langfuse

Braintrust

Inspect

Python

PostgreSQL

LLM APIs (Anthropic/OpenAI)

CI/CD (GitHub Actions)

Docker

Cloud (AWS/GCP/Azure)

Git PR workflows

Job description

We are building a retention product for online merchants, with LLM-drafted content that is validated and human-approved before anything reaches a customer. The question this role answers: How do we know a change to a prompt, a model, or the code made the system better, not worse? You will build the evaluation suites that decide whether a change may ship, and help get that change to production safely. This role suits an engineer early in their career who has already built real evaluations and shipped something, and who wants to grow into owning both quality and reliability.

The candidate will have responsibilities across the following functions:

Make AI Quality Measurable:

• Build golden sets and behavioural assertions for LLM outputs.
• Read real outputs and traces first, and group failures into a taxonomy before choosing metrics.
• Set pass thresholds and track hallucinated or unsupported claims.
• Turn every production failure into a regression case.

Build Evaluation Pipelines That Run in CI:

• Build regression suites for prompts, models, and tools.
• Run them in GitHub Actions or an equivalent platform so a regression blocks the merge.
• Pin model versions and rerun suites on provider upgrades.
• Repeat runs and compare results properly, so noise is not mistaken for progress.
• Review changes against what their descriptions and tests claim, and add checks that would have caught any gaps.
• Make evaluation part of engineering, not a final QA step.

Engineer LLM-as-a-Judge Systems:

• Write structured rubrics and scoring mechanisms.
• Calibrate the judge against human-labelled examples and report agreement using precision, recall, and Cohen's kappa.
• Monitor judge bias, including position, verbosity, and self-preference.

Keep Evaluation Affordable:

• Use deterministic checks for simple cases and reserve LLM judges for more complex cases.
• Batch and parallelise evaluation runs.
• Track cost and latency per run.

Ship and Run It Reliably:

• Containerise services with Docker and automate builds and deployments.
• Keep deployments reversible through rollback steps, health checks, and backups that have been restored at least once.
• Add tracing and cost tracking for model calls, along with actionable alerts.
• Write runbooks and concise technical decision notes.

Requirements:

• 2+ years as a software engineer, with recent hands-on work on an LLM feature that people have used. RAG or agent products count.
• At least one evaluation suite you built and ran, with a clear explanation of what it caught and what you changed because of it.
• Practical experience with LLM-as-a-judge, including rubrics, human labels, and calibration.
• Hands-on experience with at least one of: promptfoo, DeepEval, Ragas, TruLens, LangSmith, Langfuse, Braintrust, Inspect, or an equivalent tool.
• Familiarity with common evaluation measures: faithfulness, answer relevance, task completion, precision and recall, and confidence intervals.

Technical Skills:

• Python 3. x: pytest, Pydantic, httpx or requests, and asyncio basics.
• PostgreSQL / SQL: joins, aggregates, and window functions.
• LLM APIs: Anthropic and/or OpenAI SDKs, structured outputs, and tool calling.
• CI/CD: GitHub Actions or an equivalent platform, including a check that blocks a merge.
• Docker: Building and running containerised services.
• Deployment automation: Basic deploy scripts or pipeline jobs, environment configuration, and secrets handling.
• Cloud: Working-level knowledge of any one of AWS, GCP, or Azure. AWS is a plus. If you have not used a cloud yet, Docker plus solid deployment automation is enough to start.
• Git: Pull-request workflows and code reviews.