AI Evaluation Engineer - Proofline

Meraki-Labs

Bengaluru

On-site

INR 3,000,000 - 5,500,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Meraki Labs in Bengaluru is seeking an AI Evaluation Engineer to own evaluation scaffolding and quality layers for AI-native features across hosts and assistants.

You will design eval harnesses, test infrastructure, and release gates, working with founders to ensure reliability in production AI systems. 5+ years in SDET/QA or ML evaluation is required, with autonomy in a fast-moving environment.

Qualifications

  • Background in SDET/QA automation or ML evaluation.
  • Strong programming in Python or TypeScript with API-level testing.
  • Experience with Playwright or Cypress for testing.
  • Familiarity with evaluating non-deterministic AI systems.

Responsibilities

  • Build and maintain evaluation harnesses for AI-facing features to measure system quality.
  • Own end-to-end API test infrastructure supporting daily releases.
  • Design host-behavior probes via scripted user sessions across AI assistants.
  • Gate production releases with thorough UAT, regression analysis and quality reporting.

Skills

Python
TypeScript
API testing
Test automation
SDET/QA automation

Tools

Playwright
Cypress

Job description

About Us

Meraki Labs (founded by Mukesh Bansal & Peeyush Ranjan) builds and rapidly scales AI-first, "moonshot" startups. We're looking for a high-velocity, production-grade engineer to build 0-to-1 products alongside founders.

About the Role

Proofline is live in production with institutional pilots underway. Building with AI is easy to prototype, but proving reliability in production is a major challenge. In AI-native codebases, verification is the key bottleneck for scaling capabilities. As an AI Evaluation Engineer, you will take ownership of the evaluation scaffolding and quality layer, including eval harnesses for AI-facing features, test infrastructure, and release gates. You’ll play a critical role in monitoring how our product behaves across AI assistants (e.g., Claude, ChatGPT), accounting for differences by host and continual changes.

AI Evaluation Engineer — Evals & Systems Verification

Location: HSR, Bengaluru (On-site)

Experience: 5+ years

Responsibilities
  • Build and maintain evaluation harnesses for AI-facing features to measure and tune system quality (e.g., capture quality, retrieval quality, guidance quality)
  • Own end-to-end and API test infrastructure (Playwright-class), supporting a continuous, daily-release cycle
  • Design and execute host-behavior probes using scripted user sessions across diverse AI assistants, ensuring product behavior aligns with expectations
  • Gate production releases through thorough user acceptance testing (UAT), regression analysis, and quality reporting
Requirements
  • Background in SDET/QA automation or ML evaluation, with proven ownership of test or evaluation infrastructure and not just executing tests
  • Strong programming skills in Python or TypeScript, with experience in API-level testing and tools like Playwright or Cypress
  • Familiarity with LLM applications or a demonstrated interest in evaluating non-deterministic systems
  • Highly autonomous and able to define your own workflows and processes for evaluation and verification
How We Work

Small team, high trust, written decisions. Designs get adversarial review before code; PRs get automated review driven to zero open findings; features aren't done until verified on a live system. AI agents do a large share of the implementation—your leverage is judgment: framing the problem, freezing the right design, and knowing when the machine is wrong.

You Should Apply If
  • You enjoy turning messy problems into concrete solutions.
  • You thrive in fast-moving, low-process environments.
  • You're excited about production engineering (monitoring, reliability, cost, latency).
You Should Not Apply If
  • You want remote/hybrid (this is onsite Bangalore only).
  • You prefer narrow tickets and minimal ambiguity.
  • You prefer highly structured, slow-moving product organizations.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Evaluation Engineer - Proofline
AI Evaluation Engineer - Proofline

Fermi AI • Bengaluru

On-site
INR 1,500,000 - 2,100,000
QA Engineer - Proofline
QA Engineer - Proofline

Meraki Labs • Bengaluru

On-site
INR 600,000 - 1,100,000
QA Engineer - Proofline
QA Engineer - Proofline

Meraki-Labs • Bengaluru

On-site
INR 600,000 - 900,000
QA Engineer - Proofline
QA Engineer - Proofline

Meraki • Bengaluru

On-site
INR 600,000 - 1,200,000
Founding AI Engineer - Proofline
Founding AI Engineer - Proofline

Meraki-Labs • Bengaluru

On-site
INR 4,000,000 - 7,000,000
Aubergine Solutions - Senior AI Evaluation & Reliability Engineer
Aubergine Solutions - Senior AI Evaluation & Reliability Engineer

Aubergine Solutions • Ahmedabad

On-site
INR 700,000 - 1,200,000
Senior AI Evaluation & Reliability Engineer
Senior AI Evaluation & Reliability Engineer

Aubergine Solutions Pvt. Ltd. • Ahmedabad District

On-site
INR 3,000,000 - 6,000,000
Great Place To Work certified
[SLTKGEUJ] AI Tester
[SLTKGEUJ] AI Tester

ZettaMine Labs Pvt. Ltd. • Bengaluru

On-site
INR 1,400,000 - 2,100,000
AI Evaluation Engineer
AI Evaluation Engineer

Vinyl Equity, Inc. • Mangaluru

On-site
INR 1,200,000 - 1,800,000
Competitive compensation
Equity participation
Health insurance
+1
Applied AI Engineer
Applied AI Engineer

PingAura AI • Mumbai

On-site
INR 3,000,000 - 6,000,000