AI Evaluation Engineer

Vinyl Equity, Inc.; Mangaluru; Sur place; INR 1 200 000 - 1 800 000; Plein temps

Avantages offerts par ce poste

Competitive compensation

Equity participation

Health insurance

Paid leave

Résumé du poste

Vinyl Equity, Inc. is seeking an AI Evaluation Engineer to own the quality bar for AI-powered features and the product experience around them.

You will design evaluation frameworks, define rubrics, and build automated pipelines to catch regressions and ensure trustworthy releases for regulated data. You will also extend testing beyond the model into end-to-end QA, collaborating with product managers and engineers to translate vague quality bars into measurable criteria.

Qualifications

• 5+ years in QA/test engineering or ML evaluation.
• Experience testing AI/LLM features and building eval sets.
• Strong scripting and API testing skills (Python/TypeScript/Java).

Responsabilités

• Design, build, and maintain evaluation frameworks and benchmark suites for AI/LLM features.
• Define golden datasets, rubrics, and scoring methodologies.
• Build automated eval pipelines in CI/CD with clear quality metrics.
• Extend evaluation to full product/QA testing of AI-powered features.
• Collaborate with PMs and engineers to translate quality bars into criteria.
• Investigate failures and perform root-cause analysis with engineering.
• Maintain traceability and reporting on eval/QA results for compliance.

Connaissances

QA/test engineering

ML evaluation

Scripting Python

TypeScript/Java

API testing

SQL data validation

CI/CD integration

Prompt engineering

RAG pipelines

Analytical mindset

Written communication

Outils

FAISS

ChromaDB

Ragas

DeepEval

promptfoo

Jira

Description du poste

About the Role

As an AI Evaluation Engineer, you'll own the quality bar for both our AI-powered features and the broader product experience they sit inside. You'll design and run evaluation frameworks that catch regressions in model behavior (accuracy, hallucination, safety, compliance-sensitive edge cases) as well as classic product/QA issues — so that what we ship to customers handling real securities and compliance data is trustworthy every time. This is a hands‑on, blended role: part evaluation engineering for LLM/AI systems, part product quality ownership.

What You'll Do

• Design, build, and maintain evaluation frameworks and benchmark suites for AI/LLM-powered features (e.g., document extraction, compliance checks, automated workflows), covering accuracy, consistency, hallucination rate, and safety.

• Define golden datasets, rubrics, and scoring methodologies (human‑in‑the‑loop and automated/LLM-as‑judge) to measure model and product quality objectively.

• Build automated eval pipelines that run in CI/CD, flag regressions before release, and produce clear, trackable quality metrics over time.

• Extend evaluation coverage beyond the model layer into full product/QA testing — functional, regression, and end‑to‑end testing of AI-powered features and the surrounding product.

• Partner with product managers and engineers to translate ambiguous quality bars ("is this good enough to ship?") into measurable, repeatable evaluation criteria.

• Investigate failures and edge cases, perform root‑cause analysis across the model/product boundary, and drive fixes with engineering.

• Maintain traceability and reporting on eval/QA results for compliance-sensitive workflows, given the regulated nature of the data we handle.

What We're Looking For

• 5 plus years of experience in QA/test engineering, ML evaluation, or a related quality-focused engineering role.

• Hands‑on experience testing or evaluating AI/LLM-powered features — building eval sets, scoring rubrics, or benchmark harnesses (or strong adjacent automation/QA experience with a demonstrated interest in AI evaluation).

• Solid automation/testing fundamentals: scripting (Python and/or TypeScript/Java), API testing, SQL for data validation, and CI/CD integration.

• Comfort working with LLMs and AI tooling directly — prompt engineering, RAG pipelines, or AI‑assisted development tools — either as a builder or a rigorous evaluator of them.

• Strong analytical mindset: comfortable defining metrics for fuzzy, subjective quality questions and defending them with data.

• Clear written communication — you'll be documenting failure modes and quality bars for both engineers and non‑technical stakeholders.

• Bonus: experience in fintech, compliance, or another regulated domain where correctness and auditability matter.

Nice to Have

• Experience with vector databases / semantic search (FAISS, ChromaDB, or similar) and RAG evaluation.

• Familiarity with eval tooling/frameworks (e.g., Ragas, DeepEval, promptfoo, custom LLM-as-judge pipelines).

• Background in Agile/Scrum environments and defect-management tooling (Jira or similar).

What We Offer

• Competitive compensation based on experience

• Equity participation

• Comprehensive health insurance (self + family)

• Paid leave and wellness benefits