AI Eval Engineer

Hiringeye Solutions; Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Hiringeye Solutions is seeking an AI Eval Engineer to build evaluation systems for LLM-powered education products. You will develop Python scripts, create evaluation criteria, and improve AI outputs' quality and safety.

You'll work on LLM evaluation pipelines, prompts, RAG, tool calling, and multi-step reasoning, collaborating with Engineering, Product, QA, and Curriculum teams to enhance our AI products. Strong debugging and data analysis skills are essential.

Qualifications

• 3+ years of experience in Software Development, AI/ML, AI Evaluation, QA Automation, or Data Analysis.
• Strong Python programming and scripting skills.
• Experience with LLM APIs, JSON, datasets, notebooks, and API-based workflows.

Responsibilities

• Build and maintain LLM evaluation pipelines and testing tools.
• Develop Python scripts for batch testing, model comparison, regression testing, and data analysis.
• Create test datasets, golden datasets, rubrics, and evaluation criteria.
• Evaluate AI workflows involving prompts, RAG, structured outputs, tool calling, and multi-step reasoning.
• Identify hallucinations, quality issues, regressions, and recurring AI failure patterns.
• Collaborate with Engineering, Product, QA, and Curriculum teams to improve AI products.

Skills

Python

Scripting

Analytical thinking

Debugging

QA Automation

Data Analysis

AI/ML

Tools

LLM APIs

JSON

Notebooks

Experiment tracking

Git

API-based workflows

Job description

We are looking for an AI Eval Engineer to build evaluation systems and tools for LLM-powered education products. The role involves Python development, AI workflow testing, model evaluation, data analysis, and improving the quality, accuracy, and safety of AI outputs.

Key Responsibilities

• Build and maintain LLM evaluation pipelines and testing tools.
• Develop Python scripts for batch testing, model comparison, regression testing, and data analysis.
• Create test datasets, golden datasets, rubrics, and evaluation criteria.
• Evaluate AI workflows involving prompts, RAG, structured outputs, tool calling, and multi-step reasoning.
• Identify hallucinations, quality issues, regressions, and recurring AI failure patterns.
• Collaborate with Engineering, Product, QA, and Curriculum teams to improve AI products.

Required Skills

• 3+ years of experience in Software Development, AI/ML, AI Evaluation, QA Automation, or Data Analysis.
• Strong Python and scripting skills.
• Experience with LLM APIs, JSON, datasets, notebooks, and API-based workflows.
• Understanding of AI evaluation, prompt testing, regression testing, and quality measurement.
• Strong analytical and debugging skills.

Preferred Skills

• Experience with LLM-as-a-Judge, RAG, tool calling, and multi-step AI workflows.
• Experience in EdTech, curriculum, tutoring, or educational products.
• Familiarity with dashboards, annotation tools, Git, and experiment tracking.