AI Quality Assurance & Evaluation Specialist

Gramian Consulting Group; United States; Remote; USD 28,000 - 55,000; Full time

Job summary

Gramian Consultancy is seeking an AI Quality & Evaluation Specialist to validate task quality, review AI agent performance, and audit grading logic. You will analyze execution traces, tool calls, reference solutions, and evaluation criteria to identify task defects, grading errors, and unjustified model failures.

This is a remote LATAM contractor role, 40 hours per week, requiring strong analytical judgment, technical QA experience, and fluency in written English to provide precise,

Qualifications

• Experience in technical QA, AI evaluation, data analysis, software testing, or a related technical field.
• Strong analytical skills and the ability to provide clear, evidence-based feedback.
• Excellent written English and the ability to communicate findings precisely.

Responsibilities

• Validate task instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
• Review AI agent execution traces, tool calls, and generated deliverables to assess justification of outcomes.
• Audit grading logic to identify brittle checks, incorrect expected answers, and unsupported rubric criteria.
• Identify cases where valid alternative solutions are unfairly penalized.
• Investigate discrepancies between model performance, task defects, grader errors, and environment or tool failures.
• Independently assess automated QC findings and avoid accepting them without verification.
• Document concise, evidence-backed findings and actionable recommendations.
• Clearly communicate uncertainty and distinguish confirmed issues from potential problems.
• Verify that task revisions resolve identified defects.
• Collaborate with relevant teams to improve evaluation quality and reliability.

Skills

Technical QA

AI evaluation

Data analysis

Software testing

English writing

Python

SQL

Shell scripting

Attention to detail

Decision-making

Tools

Python

SQL

Shell scripting

Execution logs

Structured data

Job description

About Gramian

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.

About the Role

We are seeking an AI Quality & Evaluation Specialist to validate task quality, review AI agent performance, and audit grading logic. You will analyze execution traces, tool calls, reference solutions, and evaluation criteria to identify task defects, grading errors, and unjustified model failures. The ideal candidate combines technical fluency with strong analytical judgment and the ability to provide clear, evidence-based feedback.

CONTRACT: Contractor (Hourly)

COMMITMENT: 40 hours per week

LOCATION: Remote — Latin America (LATAM)

Responsibilities

• Validate task instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
• Review AI agent execution traces, tool calls, and generated deliverables to assess whether outcomes are justified.
• Audit grading logic to identify brittle checks, incorrect expected answers, and unsupported rubric criteria.
• Identify cases where valid alternative solutions are unfairly penalized.
• Investigate discrepancies between model performance, task defects, grader errors, and environment or tool failures.
• Independently assess automated QC findings rather than accepting them without verification.
• Document concise, evidence-backed findings and actionable recommendations.
• Clearly communicate uncertainty and distinguish confirmed issues from potential problems.
• Verify that task revisions resolve previously identified defects.
• Collaborate with relevant teams to improve evaluation quality and reliability.
• Experience in technical QA, AI evaluation, data analysis, software testing, or a related technical field.
• Comfortable reading and interpreting Python, SQL, shell scripts, structured data, and execution logs.
• Strong analytical skills, including the ability to validate calculations and reconcile conflicting evidence.
• Experience assessing the correctness and completeness of technical or professional deliverables.
• Strong written English and the ability to provide specific, reproducible feedback.
• High attention to detail when reviewing task requirements, grading behavior, and model outputs.
• Ability to independently investigate issues and make evidence-based decisions.