Remote AI Quality & Evaluation Specialist

Gramian Consulting Group; United States; Remote; USD 28,000 - 55,000; Full time

Job summary

Gramian Consultancy is seeking an AI Quality & Evaluation Specialist to validate task quality, review AI agent performance, and audit grading logic. You will analyze execution traces, tool calls, reference solutions, and evaluation criteria to identify task defects, grading errors, and unjustified model failures.

This is a remote LATAM contractor role, 40 hours per week, requiring strong analytical judgment, technical QA experience, and fluency in written English to provide precise,

Qualifications

• Experience in technical QA, AI evaluation, data analysis, software testing, or a related technical field.
• Strong analytical skills and the ability to provide clear, evidence-based feedback.
• Excellent written English and the ability to communicate findings precisely.

Responsibilities

• Validate task instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
• Review AI agent execution traces, tool calls, and generated deliverables to assess justification of outcomes.
• Audit grading logic to identify brittle checks, incorrect expected answers, and unsupported rubric criteria.
• Identify cases where valid alternative solutions are unfairly penalized.
• Investigate discrepancies between model performance, task defects, grader errors, and environment or tool failures.
• Independently assess automated QC findings and avoid accepting them without verification.
• Document concise, evidence-backed findings and actionable recommendations.
• Clearly communicate uncertainty and distinguish confirmed issues from potential problems.
• Verify that task revisions resolve identified defects.
• Collaborate with relevant teams to improve evaluation quality and reliability.

Skills

Technical QA

AI evaluation

Data analysis

Software testing

English writing

Python

SQL

Shell scripting

Attention to detail

Decision-making

Tools

Python

SQL

Shell scripting

Execution logs

Structured data

Job description

Gramian Consultancy is seeking an AI Quality & Evaluation Specialist to validate task quality, review AI agent performance, and audit grading logic. You will analyze execution traces, tool calls, reference solutions, and evaluation criteria to identify task defects, grading errors, and unjustified model failures.

This is a remote LATAM contractor role, 40 hours per week, requiring strong analytical judgment, technical QA experience, and fluency in written English to provide precise,