AI Evaluation & QA Analyst - Remote LATAM

Gramian Consulting Group; United States; Remote; USD 55,000 - 83,000; Part time

Job summary

Gramian Consulting Group is seeking an AI Evaluation & Quality Assurance Specialist to review task quality, instructions, and scoring rubrics for AI agents. You will inspect reference solutions, grading logic, execution traces, and generated deliverables to identify defects and unfair penalties.

Comfortable reading Python, SQL, and shell scripts; passable to highly accurate analysis of technical workflows and evaluation logic is required.

Qualifications

• Proven ability to read and interpret code, logs, and data outputs.
• Experience evaluating AI tasks, rubrics, and reference solutions.
• Strong analytical reasoning and ability to identify defects or biases.

Responsibilities

• Validate task quality by checking instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
• Review AI agent execution traces, tool calls, and deliverables to determine whether outcomes are justified.
• Audit grading logic to identify brittle checks, incorrect expected answers, unsupported rubric criteria, and unfair penalties for valid alternative solutions.
• Investigate discrepancies between model performance, grader results, and expected outcomes.
• Distinguish genuine model limitations from task defects, grader errors, and environment or tool failures.
• Independently assess automated QC findings rather than accepting them without verification.
• Document concise, evidence-backed findings and provide actionable, reproducible feedback.
• Flag uncertainty and verify that implemented revisions resolve previously identified issues.
• Comfortable reading and interpreting Python, SQL, shell scripts, structured data, and execution logs.
• Experience reviewing technical workflows, software behavior, data outputs, or evaluation logic.
• Strong analytical skills, including the ability to verify calculations and reconcile conflicting evidence.
• Demonstrated ability to assess the correctness and completeness of technical deliverables.
• Strong written English with experience providing clear, specific, and reproducible feedback.
• High attention to detail when identifying inconsistencies, missing information, and evaluation defects.

Skills

Python

SQL

Shell scripting

Data analysis

QA/testing

AI evaluation

English writing

Tools

Python

SQL

Shell

Job description

Gramian Consulting Group is seeking an AI Evaluation & Quality Assurance Specialist to review task quality, instructions, and scoring rubrics for AI agents. You will inspect reference solutions, grading logic, execution traces, and generated deliverables to identify defects and unfair penalties.

Comfortable reading Python, SQL, and shell scripts; passable to highly accurate analysis of technical workflows and evaluation logic is required.