AI Evaluation Analyst

Cashback24x7; India; Remote; INR 3,234,000 - 6,468,000; Full time

Job summary

Cashback24x7 is seeking an AI Evaluation Analyst to contribute to a customer project focused on advancing frontier language model capabilities. The role involves creating high-quality evaluation data and training content, with emphasis on written and verbal clarity, multi-turn conversation design, and model behavior analysis.

No prior AI experience required beyond domain knowledge. The contractor will author conversations, test drafts with large language models, and deliver transcripts, rubrics,

Qualifications

• Native-level written English with clarity and structure.
• Experience in data annotation, RLHF, SFT, evaluation, or prompt engineering for AI systems.

Responsibilities

• Author detailed, task-based multi-turn conversations and rubrics aligned with project specs.
• Test and refine conversation drafts against frontier large language models and iterate for quality and difficulty.
• Deliver evaluation assets including transcripts, target behaviors, rubrics, and supporting evidence.
• Validate and calibrate outputs with team leads and adapt to evolving guidelines.

Skills

Native-level English

Critical thinking

Analytical writing

Prompt engineering

Data annotation

Job description

Job Description

Role Title: AI Evaluation Analyst
Role Type: Contractor
Location: Remote

micro1 is engaging AI Evaluation Analysts to contribute to a customer's project focused on advancing frontier language model capabilities. In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. No prior experience in AI is required — your domain knowledge is what matters. You will play a crucial role in producing evaluation and training data that directly influence how powerful AI models understand and interact. This is an exciting opportunity to operate at the forefront of AI, with a focus on written and verbal clarity, multi-turn conversation design, and deep analysis of model behavior.

Scope of Work

• Author detailed, task-based multi-turn conversations and rubrics aligned with project specifications.
• Test and refine conversation drafts against frontier large language models, iterating to meet quality and difficulty requirements.
• Deliver comprehensive evaluation assets including transcripts, target behaviors, binary rubrics, and supporting evidence.
• Ensure strict fidelity to evolving project specs while maintaining high throughput and attention to detail.
• Validate and calibrate outputs with team leads and quality control as guidelines change.
• Work independently and consistently, meeting expected output rates for deliverable completion.

Preferred Qualifications

• Native-level written English with exceptional clarity, structure, and attention to detail.
• Prior experience in data annotation, RLHF, SFT, evaluation, or prompt engineering for AI systems.
• Working knowledge of frontier LLM behaviors and common model failure patterns.
• Demonstrated ability to interpret and apply highly detailed specifications without supervision.
• Strong critical thinking and analytical skills in writing-heavy or analysis-heavy domains.
• Experience authoring evaluation items, rubrics, or conducting deep analysis of technology outputs.
• Background in research, editorial, technical writing, or quality assurance is a plus.

Compensation Structure

Compensation is output-based; experts are paid per task that meets the project specifications. The time required to complete work may vary depending on the expert’s experience and workflow. Minimum submission requirements apply. Experts must submit a minimum of tasks per week.

Start Timeline & Availability

We typically fill roles within 48 hours and are looking for experts ready to jump in right away. If selected, we expect you to start your first tasks within 24–48 hours of completing onboarding.