Online STEM Jobs in Brazil — Remote AI/ML Evaluation Engineer

Rex.zone; Brasil; Remote; BRL 120,000 - 160,000; Full time

Job summary

A technology company in Brazil is seeking a professional to evaluate AI system outputs through structured human-in-the-loop feedback. The role includes running LLM evaluation tasks and supporting data collection for RLHF workflows. Qualified applicants should have mid-senior experience in engineering or ML and be proficient in Python or SQL. This is a remote position offering the chance to work with distributed teams on production-grade training pipelines.

Qualifications

• Mid-Senior experience in engineering, applied ML, QA evaluation, or data operations.
• Strong understanding of rubrics, preference data, and RLHF concepts.
• Ability to write clear annotation guidelines and drive calibration.

Responsibilities

• Design and run LLM evaluation tasks.
• Support RLHF-style preference data collection and analysis.
• Audit data labeling outputs for compliance.

Skills

Mid-Senior experience in engineering

Understanding of rubrics

Proficient in Python or SQL

Strong written communication

Tools

Spreadsheets

Job description

About The Role

Join Rex.zone to evaluate and improve AI system outputs using structured human-in-the-loop feedback. You will support RLHF workflows, prompt evaluation, and QA evaluation for large language models, ensuring training data quality and measurable model performance improvement across domains and languages.

What You Will Do

• Design and run LLM evaluation tasks (pairwise ranking, rubric-based grading, error categorization)
• Support RLHF-style preference data collection and analysis
• Audit data labeling outputs and QA evaluation results for annotation guidelines compliance
• Perform prompt evaluation and regression testing across model versions
• Track quality metrics and reviewer calibration using spreadsheets and/or scripting
• Document failure modes and coordinate with stakeholders to resolve systematic issues

Core Workflows

• LLM training pipelines and evaluation harnesses
• Gold sets, calibration sessions, blind audits, and inter-annotator agreement checks
• NLP evaluation (classification, NER, summarization) and content safety labeling
• Computer vision/multimodal annotation review when needed

Required Qualifications

• Mid-Senior experience in engineering, applied ML, QA evaluation, or data operations
• Strong understanding of rubrics, preference data, and RLHF concepts
• Ability to write clear annotation guidelines and drive calibration
• Proficiency with spreadsheets and/or scripting (Python or SQL preferred)
• Strong written communication for documenting edge cases and acceptance criteria

Why Rex.zone

Work remotely with distributed teams supporting production-grade LLM training pipelines and disciplined QA evaluation processes that deliver measurable outcomes.