Remote AI Evaluation Architect - Tech Docs & Code

Weekday 1; United States; Remote; USD 21,000 - 28,000; Part time

Benefits offered by this job

Fully remote

Flexible hours

Weekly payments

Job summary

Weekday 1 is seeking a seasoned technology professional to design high-quality benchmark tasks for evaluating AI systems on software engineering and data science workflows. This fully remote, independent contractor role focuses on creating realistic, multi-step tasks grounded in technical documentation, codebases, APIs, and architectures.

You will develop evaluation standards, ground-truth solutions, and scoring rubrics, collaborating with research teams while maintaining rigorous accuracy and

Qualifications

• Minimum 3 years of hands-on professional experience in software engineering, data science, or data analytics.
• Strong understanding of technical documentation, software development workflows, and engineering best practices.
• Experience with codebases, APIs, technical specifications, or system architecture documentation.
• Excellent analytical thinking and problem-solving skills.
• Strong written communication with the ability to create clear technical instructions and evaluation criteria.
• Ability to work independently with high accuracy and consistency.

Responsibilities

• Design AI evaluation tasks based on professional technology workflows.
• Develop ground-truth solutions and scoring rubrics for tasks.
• Contribute domain expertise from software engineering, data science, or analytics.
• Collaborate with research teams to improve benchmark quality.
• Refine tasks based on feedback and evolving requirements.

Skills

Software Engineering

Data Science

Data Analytics

Job description

Weekday 1 is seeking a seasoned technology professional to design high-quality benchmark tasks for evaluating AI systems on software engineering and data science workflows. This fully remote, independent contractor role focuses on creating realistic, multi-step tasks grounded in technical documentation, codebases, APIs, and architectures.

You will develop evaluation standards, ground-truth solutions, and scoring rubrics, collaborating with research teams while maintaining rigorous accuracy and