AI Evaluation Architect - Data Science Expert (Remote)

Weekday AI (YC W21); United States; On-site; USD 165,312 - 234,192; Part time

Benefits offered by this job

Fully remote

Weekly payments

Job summary

Weekday AI is seeking Data Science Experts to join as independent contractors and help define robust evaluation frameworks for AI outputs. This fully remote role focuses on real-world data science tasks, not building models, but shaping quality criteria and feedback loops to improve AI reasoning.

Ideal candidates bring 5+ years in data science within industry, strong analytic background, and expertise in experimental design, statistics, and SQL/Python.

Qualifications

• 5+ years of professional experience in Data Science within an industry environment.
• Strong background in business operations, product analytics, growth analytics, or similar data science functions at technology-driven organizations.
• Deep expertise in: Experimental design and A/B testing; Statistical analysis; SQL and Python; Metric development and performance measurement; Translating analytical findings into business recommendations.
• Excellent written communication skills with the ability to clearly explain technical reasoning.
• Exceptional attention to detail and the ability to evaluate work consistently against structured rubrics.
• Comfortable receiving feedback and calibrating evaluation standards with peer reviewers.

Responsibilities

• Design detailed, task‑specific evaluation criteria for real‑world data science deliverables, including: Data analyses; Predictive and statistical models; Dashboards and visualizations; Experiment results; Executive recommendations and business insights.
• Evaluate AI-generated and human‑produced work using objective scoring frameworks.
• Provide detailed written justifications for each evaluation, ensuring consistency and technical accuracy.
• Apply sound analytical judgment to produce reproducible and defensible assessments.
• Incorporate structured reviewer feedback and continuously refine evaluation methodologies.
• Collaborate with cross‑functional teams to improve AI training quality and evaluation standards.

Skills

SQL

Python

Statistical analysis

Experimental design

A/B testing

Data science

Job description

Weekday AI is seeking Data Science Experts to join as independent contractors and help define robust evaluation frameworks for AI outputs. This fully remote role focuses on real-world data science tasks, not building models, but shaping quality criteria and feedback loops to improve AI reasoning.

Ideal candidates bring 5+ years in data science within industry, strong analytic background, and expertise in experimental design, statistics, and SQL/Python.