Remote Senior QA & AI Evaluation Scientist

SDH Systems

United States

Remote

USD 120,000 - 155,000

Full time

14 days+

Application generator

Don’t send a generic resume — generate a resume and cover letter tailored to this exact role.

Get past ATS filters

Job summary

SDH Systems is seeking a Senior QA Automation Engineer / Data Scientist (Generative AI) for 100% remote work. The role focuses on evaluating AI-generated outputs, defining quality metrics, and testing non-deterministic systems.

Candidates from QA automation, data science, or ML backgrounds are welcome. The ideal candidate will research innovative evaluation methods, build automated testing pipelines, and collaborate across product, engineering, and data science teams to improve AI quality over

Qualifications

Background as a Data Scientist or QA Automation Engineer with related degree or experience.
Expertise and/or hands-on experience with LLM evaluation.
Proficient in Python or similar language used for testing/data analysis.
Strong analytical, research, and problem-solving abilities.

Responsibilities

Perform LLM evaluation and quality assurance for Generative AI applications.
Design and execute testing strategies for AI/LLM-powered systems.
Define evaluation criteria, quality standards, and success metrics for AI outputs.
Research innovative approaches for evaluating non-deterministic AI systems.
Create test datasets, benchmarks, and repeatable evaluation methodologies.
Identify quality issues such as hallucinations, inconsistencies, and inaccuracies.
Develop processes to measure AI quality over time.
Build and maintain automated testing/evaluation solutions.
Develop dashboards and reports communicating AI quality trends.
Collaborate with product owners, engineers, and data scientists.