AI Test & Quality Engineer (Hybrid)

System One; Vienna (VA); Hybrid; USD 83.000 - 131.000; Vollzeit

Benefits dieser Stelle

Health benefits

401(k) plan

Zusammenfassung

System One in Vienna, VA is seeking an AI Test Engineer to design and run automated tests for generative AI and AI-powered DevEx tools. You will build evaluation pipelines, validate model behavior, and ensure quality across CI/CD workflows.

The role emphasizes governance, reproducibility, and collaboration with cross-functional teams. The ideal candidate has strong Python skills, experience with test automation and AI evaluation techniques, and the ability to operate in a fast-paced, hybrid

Qualifikationen

• Bachelor's degree or equivalent in a technical field.
• 3+ years experience in software testing or AI/ML testing.
• Strong Python and scripting skills.
• Experience with automated test frameworks and CI/CD pipelines.
• Familiarity with AI models and evaluation techniques.

Aufgaben

• Help design, develop, and maintain AI testing and evaluation frameworks.
• Create and execute automated test suites for AI models and DevEx tools.
• Build automated evaluation pipelines integrated with CI/CD workflows.
• Perform functional, regression, and performance testing of AI-enabled apps.
• Analyze AI outputs to identify defects, bias, and reliability concerns.
• Maintain test datasets, benchmarks, and evaluation artifacts for AI systems.
• Collaborate with engineers to improve model quality and coverage.
• Define and report AI quality metrics like accuracy and latency.
• Support root-cause analysis and validation of AI defects.
• Document test strategies and recommendations for stakeholders.
• Contribute to governance and responsible AI efforts in testing.
• Support delivery of DevEx initiatives across the software lifecycle.

Kenntnisse

Python

CI/CD

Test Automation

Problem solving

Ausbildung

Bachelor's degree in Computer Science or related field

Tools

DeepEval

LangSmith

Azure AI

GitHub Copilot

Jobbeschreibung

System One in Vienna, VA is seeking an AI Test Engineer to design and run automated tests for generative AI and AI-powered DevEx tools. You will build evaluation pipelines, validate model behavior, and ensure quality across CI/CD workflows.

The role emphasizes governance, reproducibility, and collaboration with cross-functional teams. The ideal candidate has strong Python skills, experience with test automation and AI evaluation techniques, and the ability to operate in a fast-paced, hybrid