AI Quality Analyst: Model Evaluation & Safety

RXinsider LTD.; Lincolnshire (IL); Hybrid; USD 123,000 - 184,000; Full time

Benefits offered by this job

Hybrid work

Healthcare

Well-being day

Flexible hours

Job summary

Zebra Technologies is seeking an AI Quality Analyst to ensure performance, safety, and reliability of cutting-edge AI/ML models. Design evaluation strategies, identify edge cases, bias sources, and provide actionable insights to drive model improvements across the development lifecycle.

The role requires hands-on data annotation, test automation, and cross-team collaboration, with responsibilities spanning benchmarking, error analysis, data curation, red teaming, and reporting results to

Qualifications

• Proven experience in a quality assurance, testing, or data analysis role, preferably within the AI/ML domain.
• A deep understanding of the machine learning lifecycle and the common failure modes of AI models.
• Hands‑on experience with data annotation, data validation, and managing large datasets.
• Meticulous attention to detail and a methodical approach to problem‑solving.
• Strong analytical skills with the ability to identify patterns in data and draw meaningful conclusions.
• Expertise with industry‑standard test automation tools and libraries (e.g., Selenium, Playwright, Cypress, REST‑assured).
• Experience in testing across different platforms (e.g. mobile Android/iOS and web).
• Experience with bug tracking systems (e.g., Jira) and test case management tools.
• Scripting skills (e.g., Python) for test automation and data manipulation.
• (Preferred) experience in testing AI systems, including evaluating agentic responses, model performance metrics, and data integrity.
• Excellent communication skills, with the ability to clearly document bugs and articulate complex technical issues.
• Familiarity with computer vision or other specific AI domains relevant to our work.

Responsibilities

• Evaluation Strategy & Benchmark Development: Design, develop, and maintain a comprehensive suite of test cases and evaluation benchmarks. Proactively identify potential model failure points, including edge cases, adversarial inputs, and sources of bias.
• Error Analysis & Failure Triage: Conduct systematic error analysis to categorize model failures and identify underlying patterns. Triage defects, prioritize them based on severity and impact, and work with the development team to ensure resolution.
• Data Sourcing & Curation: Source, curate, and manage high‑quality datasets for model evaluation and testing. This includes performing data annotation and validation to ensure the integrity of our ground‑truth data.
• Exploratory & Adversarial Testing (Red Teaming): Perform unscripted, exploratory testing to discover unexpected model behaviors. Participate in red teaming exercises to intentionally challenge our models and identify potential safety and security vulnerabilities.
• Test Environment Management: Set up, maintain, and troubleshoot testing and demonstration environments to ensure a stable and reliable evaluation pipeline.
• Reporting & Insights: Analyze and synthesize test results into clear, actionable reports for both technical and non‑technical stakeholders. Translate complex findings into concrete recommendations for model improvement.
• Process Improvement: Actively participate in post‑hoc evaluation reviews and contribute to the continuous improvement of our testing methodologies, tools, and overall quality assurance processes.

Job description

Zebra Technologies is seeking an AI Quality Analyst to ensure performance, safety, and reliability of cutting-edge AI/ML models. Design evaluation strategies, identify edge cases, bias sources, and provide actionable insights to drive model improvements across the development lifecycle.

The role requires hands-on data annotation, test automation, and cross-team collaboration, with responsibilities spanning benchmarking, error analysis, data curation, red teaming, and reporting results to