Mid-Level AI Software/QA AutomationTest Engineer

SPECTRAFORCE; Ann Arbor (MI); Hybrid; USD 51,000 - 55,000; Full time

Job summary

Spectraforce in Ann Arbor, MI, is seeking a Mid-Level AI Software Test Engineer to design, automate, and execute testing strategies for AI-powered applications, ML systems, and AI agents. The role focuses on reliability, safety, and production readiness.

The ideal candidate will have a solid software testing background, automated test frameworks, and interest in AI tech such as LLMs, RAG, and integrations with enterprise services; 100% onsite for six months, then hybrid.

Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or related field.
• 3–6 years of software testing, QA automation, or quality engineering experience.
• Experience with API testing and automation tools.
• Strong understanding of test automation, SDLC, Agile, and CI/CD pipelines.
• Experience testing distributed systems, web applications, and APIs.

Responsibilities

• Develop comprehensive testing strategies for AI applications, platforms, and services.
• Validate AI model outputs for accuracy, consistency, reliability, and safety.
• Design and execute functional, integration, end-to-end, regression, and performance tests for AI solutions.
• Create test cases for prompt-driven, agentic, and retrieval-based AI workflows.
• Validate integrations with external systems, data sources, and enterprise services.

Skills

AI Testing

Test Automation

API Testing

CI/CD

Distributed Systems

Agile

Quality Engineering

Education

Bachelor’s degree in CS/SE/IS

Tools

Selenium

Playwright

Postman

JUnit

PyTest

GitHub Actions

Jenkins

Docker

Kubernetes

Job description

Position: Mid-Level AI Software Test Engineer

Duration: 18 Months

Location: Ann Arbor, MI. 100% onsite for the first 6 months. After 6 months, the schedule will transition to 4 days onsite and 1 day remote.

We are seeking a Mid-Level AI Software Test Engineer to design, automate, and execute testing strategies for AI-powered applications, machine learning systems, AI agents, copilots, and generative AI solutions.

This role combines traditional software quality engineering with emerging AI validation techniques to ensure AI systems are reliable, safe, accurate, performant, and production-ready.

The ideal candidate has a strong software testing background, experience building automated test frameworks, and an interest in AI technologies such as LLMs, agents, RAG systems, MCP integrations, and machine learning models.

Responsibilities

• Develop comprehensive testing strategies for AI applications, platforms, and services.
• Validate AI model outputs for accuracy, consistency, reliability, and safety.
• Design and execute functional, integration, end-to-end, regression, and performance tests for AI solutions.
• Create test cases for prompt-driven, agentic, and retrieval-based AI workflows.
• Validate AI guardrails, business rules, permissions, and governance controls.
• Perform adversarial, negative, and edge-case testing to identify model failures and hallucinations.
• Build and maintain automated test frameworks for AI applications.
• Develop automated evaluation pipelines for AI responses and workflows.
• Implement automated quality scoring and regression detection.
• Create reusable test data, mocks, simulators, and validation frameworks.
• Test AI agents, workflows, APIs, MCP integrations, and tool-calling capabilities.
• Validate integrations with external systems, data sources, and enterprise services.
• Verify performance, reliability, scalability, and resiliency of AI workloads.
• Execute load and stress testing for AI services.
• Partner with software engineers, AI engineers, product owners, architects, and security teams.
• Participate in design reviews and provide quality feedback during development.
• Contribute to test strategy, quality standards, and best practices.
• Support production readiness reviews and defect triage activities.

Required Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or related field.
• 3–6 years of software testing, QA automation, or quality engineering experience.
• Experience with API testing and automation tools.
• Strong understanding of test automation, software development lifecycle (SDLC), Agile methodologies, and CI/CD pipelines.
• Experience testing distributed systems, web applications, and APIs.

Preferred Qualifications

• Experience testing generative AI applications, LLM-based systems, AI agents, RAG applications, or MCP-based integrations.
• Familiarity with OpenAI, Claude, Gemini, Azure OpenAI, or similar AI platforms.
• Experience building evaluation and benchmarking frameworks for AI solutions.
• Experience testing cloud-native applications on Azure, AWS, or GCP.
• Knowledge of responsible AI, AI governance, and AI risk management practices.
• Understanding of enterprise governance, risk, privacy, and compliance requirements.
• Experience applying performance and load testing to AI services.

Technical Skills

• AI Testing: Prompt Testing, Response Evaluation, Hallucination Detection, Agent Workflow Validation, RAG Validation, AI Safety Testing, AI Regression Testing, AI Benchmarking
• Quality Engineering: Test Automation, API Testing, Integration Testing, Performance Testing, Load Testing, Security Testing, Defect Analysis, Root Cause Investigation
• Tools and Technologies: Selenium, Playwright, Postman, JUnit, PyTest, GitHub Actions, Jenkins, Docker, Kubernetes

Success Measures

• Detect AI quality issues before production deployment.
• Reduce regression defects across AI releases.
• Improve confidence in AI model and agent behavior.
• Enable safe, scalable delivery of AI-powered features.
• Ensure AI solutions meet quality, security, and governance requirements.

Experience

3–6 years of software quality engineering experience, with 1–3 years of exposure to AI/ML or Generative AI technologies.

• For information on benefits, equal opportunity employment, and location-specific applicant notices, click here

At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws.This position's pay range is $37.00/hr - $40.00/hr.