AI QA Engineer: Automated Testing & Model Evaluation

Kforce Inc; New York (NY); On-site; USD 70,000 - 110,000; Full time

Benefits offered by this job

Medical/dental/vision insurance

HSA

FSA

401(k)

Life insurance

Disability insurance

AD&D insurance

Paid time off

Job summary

Kforce Inc is seeking a QA Engineer in New York, NY to design, develop, and maintain automated test suites for AI platforms. You will integrate testing into CI/CD pipelines, build end-to-end testing frameworks, and collaborate with platform teams to improve coverage and release confidence.

The role requires 1-3 years in QA or related fields, experience with Python/Java/JavaScript, and familiarity with testing methodologies.

Qualifications

• 1-3 years of experience in Software Quality Assurance, Test Automation, Quality Engineering, Software Development, or related technical discipline.
• Experience building automated tests using Python, Java, JavaScript, or similar programming languages.
• Familiarity with CI/CD pipelines and modern software delivery practices.
• Understanding of software testing methodologies, including integration, regression, and performance testing.
• Strong analytical and troubleshooting skills.
• Excellent written and verbal communication skills.

Responsibilities

• Design, develop, and maintain automated test suites for AI platforms and services.
• Integrate testing into CI/CD pipelines to support rapid releases.
• Build automated regression, functional, integration, and end-to-end testing frameworks.
• Collaborate with platform engineering teams to improve testing coverage and release confidence.
• Monitor test execution results and identify quality risks.
• Develop evaluation harnesses to assess AI model and agent performance.
• Create and curate golden datasets representing expected outcomes and user interactions.
• Establish deterministic validation methods to verify AI responses against expectations.
• Design scoring methodologies, evaluation metrics, and performance rubrics aligned to business objectives.
• Build adversarial and edge-case testing suites to assess model robustness, safety, and resilience.
• Implement continuous evaluation extending beyond pre-production into live environments.
• Monitor AI outputs against quality thresholds and service-level expectations.
• Develop dashboards and reporting mechanisms for model quality and drift visibility.
• Identify performance degradation, anomalous behavior, and emerging risks.
• Support root-cause analysis and remediation for quality issues.

Skills

Test Automation

Python

Java

JavaScript

CI/CD

Testing methodologies

Analytical skills

Communication

Tools

PyTest

Playwright

Selenium

JUnit

Git/GitHub

Azure DevOps

GitHub Actions

Job description

Kforce Inc is seeking a QA Engineer in New York, NY to design, develop, and maintain automated test suites for AI platforms. You will integrate testing into CI/CD pipelines, build end-to-end testing frameworks, and collaborate with platform teams to improve coverage and release confidence.

The role requires 1-3 years in QA or related fields, experience with Python/Java/JavaScript, and familiarity with testing methodologies.