QA Engineer

Kforce Inc; New York (NY); On-site; USD 70,000 - 110,000; Full time

Benefits offered by this job

Medical/dental/vision insurance

HSA

FSA

401(k)

Life insurance

Disability insurance

AD&D insurance

Paid time off

Job summary

Kforce Inc is seeking a QA Engineer in New York, NY to design, develop, and maintain automated test suites for AI platforms. You will integrate testing into CI/CD pipelines, build end-to-end testing frameworks, and collaborate with platform teams to improve coverage and release confidence.

The role requires 1-3 years in QA or related fields, experience with Python/Java/JavaScript, and familiarity with testing methodologies.

Qualifications

• 1-3 years of experience in Software Quality Assurance, Test Automation, Quality Engineering, Software Development, or related technical discipline.
• Experience building automated tests using Python, Java, JavaScript, or similar programming languages.
• Familiarity with CI/CD pipelines and modern software delivery practices.
• Understanding of software testing methodologies, including integration, regression, and performance testing.
• Strong analytical and troubleshooting skills.
• Excellent written and verbal communication skills.

Responsibilities

• Design, develop, and maintain automated test suites for AI platforms and services.
• Integrate testing into CI/CD pipelines to support rapid releases.
• Build automated regression, functional, integration, and end-to-end testing frameworks.
• Collaborate with platform engineering teams to improve testing coverage and release confidence.
• Monitor test execution results and identify quality risks.
• Develop evaluation harnesses to assess AI model and agent performance.
• Create and curate golden datasets representing expected outcomes and user interactions.
• Establish deterministic validation methods to verify AI responses against expectations.
• Design scoring methodologies, evaluation metrics, and performance rubrics aligned to business objectives.
• Build adversarial and edge-case testing suites to assess model robustness, safety, and resilience.
• Implement continuous evaluation extending beyond pre-production into live environments.
• Monitor AI outputs against quality thresholds and service-level expectations.
• Develop dashboards and reporting mechanisms for model quality and drift visibility.
• Identify performance degradation, anomalous behavior, and emerging risks.
• Support root-cause analysis and remediation for quality issues.

Skills

Test Automation

Python

Java

JavaScript

CI/CD

Testing methodologies

Analytical skills

Communication

Tools

PyTest

Playwright

Selenium

JUnit

Git/GitHub

Azure DevOps

GitHub Actions

Job description

Responsibilities

Kforce has a client that is seeking a QA Engineer in New York, NY.

• Design, develop, and maintain automated test suites for AI platforms and services
• Integrate testing capabilities into CI/CD pipelines to support rapid and reliable releases
• Build automated regression, functional, integration, and end-to-end testing frameworks
• Collaborate with platform engineering teams to improve testing coverage and release confidence
• Monitor test execution results and proactively identify quality risks
• Develop and maintain evaluation harnesses used to assess AI model and agent performance
• Create and curate golden datasets that represent expected business outcomes and user interactions
• Establish deterministic validation methods to verify AI responses against predefined expectations
• Design scoring methodologies, evaluation metrics, and performance rubrics aligned to business objectives
• Build adversarial and edge-case testing suites to assess model robustness, safety, and resilience
• Implement continuous evaluation capabilities that extend beyond pre-production testing into live environments
• Monitor AI outputs against established quality thresholds and service-level expectations
• Develop dashboards and reporting mechanisms that provide visibility into model quality and drift
• Identify performance degradation, anomalous behavior, and emerging risks
• Support root-cause analysis and remediation efforts when quality issues are identified
• Validate sandbox environments, platform guardrails, and operational control frameworks
• Verify that platform controls function as intended across development, testing, and production environments
• Support testing of AI safety mechanisms, access controls, content filters, and governance policies
• Ensure evaluation standards remain aligned with regulatory, compliance, and organizational requirements

Requirements

• 1-3 years of experience in Software Quality Assurance, Test Automation, Quality Engineering, Software Development, or a related technical discipline
• Experience building automated tests using Python, Java, JavaScript, or similar programming languages
• Familiarity with CI/CD pipelines and modern software delivery practices
• Understanding of software testing methodologies, including integration, regression, and performance testing
• Strong analytical and troubleshooting skills
• Excellent written and verbal communication skills

Preferred Qualifications

• Experience building evaluation frameworks or automated validation tools
• Experience with cloud platforms such as Azure, AWS, or Google Cloud
• Experience with testing frameworks such as PyTest, Playwright, Selenium, JUnit, or equivalent tools
• Exposure to Generative AI, Large Language Models (LLMs), AI agents, or machine learning systems
• Exposure to observability and monitoring platforms
• Familiarity with prompt engineering, retrieval-augmented generation (RAG), or agentic workflows
• Understanding of AI safety, governance, model evaluation, and risk management concepts

Technical Skills

• Testing & Automation
• Python
• PyTest
• API Testing
• Test Automation Frameworks
• CI/CD Pipelines
• Git/GitHub

AI & Data

• Generative AI
• LLM Evaluation
• Prompt Testing
• Golden Dataset Creation
• Evaluation Metrics
• Continuous Monitoring

Cloud & Platforms

• Azure
• Azure DevOps
• GitHub Actions
• Container Technologies
• Monitoring & Observability Tools

The pay range is the lowest to highest compensation we reasonably in good faith believe we would pay at posting for this role. We may ultimately pay more or less than this range. Employee pay is based on factors like relevant education, qualifications, certifications, experience, skills, seniority, location, performance, union contract and business needs. This range may be modified in the future.

We offer comprehensive benefits including medical/dental/vision insurance, HSA, FSA, 401(k), and life, disability & ADD insurance to eligible employees. Salaried personnel receive paid time off. Hourly employees are not eligible for paid time off unless required by law. Hourly employees on a Service Contract Act project are eligible for paid sick leave.

Note: Pay is not considered compensation until it is earned, vested and determinable. The amount and availability of any compensation remains in Kforce's sole discretion unless and until paid and may be modified in its discretion consistent with the law.

This job is not eligible for bonuses, incentives or commissions.

Kforce is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.