AI SDET

CIS Technologies INC; New York (NY); On-site; USD 120,000 - 170,000; Full time

Job summary

CIS Technologies INC. is seeking an AI SDET / Evaluation Engineer to build, test, and evaluate AI-powered applications and systems in a fast-paced software quality environment.

You will leverage AI coding agents to accelerate traditional SDET activities and ensure reliable, accurate, and production-ready AI solutions. The role emphasizes testing AI workflows, datasets, and model comparisons, with responsibilities spanning dataset development, AI output evaluation, and maintenance of regression

Qualifications

• 15+ years of software development experience.
• Foundational understanding of LLMs, Generative AI, and AI failure modes.
• Hands-on ability to work effectively with AI coding agents.
• Knowledge of APIs, Git, automated testing, and SDLC practices.
• Familiarity with CI/CD tools such as Jenkins and Jira.
• Understanding of statistics, sampling, precision, and recall.
• Strong analytical, troubleshooting, and documentation skills.

Responsibilities

• Build and maintain automated evaluation tests and scripts.
• Develop and manage evaluation datasets and test scenarios.
• Evaluate AI outputs for accuracy, completeness, citations, and instruction following.
• Test AI workflows, APIs, document ingestion, integrations, and structured outputs.
• Analyze failures, investigate root causes, and document defects.
• Compare performance across different models, prompts, and application versions.
• Help calibrate AI graders against human evaluations.
• Maintain regression suites and support CI/CD integration.

Skills

Python

Java

C#

Tools

APIs

Git

Automated testing

CI/CD (Jenkins)

Jira

Job description

We are looking for an AI SDET / Evaluation Engineer to help build, test, and evaluate our AI-powered applications and systems.

This is an exciting opportunity for software testing professionals who are interested in working at the intersection of Software Quality Engineering, Generative AI, and AI-powered automation.

What makes this role different?

You will leverage AI coding agents to perform and accelerate traditional SDET activities. You will also set up and use AI coding agents to test our company's AI systems, ensuring the applications are reliable, accurate, secure, and production ready.

Key Responsibilities:

• Build and maintain automated evaluation tests and scripts.
• Develop and manage evaluation datasets and test scenarios.
• Evaluate AI outputs for accuracy, completeness, citations, and instruction following.
• Test AI workflows, APIs, document ingestion, integrations, and structured outputs.
• Analyze failures, investigate root causes, and document defects.
• Compare performance across different models, prompts, and application versions.
• Help calibrate AI graders against human evaluations.
• Maintain regression suites and support CI/CD integration.

Required Skills & Qualifications:

• 15+ years of software development experience using Python, Java, or C#.
• Foundational understanding of LLMs, Generative AI, and common AI failure modes.
• Hands-on ability to work effectively with AI coding agents.
• Knowledge of APIs, Git, automated testing, and SDLC practices.
• Familiarity with CI/CD tools such as Jenkins and Jira.
• Understanding of statistics, sampling, precision, and recall.
• Strong analytical, troubleshooting, and documentation skills.

Preferred:

• Experience with AI application testing, LLM evaluations, or AI quality assurance. Academic, internship, and project experience in AI evaluation is also welcome.