AI SDET & Evaluation Engineer — Automate & Elevate AI

CIS Technologies INC; New York (NY); On-site; USD 120,000 - 170,000; Full time

Job summary

CIS Technologies INC. is seeking an AI SDET / Evaluation Engineer to build, test, and evaluate AI-powered applications and systems in a fast-paced software quality environment.

You will leverage AI coding agents to accelerate traditional SDET activities and ensure reliable, accurate, and production-ready AI solutions. The role emphasizes testing AI workflows, datasets, and model comparisons, with responsibilities spanning dataset development, AI output evaluation, and maintenance of regression

Qualifications

• 15+ years of software development experience.
• Foundational understanding of LLMs, Generative AI, and AI failure modes.
• Hands-on ability to work effectively with AI coding agents.
• Knowledge of APIs, Git, automated testing, and SDLC practices.
• Familiarity with CI/CD tools such as Jenkins and Jira.
• Understanding of statistics, sampling, precision, and recall.
• Strong analytical, troubleshooting, and documentation skills.

Responsibilities

• Build and maintain automated evaluation tests and scripts.
• Develop and manage evaluation datasets and test scenarios.
• Evaluate AI outputs for accuracy, completeness, citations, and instruction following.
• Test AI workflows, APIs, document ingestion, integrations, and structured outputs.
• Analyze failures, investigate root causes, and document defects.
• Compare performance across different models, prompts, and application versions.
• Help calibrate AI graders against human evaluations.
• Maintain regression suites and support CI/CD integration.

Skills

Python

Java

C#

Tools

APIs

Git

Automated testing

CI/CD (Jenkins)

Jira

Job description

CIS Technologies INC. is seeking an AI SDET / Evaluation Engineer to build, test, and evaluate AI-powered applications and systems in a fast-paced software quality environment.

You will leverage AI coding agents to accelerate traditional SDET activities and ensure reliable, accurate, and production-ready AI solutions. The role emphasizes testing AI workflows, datasets, and model comparisons, with responsibilities spanning dataset development, AI output evaluation, and maintenance of regression