SDET: AI Evaluation & Trust for Multi-Agent Systems

SOFICO; Dallas (TX); Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Health care

Remote-first

Flexible PTO

Conference & training

Job summary

SOFICO Intelligence is seeking an SDET to own quality for autonomous AI systems. You will design tests to prove AI reasons correctly, catch hallucinations, and ensure data isolation in a multi-tenant environment.

You’ll build the evaluation harness for agentic AI, verify grounding and faithfulness, and contribute to CI/CD pipelines enabling production-grade assurance at scale. Strong Go/Python skills and experience with modern testing tools are essential.

Qualifications

• Experience testing non-deterministic, multi-agent AI systems.
• Proven ability to evaluate AI outputs for faithfulness.
• Strong coding skills in Go and Python.
• Experience with automated testing tools and test harnesses.
• Familiarity with data privacy and multi-tenant isolation.

Responsibilities

• Design and build the evaluation harness for agentic AI: golden datasets, automated scoring, faithfulness checks, and regression detection across prompt, model, and agent changes.
• Verify grounding and faithfulness, ensuring every insight traces back to real data and prevent hallucinations from reaching production.
• Test multi-agent orchestration end-to-end: agentic tool-calling loops, supervisor-specialist pipelines, autonomous background jobs, and synthesize findings into trustworthy reports.
• Red-team the system for prompt injection, jailbreaks, and tool misuse; prove multi-tenant data isolation.
• Validate guard rails: iteration limits, agent-recursion depth, and token/cost ceilings at the edges.
• Pressure-test performance and scale: real-time streaming, high-concurrency conversations, latency, and cost under load.
• Build automated tests for Go microservices, data/SQL tools, and APIs; wire into CI/CD for continuous evaluation.
• Partner with AI and data engineers to evolve quality bars and acceptance criteria.

Skills

Go

Python

Quality Assurance

LLM testing

CI/CD

Distributed systems

Tools

Playwright

Postman

REST Assured

PyTest

JMeter

Docker

Kubernetes

Job description

SOFICO Intelligence is seeking an SDET to own quality for autonomous AI systems. You will design tests to prove AI reasons correctly, catch hallucinations, and ensure data isolation in a multi-tenant environment.

You’ll build the evaluation harness for agentic AI, verify grounding and faithfulness, and contribute to CI/CD pipelines enabling production-grade assurance at scale. Strong Go/Python skills and experience with modern testing tools are essential.