Agentic QA Engineer

Kaltechsoft Inc; Dallas (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Kaltechsoft Inc. is seeking a hands-on QA Engineer to design and execute end-to-end testing strategies for agentic AI solutions, including multi-agent systems in production-grade environments.

You will partner with the Operations Team to ensure resiliency, reliability, accuracy, latency, orchestration correctness, and scale. The role requires 7+ years in software QA/testing with AI/ML exposure, strong Python skills, and experience with LLM evaluation and orchestrations frameworks.

Qualifications

• 7+ years in Software QA/testing with AI/ML exposure.
• Hands-on testing of agentic/multi-agent architectures.
• Strong Python skills for test harnesses and fixtures.
• Experience with LLM evaluation and guardrails.
• Knowledge of distributed systems latency and resiliency.
• Familiarity with orchestration frameworks (LangChain, LangGraph, LlamaIndex).
• Proficiency in CI/CD, observability, and feature flags.

Responsibilities

• Define and own the QA strategy for agentic/multi-agent AI systems across dev, staging, and prod.
• Mentor a team of QA engineers; establish testing standards and guidelines for test harnesses.
• Partner with Agentic Operations, Data Science, MLOps, and Platform teams to embed QA in the SDLC and incident response.
• Design tests for agent orchestration, tool calling, planner-executor loops.
• Establish latency SLOs and measure end-to-end response times across orchestration layers.
• Ensure reliability through soak tests, canary verifications, and automated rollbacks.
• Instrument guardrail validations (toxicity, PII, hallucination, policy compliance).

Skills

Python programming

AI/ML testing

Software QA

Distributed systems testing

Chaos engineering

CI/CD

Observability

Tools

LangChain

LangGraph

LlamaIndex

DSPy

OpenAI Assistants/Actions

Azure OpenAI

GitHub Actions

Prometheus

Grafana

OpenTelemetry

Datadog

Job description

Agentic QA Engineer Generative AI & Agentic Systems (Agent, Multi Agent Testing)

Location: Dallas, TX - Onsite

Duration: Long term

Summary

We are seeking a hands-on AI Engineer to design and execute end-to-end testing strategies for agentic AI solutions, including multi-agent systems in production-grade environments. This role partners with the Agentic Operations Team to ensure resiliency, reliability, accuracy, latency, orchestration correctness, and scale.

Key Responsibilities

• Agentic & Multi Agent Testing
• Reliability, Resiliency, and Latency
• Accuracy & Macro-Level Validations
• Scale & Orchestration
• Dev Prod Readiness
• Define and own the QA strategy for agentic/multi-agent AI systems across dev, staging, and prod.
• Mentor a team of QA engineers; establish testing standards, coding guidelines for test harnesses, and review practices.
• Partner with Agentic Operations, Data Science, MLOps, and Platform teams to embed QA in the SDLC and incident response.
• Design tests for agent orchestration, tool calling, planner-executor loops, and
• Establish latency SLOs and measure end-to-end response times across orchestration layers (LLM calls, tool invocations, queues).
• Ensure reliability through soak tests, canary verifications, and automated rollbacks.
• Define ground-truth and reference pipelines for task accuracy (exact match, semantic similarity, factuality checks).
• Build macro validation frameworks that validate task outcomes across multi-step agent workflows (e.g., complex data pipelines, content generation + verification agent loops).
• Instrument guardrail validations (toxicity, PII, hallucination, policy compliance).

Required Qualifications

• 7+ years in Software QA/Testing, with 2+ years in AI/ML or LLM-based systems; hands-on experience testing agentic/multi-agent architectures.
• Strong programming skills in Python experience building test harnesses, simulators, and fixtures.
• Experience with LLM evaluation (exact/soft match, BLEU/ROUGE, BERTScore, semantic similarity via embeddings), guardrails, and prompt testing.
• Expertise in distributed systems testing latency profiling, resiliency patterns (circuit breakers, retries), chaos engineering, and message queues.
• Familiarity with orchestration frameworks (LangChain, LangGraph, LlamaIndex, DSPy, OpenAI Assistants/Actions, Azure OpenAI orchestration, or similar).
• Proficiency with CI/CD (GitHub Actions/Azure DevOps), observability (OpenTelemetry, Prometheus/Grafana, Datadog), and feature flags/canaries.

Preferred Qualifications

• Experience with multi-agent simulators, agent graph testing, and tooling latency emulation.
• Knowledge of MLOps (model versioning, datasets, evaluation pipelines) and A/B experimentation for LLMs.
• Background in cloud (AWS), serverless, containerization, and event-driven architectures.