Remote AI QA & Evaluation Engineer

Scopic, Inc.; United States; Remote; USD 55,000 - 124,000; Full time

Benefits offered by this job

Paid training

Flexible schedule

Work from anywhere

Annual pay increases

Job summary

Scopic, Inc. is seeking a founding AI Evaluation Engineer for a fully remote contractor role. Create evaluation, testing, and observability infrastructure for production‑grade AI agents handling complex reasoning and multi‑turn conversations.

You will design automated testing pipelines, maintain regression datasets, validate outputs, and implement tracing with OpenTelemetry, LangSmith, Phoenix, and related tools while collaborating across time zones.

Qualifications

• Proficiency in Python or TypeScript, with a focus on API testing, headless execution, and asynchronous systems.
• Hands-on experience building LLM evaluation pipelines or statistical testing frameworks, such as confidence intervals or pass rates over N runs.
• Experience integrating tests into GitHub Actions or similar CI systems using SDK-level execution.
• Expertise in validating structured outputs: JSON schema, OpenAPI specs, and function-calling payloads.
• Familiarity with tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging.
• Ability to independently design and own an evaluation architecture from scratch.
• Strong written and spoken English.
• At least 3 hours of daily overlap with US Central Time business hours.
• Experience with code execution sandboxes or agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI.
• Deep understanding of embedding models and vector databases.
• Experience load testing non-deterministic endpoints.

Responsibilities

• Design and deploy automated CI/CD testing pipelines that validate agent behavior before deployment.
• Build and maintain golden datasets of regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, and latency.
• Implement fast, low-cost code-based checks before relying on LLM-as-a-judge. Examples include JSON schema validation, regex matching, API status checks, and embedding similarity.
• Design test suites for multi-turn conversations, validating context retention, state drift, and safeguards against infinite tool‑calling loops.
• Create adversarial test suites covering prompt injections, malformed inputs, and chaotic tool responses to keep sandboxes and guardrails secure.
• Implement tracing with OpenTelemetry, LangSmith, Phoenix, or similar tools, and reduce LLM API costs through caching and mocking.

Skills

Python

TypeScript

API testing

CI/CD pipelines

Asynchronous systems

English proficiency

Tools

OpenTelemetry

LangSmith

Arize Phoenix

W&B Weave

LangChain

LlamaIndex

AutoGen

CrewAI

Job description

Scopic, Inc. is seeking a founding AI Evaluation Engineer for a fully remote contractor role. Create evaluation, testing, and observability infrastructure for production‑grade AI agents handling complex reasoning and multi‑turn conversations.

You will design automated testing pipelines, maintain regression datasets, validate outputs, and implement tracing with OpenTelemetry, LangSmith, Phoenix, and related tools while collaborating across time zones.