Founding Remote AI Evaluation Engineer — Testing Pipelines

Scopic; United States; Remote; USD 83,000 - 165,000; Part time

Benefits offered by this job

Paid training

Work from anywhere

Competitive hourly rate

Monthly wire transfer payments

No PTO (contractor)

Job summary

Scopic is seeking a founding AI Evaluation Engineer to create evaluation, testing, and observability infrastructure for production‑grade AI agents. You will design headless, deterministic pipelines for non‑deterministic LLM behavior.

You will build CI/CD testing, golden datasets, and adversarial tests, focusing on JSON schema validation, embedding similarity, and tool execution safety. This is an hourly, independent contractor role that can be done remotely.

Qualifications

• Proficiency in Python or TypeScript, with API testing and headless execution.
• Hands-on experience building evaluation pipelines or statistical tests.
• Experience integrating tests into GitHub Actions or CI systems.
• Expertise in validating JSON schema, OpenAPI specs, and function‑calling payloads.

Responsibilities

• Design and deploy automated CI/CD testing pipelines for agent behavior.
• Build golden datasets for regression prompts, edge cases, and synthetic scenarios.
• Implement fast, low-cost code-based checks before LLM-as-a-judge (JSON schema, regex, API health).
• Design multi-turn test suites to validate context retention and safeguards.
• Create adversarial test suites for prompt injections and chaotic tool responses.

Skills

Python/TypeScript

API testing

CI/CD

JSON schema

OpenTelemetry

LangChain LangSmith

Code execution sandbox

Vector databases

English fluency

Tools

LangChain

LlamaIndex

AutoGen

CrewAI

Job description

Scopic is seeking a founding AI Evaluation Engineer to create evaluation, testing, and observability infrastructure for production‑grade AI agents. You will design headless, deterministic pipelines for non‑deterministic LLM behavior.

You will build CI/CD testing, golden datasets, and adversarial tests, focusing on JSON schema validation, embedding similarity, and tool execution safety. This is an hourly, independent contractor role that can be done remotely.