Remote AI Evaluation Engineer

Scopic; United States; Remote; USD 83,000 - 165,000; Part time

Benefits offered by this job

Paid training

Work from anywhere

Competitive hourly rate

Monthly wire transfer payments

No PTO (contractor)

Job summary

Scopic is seeking a founding AI Evaluation Engineer to create evaluation, testing, and observability infrastructure for production‑grade AI agents. You will design headless, deterministic pipelines for non‑deterministic LLM behavior.

You will build CI/CD testing, golden datasets, and adversarial tests, focusing on JSON schema validation, embedding similarity, and tool execution safety. This is an hourly, independent contractor role that can be done remotely.

Qualifications

• Proficiency in Python or TypeScript, with API testing and headless execution.
• Hands-on experience building evaluation pipelines or statistical tests.
• Experience integrating tests into GitHub Actions or CI systems.
• Expertise in validating JSON schema, OpenAPI specs, and function‑calling payloads.

Responsibilities

• Design and deploy automated CI/CD testing pipelines for agent behavior.
• Build golden datasets for regression prompts, edge cases, and synthetic scenarios.
• Implement fast, low-cost code-based checks before LLM-as-a-judge (JSON schema, regex, API health).
• Design multi-turn test suites to validate context retention and safeguards.
• Create adversarial test suites for prompt injections and chaotic tool responses.

Skills

Python/TypeScript

API testing

CI/CD

JSON schema

OpenTelemetry

LangChain LangSmith

Code execution sandbox

Vector databases

English fluency

Tools

LangChain

LlamaIndex

AutoGen

CrewAI

Job description

Category: IT Services Location:

Are you a QA or eval engineer who would rather build the testing system than run someone else's? We're looking for a founding AI Evaluation Engineer to create the evaluation, testing, and observability infrastructure for production-grade AI agents. These agents handle complex reasoning, multi-turn conversations, and direct tool execution (APIs, code execution, database calls). You'll go beyond UI automation and manual prompt checks to build headless, statistical, and deterministic evaluation pipelines for non-deterministic LLM behavior.

Responsibilities

• Design and deploy automated CI/CD testing pipelines that validate agent behavior before deployment
• Build and maintain golden datasets of regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, and latency
• Implement fast, low-cost code-based checks before relying on LLM-as-a-judge. Examples include JSON schema validation, regex matching, API status checks, and embedding similarity.
• Design test suites for multi-turn conversations, validating context retention, state drift, and safeguards against infinite tool‑calling loops
• Create adversarial test suites covering prompt injections, malformed inputs, and chaotic tool responses to keep sandboxes and guardrails secure
• Implement tracing with OpenTelemetry, LangSmith, Phoenix, or similar tools, and reduce LLM API costs through caching and mocking

Requirements

• Proficiency in Python or TypeScript, with a focus on API testing, headless execution, and asynchronous systems
• Hands‑on experience building LLM evaluation pipelines or statistical testing frameworks, such as confidence intervals or pass rates over N runs
• Experience integrating tests into GitHub Actions or similar CI systems using SDK‑level execution
• Expertise in validating structured outputs: JSON schema, OpenAPI specs, and function‑calling payloads
• Familiarity with tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging
• Ability to independently design and own an evaluation architecture from scratch
• Strong written and spoken English
• At least 3 hours of daily overlap with US Central Time business hours
• Experience with code execution sandboxes or agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI
• Deep understanding of embedding models and vector databases
• Experience load testing non-deterministic endpoints

The secret ingredients that make us special:

• Your growth is our growth. We invest in your future with paid training and other professional opportunities.
• We're industry innovators. You'll work with modern technologies and a team that values strong engineering and continuous improvement.
• Your location. Your schedule. Pick your time-zone, choose your preferred hours, and work from where you feel most productive.
• Flexibility and freedom are in our DNA. As long as you have a stable internet connection and the drive to thrive, you can work from anywhere.
• A workload you can rely on. We ensure a consistent flow of engaging, challenging product work.
• Recognition and reward. We acknowledge diligence and strong performance with annual pay increases.
• Down to business!
• Rate Range: Your starting rate is negotiable depending on your skills and experience.
• This is an hourly, independent contractor position.
• Compensation is based solely on hours worked.
• As an independent contractor role, this position does not include PTO.
• Compensation is paid monthly via wire transfer.

Our values

Scopic offers equal opportunity. We value diversity and do not discriminate based on race, religion, color, marital status, national origin, gender, veteran status, sexual orientation, age, or disability status.

Have the skills, the drive, and the passion to join the Scopic family?