Remote AI Evaluation Engineer

Scopic, Inc.; United States; Remote; USD 55,000 - 124,000; Full time

Benefits offered by this job

Paid training

Flexible schedule

Work from anywhere

Annual pay increases

Job summary

Scopic, Inc. is seeking a founding AI Evaluation Engineer for a fully remote contractor role. Create evaluation, testing, and observability infrastructure for production‑grade AI agents handling complex reasoning and multi‑turn conversations.

You will design automated testing pipelines, maintain regression datasets, validate outputs, and implement tracing with OpenTelemetry, LangSmith, Phoenix, and related tools while collaborating across time zones.

Qualifications

• Proficiency in Python or TypeScript, with a focus on API testing, headless execution, and asynchronous systems.
• Hands-on experience building LLM evaluation pipelines or statistical testing frameworks, such as confidence intervals or pass rates over N runs.
• Experience integrating tests into GitHub Actions or similar CI systems using SDK-level execution.
• Expertise in validating structured outputs: JSON schema, OpenAPI specs, and function-calling payloads.
• Familiarity with tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging.
• Ability to independently design and own an evaluation architecture from scratch.
• Strong written and spoken English.
• At least 3 hours of daily overlap with US Central Time business hours.
• Experience with code execution sandboxes or agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI.
• Deep understanding of embedding models and vector databases.
• Experience load testing non-deterministic endpoints.

Responsibilities

• Design and deploy automated CI/CD testing pipelines that validate agent behavior before deployment.
• Build and maintain golden datasets of regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, and latency.
• Implement fast, low-cost code-based checks before relying on LLM-as-a-judge. Examples include JSON schema validation, regex matching, API status checks, and embedding similarity.
• Design test suites for multi-turn conversations, validating context retention, state drift, and safeguards against infinite tool‑calling loops.
• Create adversarial test suites covering prompt injections, malformed inputs, and chaotic tool responses to keep sandboxes and guardrails secure.
• Implement tracing with OpenTelemetry, LangSmith, Phoenix, or similar tools, and reduce LLM API costs through caching and mocking.

Skills

Python

TypeScript

API testing

CI/CD pipelines

Asynchronous systems

English proficiency

Tools

OpenTelemetry

LangSmith

Arize Phoenix

W&B Weave

LangChain

LlamaIndex

AutoGen

CrewAI

Job description

Begin a career with one of the world’s largest virtual companies where you have flexibility and space to grow, just as we have been able to thanks to our team. Use the latest tools and technology to produce your best work and enjoy remuneration that’s reflective of your skills and experience.

Job Description

This is a remote position.

Are you a QA or eval engineer who would rather build the testing system than run someone else’s? We’re looking for a founding AI Evaluation Engineer to create the evaluation, testing, and observability infrastructure for production‑grade AI agents. These agents handle complex reasoning, multi‑turn conversations, and direct tool execution (APIs, code execution, database calls). You’ll go beyond UI automation and manual prompt checks to build headless, statistical, and deterministic evaluation pipelines for non‑deterministic LLM behavior.

Responsibilities

• Design and deploy automated CI/CD testing pipelines that validate agent behavior before deployment
• Build and maintain golden datasets of regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, and latency
• Implement fast, low‑cost code‑based checks before relying on LLM‑as‑a‑judge. Examples include JSON schema validation, regex matching, API status checks, and embedding similarity.
• Design test suites for multi‑turn conversations, validating context retention, state drift, and safeguards against infinite tool‑calling loops
• Create adversarial test suites covering prompt injections, malformed inputs, and chaotic tool responses to keep sandboxes and guardrails secure
• Implement tracing with OpenTelemetry, LangSmith, Phoenix, or similar tools, and reduce LLM API costs through caching and mocking

Requirements

• Proficiency in Python or TypeScript, with a focus on API testing, headless execution, and asynchronous systems
• Hands‑on experience building LLM evaluation pipelines or statistical testing frameworks, such as confidence intervals or pass rates over N runs
• Experience integrating tests into GitHub Actions or similar CI systems using SDK‑level execution
• Expertise in validating structured outputs: JSON schema, OpenAPI specs, and function‑calling payloads
• Familiarity with tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging
• Ability to independently design and own an evaluation architecture from scratch
• Strong written and spoken English
• At least 3 hours of daily overlap with US Central Time business hours
• Experience with code execution sandboxes or agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI
• Deep understanding of embedding models and vector databases
• Experience load testing non‑deterministic endpoints

The secret ingredients that make us special:

• Your growth is our growth. We invest in your future with paid training and other professional opportunities.
• We’re industry innovators. You’ll work with modern technologies and a team that values strong engineering and continuous improvement.
• Your location. Your schedule. Pick your time‑zone, choose your preferred hours, and work from where you feel most productive.
• Flexibility and freedom are in our DNA. As long as you have a stable internet connection and the drive to thrive, you can work from anywhere.
• A workload you can rely on. We ensure a consistent flow of engaging, challenging product work.
• Recognition and reward. We acknowledge diligence and strong performance with annual pay increases.

Down to business!

• Rate Range: Your starting rate is negotiable depending on your skills and experience.
• This is an hourly, independent contractor position. Compensation is based solely on hours worked. As an independent contractor role, this position does not include PTO.
• Compensation is paid monthly via wire transfer.

Our values:

Scopic offers equal opportunity. We value diversity and do not discriminate based on race, religion, color, marital status, national origin, gender, veteran status, sexual orientation, age, or disability status.

Have the skills, the drive, and the passion to join the Scopic family?