Tech AI Data Delivery Manager

OpenKrill; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

OpenKrill is seeking an engineer to design and run AI data projects and to build the automation that powers quality management, project management, and data analytics. You will pair deterministic tooling with LLM agents, calibrate QA checks, and own end-to-end data collection and reporting workflows.

You’ll work with Claude and our enterprise agentic environment to implement task specs, data connections, and monitoring, while documenting runbooks for operational teams.

Qualifications

• Intermediate JavaScript and Python for production automation.
• Hands-on agent building in Claude and enterprise environment.
• LLM evaluation: prompt design, rubric calibration, human judgement checks.
• Data analytics experience in data collection or annotation.
• Experience managing projects with external vendors or contributors.

Responsibilities

• Build hybrid automation combining deterministic scripts with LLM agents.
• Set up and run agents in Claude and our enterprise agentic environment.
• Design QA workflows, rubrics, and scorecards; calibrate automated checks against human reviewers.
• Run data collection and annotation projects end to end with tooling for coordinators.
• Collect, clean, and analyze production and quality data and turn into actionable reporting.
• Verify everything by testing automation and building continuous auditing workflows.
• Document specs and runbooks to enable others to operate and extend the system.

Skills

JavaScript

Python

Claude agent building

LLM evaluation

Data analytics

Vendor management

Decision making

Tools

Claude

Job description

Role Summary

You’ll design and run AI data projects and build the automation behind them, combining deterministic tooling with LLM agents to deliver solutions in quality management, project management, and data analytics.

Key Responsibilities

• Build hybrid automation: Pair deterministic components (scripts, validation rules, trackers, scheduled jobs) with stochastic ones (LLM agents that score, classify, check, report, etc.). Decide which steps must be rules and which can be models, and design the hand-offs between them.
• Set up and run agents in Claude and in our native enterprise agentic environment: task specs, connections to data and tools, schedules, monitoring and failure handling.
• Quality management: Design QA workflows, guidelines, rubrics and scorecards; calibrate automated checks against human reviewers; turn review findings into error categories you can measure and fix.
• Project management: Run data collection and annotation projects end to end (scoping, vendors, workflows, tracking, delivery) and build the tooling that lets coordinators run the day.
• Data analytics: Collect, clean, and analyse production and quality data (throughput, rework, error rates, contributor performance) and turn it into reporting that teams and clients act on.
• Verify everything: Test automation before release and build continuous auditing parallel workflows to ensure quality.
• Document so others can run it: short specs, tickets and runbooks that let coordinators and engineers operate and extend what you build.

Required Qualifications

• Intermediate JavaScript and Python: enough to build and maintain automation in production, not just prototypes.
• Hands-on agent building in Claude (projects, skills, connectors, scheduled tasks), and the ability to do the same in our native enterprise agentic environment (connecting different APIs with our native infrastructure).
• LLM evaluation experience: prompt and rubric design, calibration against human judgement, and the skepticism to catch a confident wrong answer.
• A data analytics background, with experience in the data collection or annotation industry.
• Experience managing projects with external vendors or contributors and multiple stakeholders.
• Comfort making calls on incomplete information, and knowing how to make and own your decisions.