AI Engineer – Agentic Systems & RLAIF

Capgemini; Dallas (TX); On-site; USD 54,000 - 85,000; Part time

Benefits offered by this job

Medical benefits

Dental benefits

Vision benefits

Retirement benefits

Job summary

Tundra Technical Solutions is seeking an AI Engineer to own the engineering behind a Reinforcement Learning from AI Feedback (RLAIF) plane. You will detect capability and context failures, convert root causes into actionable variants, and push only approved, versioned changes into production.

The role covers building agent runtimes, observability instrumentation, evaluation harnesses, and end-to-end deployment on Google Cloud with strong collaboration across architects, data scientists, SREs,

Qualifications

• Demonstrable experience building and running AI agents in production with ownership of reliability, latency, cost and incident response.
• Hands-on experience with AI observability and evals: tracing frameworks, LLM-as-judge design, and regression testing.
• Deep practical experience with LangGraph and LangChain, including planning, sub-agents, tool use, memory and human-in-the-loop.

Responsibilities

• Build the agent runtime and multi-agent workflows with governance and versioning.
• Instrument for observability across agents, tools, memory, and knowledge sources.
• Own golden sets, offline/online evals, and AI-judge pipelines with bias control.
• Drive harness optimization with one-change-at-a-time experiments and safe release workflows.
• Separate optimization paths for harness, contextual-policy, and model-level changes and ensure governed updates.
• Ship on Google Cloud (Vertex AI, GKE/Cloud Run, BigQuery, Pub/Sub) with CI/CD and IaC.
• Collaborate with architects, data scientists, SRE, and Client stakeholders; mentor engineers.

Skills

Python

AI agents

LangGraph

LangChain

Observability

CI/CD

MLOps

Root cause analysis

Async services

Distributed systems

Tools

Vertex AI

GKE

Cloud Run

BigQuery

Pub/Sub

Job description

AI Engineer – Agentic Systems & RLAIF (Contract)

Contract (1 month 26 days)

Published 2 days ago

Python Engineering

root cause analysis

mlops

CI/CD

AI Agent Development

We are building an RLAIF (Reinforcement Learning from AI Feedback) Plane: a governed learning system that turns production evidence from Client's customer-facing agents into better agent decisions—safely, measurably, and reversibly. Today, small errors compound across interconnected agents, missing capabilities and stale business context silently produce fluent-but-wrong answers, and prompt, policy, and model changes lack one governed path to independent proof.

As an AI Engineer you will own the engineering behind that path: detecting capability and context failures before they become customer-visible, converting root-cause analysis into one-factor variants, running them through hard gates and calibrated AI judges, and promoting only what is approved, versioned, monitored, and reversible.

What You Will Do:

Build the agent runtime- Design, build, and operate production agents and multi-agent workflows using LangGraph, LangChain, and deep-agent patterns—governed, monitored, and versioned from day one.

Instrument for observability- Implement tracing, telemetry, and failure taxonomies across agents, tools, memory, and knowledge sources so every failure has a reproducible trail back to its root cause.

Engineer the evaluation harness- Own golden sets, offline and online evals, and AI-judge pipelines with order-swap bias control and human calibration loops; report against business outcomes, not just model metrics.

Drive harness optimization- Use observed failures to improve prompts, tools, skills, and rules through one-change-at-a-time experiments with regression-safe release: hard gates, approval, canary, and rollback.

Separate the optimization paths- Keep harness, contextual-policy, and model-level changes distinct and independently evaluated; build the mechanisms that detect capability and knowledge-coverage gaps and propose governed updates.

Ship on GCP- Deploy and scale the RLAIF Plane on Google Cloud (Vertex AI, GKE/Cloud Run, BigQuery, Pub/Sub) with CI/CD, IaC, and cost/latency controls.

Collaborate and lead- Partner with architects, data scientists, SRE, and Client business stakeholders; document decisions, mentor engineers, and raise the bar on agent engineering practice.

Key Responsibilities:

Ingesting and analyzing traces

Reconstructing the sequence of events, tool calls, and business flow from traces

Creating prompt/context/routing variants

Defining and implementing evaluation dimensions

Building and calibrating AI judges

Designing scoring and reward-signal frameworks

Generating autonomous optimization recommendations

Identifying root causes such as:

Hallucinations

Misrouting

Policy/compliance issue

Missing or underperforming tools

Re-configuration opportunities (agent) for identification and recommendation and project outcomes.

What You Bring (Required):

Demonstrable experience building and running AI agents in production—shipped systems serving real users, with ownership of reliability, latency, cost, and incident response.

Hands-on experience with AI observability and evals: tracing frameworks (e.g., LangSmith, Arize Phoenix, OpenTelemetry), LLM-as-judge design, golden-set curation, and regression testing.

Deep, practical experience with LangGraph and LangChain, including deep-agent patterns (planning, sub-agents, tool use, memory, human-in-the-loop, and state persistence).

Strong Python engineering skills; comfortable with async services, APIs, containers, and testing discipline in a CI/CD environment.

Working understanding of RLAIF/RLHF concepts, prompt and policy optimization, and how to isolate a single causal change in a complex system.

5+ years of software engineering experience, including 2+ years focused on LLM or agentic applications.

Preferred Qualifications:

Cloud experience on Google Cloud (Vertex AI, GKE, Cloud Run, BigQuery, Pub/Sub, IAM); GCP certification is a plus.

Experience with retrieval, enterprise knowledge bases, and memory systems with provenance, trust scoring, and decay.

Familiarity with governed release practices in regulated or large-enterprise settings (approval workflows, canary, rollback, audit trails).

Telecom or large-scale customer-service domain exposure; experience with MLOps tooling and feature flags.

The pay range that the employer in good faith reasonably expects to pay for this position is $39.30/hour - $61.40/hour. Our benefits include medical, dental, vision and retirement benefits. Applications will be accepted on an ongoing basis.

Tundra Technical Solutions is among North America’s leading providers of Staffing and Consulting Services. Our success and our clients’ success are built on a foundation of service excellence. We are an equal opportunity employer, and we do not discriminate on the basis of race, religion, color, national origin, sex, sexual orientation, age, veteran status, disability, genetic information, or other applicable legally protected characteristic. Qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable law, including the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. Unincorporated LA County workers: we reasonably believe that criminal history may have a direct, adverse and negative relationship with the following job duties, potentially resulting in the withdrawal of a conditional offer of employment: client provided property, including hardware (both of which may include data) entrusted to you from theft, loss or damage; return all portable client computer hardware in your possession (including the data contained therein) upon completion of the assignment, and; maintain the confidentiality of client proprietary, confidential, or non-public information. In addition, job duties require access to secure and protected client information technology systems and related data security obligations.