Sr. Applied AI Engineer

O.C. Tanner; Salt Lake City (UT); On-site; USD 120,000 - 170,000; Full time

Job summary

O.C. Tanner is seeking an Applied AI Engineer to turn AI capability into secure, governed production systems that drive real value in user workflows.

You will work at the intersection of software engineering, AI platform engineering, and responsible AI, partnering with Product, UX, Design, Architecture, Security, and Engineering to build useful and safe AI experiences. You will design, deploy, and operate agentic AI systems with orchestration, memory, tool calling, and robust evaluation,

Qualifications

• 5+ years of software engineering experience with strong Python proficiency.
• 2+ years building production ML or agentic AI systems.
• 1+ years hands‑on experience with agentic frameworks (LangGraph, CrewAI, AutoGen, or equivalent).
• Built production AI systems including agents, MCP servers, multi‑step reasoning, and multi‑turn conversation.
• Deployed RAG systems including embedding models, vector databases, hybrid search, and retrieval optimization.
• Designed LLM strategies covering tool calling, structured outputs, prompt engineering, and context window management.
• Implemented AI safety and evaluation pipelines covering bias detection, PII leakage, faithfulness scoring, toxicity, and prompt injection mitigation.
• Optimized models for inference efficiency, latency, and cost management.

Responsibilities

• Design, build, deploy, and support production‑grade agentic AI systems that operate against explicit goals, constraints, policies, and guardrails.
• Build agent orchestration patterns for multi‑step workflows, tool calling, MCP servers, state management, memory, retries, recovery paths, and human‑in‑the‑loop controls.
• Partner closely with Product, UX, Design, Architecture, Security, and Engineering teams to create AI experiences that are useful, understandable, reliable, and aligned with real user workflows.
• Design user‑centered AI interactions, including conversational flows, feedback loops, confidence handling, explainability, graceful failure modes, escalation paths, and clear boundaries for autonomous behavior.
• Develop and operate RAG systems that ground model behavior in enterprise knowledge, including ingestion, chunking, embeddings, vector and hybrid retrieval, reranking, retrieval evaluation, and citation or traceability strategies.
• Define and implement evaluation frameworks for AI systems, including offline test sets, regression suites, adversarial testing, groundedness and faithfulness scoring, task completion metrics, and production quality monitoring.
• Instrument agentic systems for observability, including traces of model calls, prompts, tool usage, decisions, retrieved context, latency, cost, errors, and user feedback.
• Establish safeguards for responsible AI use, including prompt injection defense, data access controls, PII protection, bias and toxicity detection, misuse prevention, audit logging, and policy enforcement.
• Optimize model selection, prompts, context windows, caching, routing, inference patterns, latency, throughput, reliability, and cost across production workloads.
• Mentor engineers on applied AI practices, including prompt and context engineering, agent design, RAG, evaluation, safety, observability, and production support.
• Stay current with emerging AI platforms, frameworks, models, and standards.

Skills

Python

Software engineering

ML systems

Agentic frameworks

RAG systems

LLM strategies

AI safety & eval

Performance optimization

Education

Bachelor's degree

Tools

LangGraph

CrewAI

AutoGen

LangChain

Kafka

AWS Bedrock

Terraform

OpenTelemetry

Job description

AI is becoming part of the product and platform architecture we need to build, operate, and scale. We are looking for an Applied AI Engineer who can turn AI capability into secure, measurable, governed production systems, not prototypes or demos. This person will help define how O.C. Tanner builds agentic systems that pursue goals, use tools, follow guardrails, recover from failure, and deliver real value inside user workflows.

This role sits at the intersection of software engineering, product experience, AI platform engineering, and responsible AI. You will partner with Product, UX, Design, Architecture, Security, and Engineering to build AI experiences that are useful, understandable, reliable, and safe to operate in production. The right person has hands‑on experience building agentic systems with orchestration, tool calling, memory or state, RAG, evaluation, observability, and human‑in‑the‑loop controls.

Responsibilities

• Design, build, deploy, and support production‑grade agentic AI systems that operate against explicit goals, constraints, policies, and guardrails.
• Build agent orchestration patterns for multi‑step workflows, tool calling, MCP servers, state management, memory, retries, recovery paths, and human‑in‑the‑loop controls.
• Partner closely with Product, UX, Design, Architecture, Security, and Engineering teams to create AI experiences that are useful, understandable, reliable, and aligned with real user workflows.
• Design user‑centered AI interactions, including conversational flows, feedback loops, confidence handling, explainability, graceful failure modes, escalation paths, and clear boundaries for autonomous behavior.
• Develop and operate RAG systems that ground model behavior in enterprise knowledge, including ingestion, chunking, embeddings, vector and hybrid retrieval, reranking, retrieval evaluation, and citation or traceability strategies.
• Define and implement evaluation frameworks for AI systems, including offline test sets, regression suites, adversarial testing, groundedness and faithfulness scoring, task completion metrics, and production quality monitoring.
• Instrument agentic systems for observability, including traces of model calls, prompts, tool usage, decisions, retrieved context, latency, cost, errors, and user feedback.
• Establish safeguards for responsible AI use, including prompt injection defense, data access controls, PII protection, bias and toxicity detection, misuse prevention, audit logging, and policy enforcement.
• Optimize model selection, prompts, context windows, caching, routing, inference patterns, latency, throughput, reliability, and cost across production workloads.
• Mentor engineers on applied AI practices, including prompt and context engineering, agent design, RAG, evaluation, safety, observability, and production support.
• Stay current with emerging AI platforms, frameworks, models, and standards.

Our stack

• Python / FastAPI microservices
• LangChain / LangGraph
• GraphQL / REST
• Kafka
• AWS Bedrock
• OpenTelemetry
• Terraform

Qualifications

Required Qualifications

• 5+ years of software engineering experience with strong Python proficiency
• 2+ years building production ML or agentic AI systems
• 1+ years hands‑on experience with agentic frameworks (LangGraph, CrewAI, AutoGen, or equivalent)
• Built production AI systems including agents, MCP servers, multi‑step reasoning, and multi‑turn conversation
• Deployed RAG systems including embedding models, vector databases, hybrid search, and retrieval optimization
• Designed LLM strategies covering tool calling, structured outputs, prompt engineering, and context window management
• Implemented AI safety and evaluation pipelines covering bias detection, PII leakage, faithfulness scoring, toxicity, and prompt injection mitigation
• Optimized models for inference efficiency, latency, and cost management

Strongly Preferred

• Bachelor's degree in Computer Science, Machine Learning, or a related field
• AWS Certified Machine Learning Engineer – Associate or equivalent
• Cloud AI infrastructure management using AWS services and Terraform
• AI observability experience with OpenTelemetry, Langfuse, or equivalent