Senior Agentic AI Engineer

Select Minds LLC

Dallas (TX)

Hybrid

USD 180,000 - 240,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Hybrid work arrangement
Competitive salary
Opportunity for advancement

Job summary

Select Minds LLC in Dallas, TX is seeking a Senior/Lead Agentic AI Engineer focused on GenAI evaluation, instrumentation, and agent orchestration. The role emphasizes building production-grade AI applications, multi-agent workflows, and rigorous testing across RAG pipelines and observability tools.

You will collaborate with Engineering, QA, and platform teams to establish best practices, implement automated evaluation frameworks, and drive continuous quality improvements for AI-powered

Qualifications

  • Hands-on experience building production-grade GenAI/agentic AI apps.
  • Experience with orchestration of AI agents and tool calls.
  • Strong background in LLM evaluation, RAG, and observability.

Responsibilities

  • Design, develop, and deploy production-grade agentic AI applications.
  • Build and orchestrate AI agents using LangGraph, LangChain, Bedrock, and AgentCore.
  • Implement multi-agent workflows, tool calling, state management, memory, and orchestration.
  • Create scalable LLM-powered applications and reusable AI components.
  • Deploy agentic workflows with Bedrock AgentCore, handling runtime management, tool integration, and execution orchestration.

Skills

LangGraph
LangChain
Amazon Bedrock
AgentCore
AI agents orchestration
Tool calling
Multi-agent workflows

Tools

OpenTelemetry
LangSmith
Langfuse
REST APIs

Job description

Benefits:

hybrid

Competitive salary

Opportunity for advancement

Job Title: Senior / Lead Agentic AI Engineer – LLM Evaluation & Observability

Location: Dallas, TX - Hybrid/Remote; may need to visit for occasional office meetings. Duration: Long‑Term Experience: 8+ years Interview: Client round, in‑person

Important: This is a hands‑on engineering role. Candidates with practical experience building and evaluating AI applications are preferred over those whose experience is limited to AI strategy, architecture documentation, conventional QA automation, or using AI coding assistants.

We seek a Senior/Lead AI Engineer with experience building, orchestrating, monitoring, and evaluating production‑grade Generative AI and Agentic AI applications. The ideal candidate is expert in Amazon Bedrock, AgentCore, LangGraph, LangChain, RAG, LLM observability, and evaluation frameworks. The role focuses on creating reliable, scalable, production‑ready AI applications, emphasizing agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring.

The candidate will collaborate with Engineering, QA, and platform teams to establish best practices for building, testing, monitoring, and improving AI‑powered applications. The testing focus is on GenAI evaluation, RAG pipeline assessment, agent behavior validation, LLM quality benchmarking, and automated evaluation frameworks, not conventional UI test automation.

Key Responsibilities
  1. Agentic AI & GenAI Engineering
    • Design, develop, and deploy production‑grade agentic and generative AI applications.
    • Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
    • Implement multi‑agent workflows, tool calling, state management, memory, and orchestration.
    • Create scalable LLM‑powered applications and reusable AI components.
    • Deploy agentic workflows with Amazon Bedrock AgentCore, handling runtime management, tool integration, and execution orchestration.
  2. RAG & LLM Application Development
    • Design and implement Retrieval‑Augmented Generation (RAG) solutions.
    • Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management.
    • Optimize retrieval accuracy, response quality, latency, and cost.
    • Apply grounding, hallucination reduction, and context‑relevance techniques.
  3. LLM Evaluation & AI Testing
    • Build automated evaluation frameworks with RAGAS, DeepEval, and similar tools.
    • Evaluate RAG pipelines using metrics such as Faithfulness, Answer Relevancy, Context Precision, Context Recall, and Answer Correctness.
    • Implement LLM‑as‑a‑Judge techniques to assess generated responses, retrieval quality, agent behavior, and workflow execution.
    • Establish evaluation datasets, benchmarks, and regression‑testing pipelines for continuous quality assessment.
  4. 3a. Agent Evaluation and Observability
    • Implement end‑to‑end tracing of multi‑agent workflows, including tool invocation, agent handoffs, execution paths, and failure handling.
    • Establish observability practices to detect hallucinations, reasoning failures, tool‑calling errors, and execution inconsistencies.
  5. AI Observability & Monitoring
    • Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse.
    • Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance.
    • Develop debugging mechanisms for complex multi‑step agentic workflows.
    • Create monitoring practices to detect performance degradation and operational issues.
  6. AWS AI Platform & Production Engineering
    • Deploy AI applications using Amazon Bedrock and Bedrock AgentCore.
    • Integrate foundation models, agent frameworks, APIs, and enterprise data sources.
    • Apply production‑readiness practices covering scalability, reliability, security, and performance.
    • Support CI/CD integration and automated evaluation of AI applications.
    • Optimize model selection, inference costs, latency, and resource utilization.
  7. Technical Leadership & Collaboration
    • Provide technical guidance to engineering and QA teams.
    • Establish reusable frameworks, implementation patterns, and best practices.
    • Collaborate with architecture, DevOps, QA, and engineering teams.
    • Participate in design discussions, code reviews, proof‑of‑concept development, and implementation.
    • Mentor engineers and help teams adopt effective AI development and testing practices.
Required Technical Skills
Agentic AI & Orchestration
  • LangGraph
  • LangChain
  • Amazon Bedrock
  • Amazon Bedrock AgentCore
  • AI agents, multi‑agent orchestration, tool calling, workflow management
Generative AI & RAG
  • Large Language Models (LLMs)
  • Retrieval‑Augmented Generation (RAG)
  • Embeddings and vector search
  • Prompt engineering and context management
  • LLM application development
LLM Evaluation & Testing
  • DeepEval
  • LLM‑as‑a‑Judge techniques
  • Evaluation metrics and benchmarking
  • Automated evaluation and regression testing
  • Assessment of hallucination, faithfulness, relevance, and response quality
  • Hands‑on experience with RAGAS, DeepEval, or equivalent frameworks
  • Strong understanding of RAG evaluation metrics (Faithfulness, Answer Relevancy, Context Precision, Context Recall, Answer Correctness)
  • Experience building automated evaluation pipelines for RAG‑based applications
AI Observability & Monitoring
  • OpenTelemetry
  • LangSmith
  • Langfuse
  • LLM tracing, debugging, and performance monitoring
Production Engineering
  • Deploying and operating production‑grade GenAI applications
  • API integration and backend engineering
  • CI/CD and automated testing integration
  • Scalability, reliability, security, and performance optimization
  • Strong hands‑on programming experience in Python.
  • Experience developing REST APIs and integrating LLM frameworks with enterprise applications.
  • Familiarity with asynchronous programming, API integrations, and backend development.
Preferred Qualifications
  • 8+ years of software engineering or IT experience.
  • Demonstrated hands‑on experience building and deploying GenAI and Agentic AI applications.
  • Recent experience in LLM engineering, agent orchestration, and AI evaluation.
  • Experience with enterprise‑scale AI implementations.
  • Familiarity with AWS cloud services and enterprise AI security practices.
  • Experience establishing AI quality standards and evaluation methodologies.
  • Strong problem‑solving, communication, and technical collaboration skills.
What Success Looks Like
  • Development and deployment of production‑grade agentic AI applications.
  • Reliable orchestration and execution of AI agents and workflows.
  • Effective LLM evaluation and automated AI testing frameworks.
  • Improved AI application accuracy, reliability, observability, and performance.
  • Reusable engineering practices for building and maintaining enterprise GenAI solutions.

This is a remote position.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

ZR_3092_JOB
ZR_3092_JOB

Simple Solutions • Jacksonville (FL)

Hybrid
USD 160,000 - 230,000
Senior Backend Ai Engineer
Senior Backend Ai Engineer

Neshent Technologies • Newark (NJ)

On-site
USD 180,000 - 240,000
Hiring: AI Engineer / Generative AI Engineer | (Agentic AI,AWS Bedrock AgentCore,Claude Agent SDK,Langfuse) | Los Angeles, CA (Hybrid)
Hiring: AI Engineer / Generative AI Engineer | (Agentic AI,AWS Bedrock AgentCore,Claude Agent SDK,Langfuse) | Los Angeles, CA (Hybrid)

BlackCube Labs • Los Angeles (CA)

Hybrid
USD 120,000 - 180,000
Ai Software Architect Llm Agentic Systems Msys Tech India Pvt Ltd Chennai
Ai Software Architect Llm Agentic Systems Msys Tech India Pvt Ltd Chennai

Vibehackers • United States

On-site
USD 180,000 - 260,000
Bizoforce Hiring – AI Engineer — Agentic Systems | Austin, Texas, USA
Bizoforce Hiring – AI Engineer — Agentic Systems | Austin, Texas, USA

Confidential • Austin (TX)

On-site
USD 150,000 - 210,000
Staff AI Engineer
Staff AI Engineer

Harnham • New York (NY)

Remote
USD 160,000 - 185,000
Agentic AI Engineer
Agentic AI Engineer

Aimlroles • United States

Remote
USD 62,000 - 93,000
Agentic AI Engineer
Agentic AI Engineer

Select Minds LLC • Dallas (TX)

Hybrid
USD 150,000 - 200,000
Opportunity for advancement
AI Engineer / Generative AI Engineer
AI Engineer / Generative AI Engineer

Nexiva • Los Angeles (CA)

On-site
USD 140,000 - 230,000
Senior Agentic AI Engineer — Remote
Senior Agentic AI Engineer — Remote

Build • United States

Hybrid
USD 150,000 - 190,000