Senior AI Tools Testing Engineer

Select Minds LLC; Dallas (TX); Ibrido; USD 140.000 - 190.000; Tempo pieno

Vantaggi offerti da questo lavoro

Hybrid work model

Competitive salary

Opportunity for advancement

Descrizione del lavoro

Select Minds LLC in Dallas, TX is seeking a Senior/Lead AI Engineer specializing in generative and agentic AI. This hands-on role focuses on building, evaluating, and observability of production-grade GenAI applications with tight collaboration across Engineering, QA, and platform teams.

You will work with LangGraph, LangChain, Bedrock, and AgentCore to implement multi-agent workflows, RAG pipelines, and rigorous evaluation frameworks, delivering reliable, scalable AI solutions while driving

Competenze

• Hands-on experience building and evaluating production-grade GenAI applications.
• 8+ years software engineering with AI systems.
• Experience with agent orchestration and evaluation frameworks.
• Familiarity with AWS and enterprise AI security practices.

Mansioni

• Design, develop, and deploy production-grade agentic and generative AI applications.
• Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
• Implement multi-agent workflows, tool calling, memory, and orchestration.
• Create scalable LLM-powered applications and reusable AI components.
• Deploy agentic workflows with Bedrock AgentCore, handling runtime management.
• Design and implement RAG solutions and document ingestion pipelines.
• Design automated evaluation frameworks and LLM-as-a-Judge techniques.
• Establish end-to-end tracing and observability for multi-agent workflows.
• Lead AI production engineering with CI/CD and security considerations.
• Mentor engineers and drive adoption of AI testing practices.

Conoscenze

AI agents orchestration

LangChain

LangGraph

AgentCore

Tool calls

LLM development

RAG integration

Observability concepts

Strumenti

LangGraph

LangChain

Amazon Bedrock

Amazon Bedrock AgentCore

OpenTelemetry

Descrizione del lavoro

• hybrid
• Competitive salary
• Opportunity for advancement

Job Title: Senior / Lead Agentic AI Engineer – LLM Evaluation & Observability

Location: Dallas, TX (Hybrid)

Duration: Long‑Term

Experience: 8+ years

Interview: Client round, in‑person

Important: This is a hands‑on engineering role. Candidates with practical experience building and evaluating AI applications are preferred over those whose experience is limited to AI strategy, architecture documentation, conventional QA automation, or using AI coding assistants.

We seek a Senior/Lead AI Engineer with experience building, orchestrating, monitoring, and evaluating production‑grade Generative AI and Agentic AI applications. The ideal candidate is expert in Amazon Bedrock, AgentCore, LangGraph, LangChain, RAG, LLM observability, and evaluation frameworks. The role focuses on creating reliable, scalable, production‑ready AI applications, emphasizing agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring.

The candidate will collaborate with Engineering, QA, and platform teams to establish best practices for building, testing, monitoring, and improving AI‑powered applications. The testing focus is on GenAI evaluation, RAG pipeline assessment, agent behavior validation, LLM quality benchmarking, and automated evaluation frameworks, not conventional UI test automation.

Key Responsibilities

1. Agentic AI & GenAI Engineering

• Design, develop, and deploy production‑grade agentic and generative AI applications.
• Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
• Implement multi‑agent workflows, tool calling, state management, memory, and orchestration.
• Create scalable LLM‑powered applications and reusable AI components.
• Deploy agentic workflows with Amazon Bedrock AgentCore, handling runtime management, tool integration, and execution orchestration.

2. RAG & LLM Application Development

• Design and implement Retrieval‑Augmented Generation (RAG) solutions.
• Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management.
• Optimize retrieval accuracy, response quality, latency, and cost.
• Apply grounding, hallucination reduction, and context‑relevance techniques.

3. LLM Evaluation & AI Testing

• Build automated evaluation frameworks with RAGAS, DeepEval, and similar tools.
• Evaluate RAG pipelines using metrics such as Faithfulness, Answer Relevancy, Context Precision, Context Recall, and Answer Correctness.
• Implement LLM‑as‑a‑Judge techniques to assess generated responses, retrieval quality, agent behavior, and workflow execution.
• Establish evaluation datasets, benchmarks, and regression‑testing pipelines for continuous quality assessment.

3a. Agent Evaluation and Observability

• Implement end‑to‑end tracing of multi‑agent workflows, including tool invocation, agent handoffs, execution paths, and failure handling.
• Establish observability practices to detect hallucinations, reasoning failures, tool‑calling errors, and execution inconsistencies.

4. AI Observability & Monitoring

• Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse.
• Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance.
• Develop debugging mechanisms for complex multi‑step agentic workflows.
• Create monitoring practices to detect performance degradation and operational issues.

5. AWS AI Platform & Production Engineering

• Deploy AI applications using Amazon Bedrock and Bedrock AgentCore.
• Integrate foundation models, agent frameworks, APIs, and enterprise data sources.
• Apply production‑readiness practices covering scalability, reliability, security, and performance.
• Support CI/CD integration and automated evaluation of AI applications.
• Optimize model selection, inference costs, latency, and resource utilization.

6. Technical Leadership & Collaboration

• Provide technical guidance to engineering and QA teams.
• Establish reusable frameworks, implementation patterns, and best practices.
• Collaborate with architecture, DevOps, QA, and engineering teams.
• Participate in design discussions, code reviews, proof‑of‑concept development, and implementation.
• Mentor engineers and help teams adopt effective AI development and testing practices.

Required Technical Skills

Agentic AI & Orchestration

• LangGraph
• LangChain
• Amazon Bedrock
• Amazon Bedrock AgentCore
• AI agents, multi‑agent orchestration, tool calling, workflow management

Generative AI & RAG

• Large Language Models (LLMs)
• Retrieval‑Augmented Generation (RAG)
• Embeddings and vector search
• Prompt engineering and context management
• LLM application development

LLM Evaluation & Testing

• DeepEval
• LLM‑as‑a‑Judge techniques
• Evaluation metrics and benchmarking
• Automated evaluation and regression testing
• Assessment of hallucination, faithfulness, relevance, and response quality
• Hands‑on experience with RAGAS, DeepEval, or equivalent frameworks
• Strong understanding of RAG evaluation metrics (Faithfulness, Answer Relevancy, Context Precision, Context Recall, Answer Correctness)
• Experience building automated evaluation pipelines for RAG‑based applications

AI Observability & Monitoring

• OpenTelemetry
• LangSmith
• Langfuse
• LLM tracing, debugging, and performance monitoring

Production Engineering

• Deploying and operating production‑grade GenAI applications
• API integration and backend engineering
• CI/CD and automated testing integration
• Scalability, reliability, security, and performance optimization
• Strong hands‑on programming experience in Python.
• Experience developing REST APIs and integrating LLM frameworks with enterprise applications.
• Familiarity with asynchronous programming, API integrations, and backend development.

Preferred Qualifications

• 8+ years of software engineering or IT experience.
• Demonstrated hands‑on experience building and deploying GenAI and Agentic AI applications.
• Recent experience in LLM engineering, agent orchestration, and AI evaluation.
• Experience with enterprise‑scale AI implementations.
• Familiarity with AWS cloud services and enterprise AI security practices.
• Experience establishing AI quality standards and evaluation methodologies.
• Strong problem‑solving, communication, and technical collaboration skills.

What Success Looks Like

• Development and deployment of production‑grade agentic AI applications.
• Reliable orchestration and execution of AI agents and workflows.
• Effective LLM evaluation and automated AI testing frameworks.
• Improved AI application accuracy, reliability, observability, and performance.
• Reusable engineering practices for building and maintaining enterprise GenAI solutions.

Flexible work from home options available.