AI Evaluation Engineering Lead

Caterpillar Inc; Chicago (IL); On-site; USD 148,000 - 240,000; Full time

Benefits offered by this job

Medical benefits

401(k) plan

HSA

FSAs

Bonuses

Tuition reimbursement

Job summary

Caterpillar Inc. is seeking a leader for the AI Engineering team at Cat Digital to guide evaluation and validation of generative AI solutions, intelligent agents, and digital assistants.

You will shape the development of AI-driven products that transform how customers and dealers interact with technology, while establishing scalable engineering practices across hybrid cloud and edge environments. The role requires expertise in GenAI architecture, AI observability, and modern software delivery

Qualifications

• Proven experience leading software quality, test automation, validation, and AI evaluation teams.

Responsibilities

• Providing strong technical direction to ensure the team evaluates advanced AI projects.
• Overseeing performance and training needs of team members and the team.
• Owning the quality of AI engineering products to ensure robustness and high standards.
• Establishing engineering best practices to maintain delivery excellence.

Skills

GenAI architecture

LLMs

Prompt engineering

AI evaluation

CI/CD

Observability

Tools

Azure AI Foundry

SageMaker

Bedrock

LangChain

LangSmith

Job description

Caterpillar Inc. is seeking a leader for the AI Engineering team at Cat Digital to guide evaluation and validation of generative AI solutions, intelligent agents, and digital assistants.

You will shape the development of AI-driven products that transform how customers and dealers interact with technology, while establishing scalable engineering practices across hybrid cloud and edge environments. The role requires expertise in GenAI architecture, AI observability, and modern software delivery