GenAI Evaluation Engineering Lead

Caterpillar Brazil; Chicago, Northern (IL, KY); Hybrid; USD 148,000 - 240,000; Full time

Job summary

Caterpillar Inc. is seeking an AI Engineering lead to guide the Cat Digital AI Engineering team. You will evaluate and validate advanced generative AI solutions, including intelligent agents and digital assistants, shaping AI-driven experiences for customers and dealers.

Leading with a focus on software quality, scalable processes, and modern deployment strategies, you will influence engineering standards and drive innovation across hybrid cloud and embedded environments.

Qualifications

• Proven experience leading software quality, test automation, validation, and AI evaluation teams.
• Experience delivering enterprise-scale GenAI solutions across hybrid cloud and embedded/edge environments.
• Deep expertise in AI evaluation methodologies, observability, tracing, telemetry, and data collection.
• Hands-on experience with modern AI toolchains and AI development ecosystems.

Responsibilities

• Providing strong technical support and clear direction to ensure the team is aligned with company goals and capable of evaluating advanced AI projects.
• Overseeing the performance of both individual team members and the team as a whole, fostering a culture of learning by identifying and addressing training and development needs.
• Taking ownership of the quality of AI engineering products, making sure all solutions are robust, reliable, and meet high standards.
• Establishing and supervising the implementation of engineering best practices to maintain consistency and excellence in development processes.

Skills

GenAI architecture

LLMs & multimodal

AI evaluation

CI/CD

Canary deployments

Prompt engineering

RAG architectures

Observability

Tools

Azure

AWS

GCP

LangChain

LangSmith

Bedrock

SageMaker

Arize

Job description

Caterpillar Inc. is seeking an AI Engineering lead to guide the Cat Digital AI Engineering team. You will evaluate and validate advanced generative AI solutions, including intelligent agents and digital assistants, shaping AI-driven experiences for customers and dealers.

Leading with a focus on software quality, scalable processes, and modern deployment strategies, you will influence engineering standards and drive innovation across hybrid cloud and embedded environments.