AI Evaluation Engineering Lead

Caterpillar Financial Services Corporation; Broomfield (CO); On-site; USD 148,000 - 240,000; Full time

Benefits offered by this job

Medical benefits

401(k) plan

PTO and holidays

Life Insurance

Career development

Job summary

Caterpillar Financial Services Corporation is seeking a leader for the AI Engineering team. You will guide a group evaluating and validating advanced generative AI solutions, shaping AI-driven experiences that redefine customer and dealer interactions with technology.

The role emphasizes deep GenAI architecture, ML engineering, and modern software practices, including observability and scalable deployment. Located in the US with full-time responsibilities in a technology-focused team.

Qualifications

• Proven experience leading software quality, test automation, validation, and AI evaluation teams.
• Experience delivering GenAI solutions across hybrid cloud and edge environments.
• Deep knowledge of modern AI observability, telemetry, and evaluation methodologies.

Responsibilities

• Provide strong technical direction to ensure alignment with company goals and AI evaluation objectives.
• Oversee performance of individuals and the team, addressing training and development needs.
• Own the quality of AI engineering products to ensure robustness and high standards.
• Establish and supervise engineering best practices for consistent development.

Skills

Team leadership

GenAI architecture

ML/AI engineering

CI/CD

AI evaluation

Observability

Tools

Azure

AWS

GCP

LangChain

LangSmith

SageMaker

Bedrock

Job description

Caterpillar Financial Services Corporation is seeking a leader for the AI Engineering team. You will guide a group evaluating and validating advanced generative AI solutions, shaping AI-driven experiences that redefine customer and dealer interactions with technology.

The role emphasizes deep GenAI architecture, ML engineering, and modern software practices, including observability and scalable deployment. Located in the US with full-time responsibilities in a technology-focused team.