Senior AI Engineer, Agent Platform & Evaluation Harness

Government Employees Insurance Company; New York (NY); On-site; USD 115,000 - 260,000; Full time

Job summary

GEICO is seeking an experienced Sr. Staff AI Engineer for its AI Agent Platform to shape agent design, context strategies, and evaluation methodology at scale.

You will own the intelligence layer across enterprise workflows, partnering with backend engineers to ensure reliability and performance in production environments. The role emphasizes experimentation, data-backed decisions, and rapid prototyping of GenAI capabilities used by thousands of associates and millions of customers.

Qualifications

• 10+ years of experience in software engineering, ML engineering, or applied science, with strong Python proficiency.
• Tech-lead experience: owned the design and delivery of significant systems involving multiple engineers or teams.
• Significant hands-on experience building GenAI applications and agentic systems with LLMs such as GPT, Claude, Llama, or Qwen, taken beyond prototype into real use.
• Demonstrated experience designing and running experiments to improve LLM or agent performance, including defining metrics, building evaluation sets, analyzing results, and iterating.
• Deep practical understanding of prompt and context engineering, tool calling, RAG, and agent architectures, including hands-on experience with MCP.

Responsibilities

• Design the core agent loop, including planning, reasoning, tool selection and calling, memory, and error recovery, as reusable patterns.
• Develop context engineering strategies: prompt architecture, context window management, summarization and compaction, memory, and retrieval (RAG) integration.
• Define how agents connect to tools and enterprise systems through the Model Context Protocol (MCP).
• Design how agents discover, load, and apply skills (packaged instructions, scripts, and resources for specific tasks).
• Explore sub-agent and multi-agent patterns, and decide with evidence what belongs in the platform.
• Evaluate new frontier and open-weight models as they are released, and understand their tradeoffs in capability, latency, and cost for platform use.
• Design evaluation methodology: task-level metrics, rubrics, golden datasets, LLM-as-judge graders, and human review, with automated graders calibrated against human judgment.
• Build the eval harness that lets any team define, run, and compare evaluations for their agents, spanning offline benchmarks, simulated users and environments, and online quality monitoring.

Skills

Python

ML engineering

Software engineering

Technical leadership

GenAI platforms

Tools

MCP

LangGraph

LangSmith

Inspect AI

Braintrust

Job description

GEICO is seeking an experienced Sr. Staff AI Engineer for its AI Agent Platform to shape agent design, context strategies, and evaluation methodology at scale.

You will own the intelligence layer across enterprise workflows, partnering with backend engineers to ensure reliability and performance in production environments. The role emphasizes experimentation, data-backed decisions, and rapid prototyping of GenAI capabilities used by thousands of associates and millions of customers.