Senior AI Engineer, Agent Platform & Evaluation Harness

GEICO

New York (NY)

Presencial

USD 115.000 - 260.000

Jornada completa

14 días+

Consigue una respuesta de este empleador

Tu currículum y tu carta de presentación, adaptados exactamente a lo que buscan contratar.

Supera los filtros ATS

Descripción de la vacante

GEICO is seeking an experienced Sr. Staff AI Engineer for its AI Agent Platform to shape agent design, context strategies, and evaluation methodology at scale.

You will own the intelligence layer across enterprise workflows, partnering with backend engineers to ensure reliability and performance in production environments. The role emphasizes experimentation, data-backed decisions, and rapid prototyping of GenAI capabilities used by thousands of associates and millions of customers.

Formación

  • 10+ years of experience in software engineering, ML engineering, or applied science, with strong Python proficiency.
  • Tech-lead experience: owned the design and delivery of significant systems involving multiple engineers or teams.
  • Significant hands-on experience building GenAI applications and agentic systems with LLMs such as GPT, Claude, Llama, or Qwen, taken beyond prototype into real use.
  • Demonstrated experience designing and running experiments to improve LLM or agent performance, including defining metrics, building evaluation sets, analyzing results, and iterating.
  • Deep practical understanding of prompt and context engineering, tool calling, RAG, and agent architectures, including hands-on experience with MCP.

Responsabilidades

  • Design the core agent loop, including planning, reasoning, tool selection and calling, memory, and error recovery, as reusable patterns.
  • Develop context engineering strategies: prompt architecture, context window management, summarization and compaction, memory, and retrieval (RAG) integration.
  • Define how agents connect to tools and enterprise systems through the Model Context Protocol (MCP).
  • Design how agents discover, load, and apply skills (packaged instructions, scripts, and resources for specific tasks).
  • Explore sub-agent and multi-agent patterns, and decide with evidence what belongs in the platform.
  • Evaluate new frontier and open-weight models as they are released, and understand their tradeoffs in capability, latency, and cost for platform use.
  • Design evaluation methodology: task-level metrics, rubrics, golden datasets, LLM-as-judge graders, and human review, with automated graders calibrated against human judgment.
  • Build the eval harness that lets any team define, run, and compare evaluations for their agents, spanning offline benchmarks, simulated users and environments, and online quality monitoring.

Conocimientos

Python
ML engineering
Software engineering
Technical leadership
GenAI platforms

Herramientas

MCP
LangGraph
LangSmith
Inspect AI
Braintrust

Descripción del empleo

GEICO is seeking an experienced Sr. Staff AI Engineer for its AI Agent Platform to shape agent design, context strategies, and evaluation methodology at scale.

You will own the intelligence layer across enterprise workflows, partnering with backend engineers to ensure reliability and performance in production environments. The role emphasizes experimentation, data-backed decisions, and rapid prototyping of GenAI capabilities used by thousands of associates and millions of customers.

Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Senior Platform Engineer – AI Agent Platform
Senior Platform Engineer – AI Agent Platform

GEICO • New York (NY)

Presencial
USD 130.000 - 300.000
Senior AI Platform Engineer, Agent Harness
Senior AI Platform Engineer, Agent Harness

GEICO • Seattle (WA)

Presencial
USD 105.000 - 230.000
Senior Staff Backend Engineer AI Agent Platform
Senior Staff Backend Engineer AI Agent Platform

GEICO Norwalk • New York (NY)

Presencial
USD 130.000 - 300.000
Lead AI Platform Engineer — Agent Harness & RAG
Lead AI Platform Engineer — Agent Harness & RAG

GEICO • California (MD)

Híbrido
USD 130.000 - 300.000
Senior AI Platform Engineer - Agent Harness & RAG
Senior AI Platform Engineer - Agent Harness & RAG

GEICO • Maryland

Híbrido
USD 105.000 - 230.000
Senior AI Platform Architect — GenAI & Agent Workflows
Senior AI Platform Architect — GenAI & Agent Workflows

GEICO Norwalk • Palo Alto (CA)

Presencial
USD 210.000 - 350.000
Senior GenAI Platform Architect and Technical Leader
Senior GenAI Platform Architect and Technical Leader

GEICO • Palo Alto (CA)

Presencial
USD 150.000 - 300.000
Senior Staff AI Engineer: GenAI Platform Architect
Senior Staff AI Engineer: GenAI Platform Architect

GEICO Norwalk • Palo Alto (CA)

Presencial
USD 150.000 - 300.000
Senior AI Platform Engineer - Durable Agent Workflows
Senior AI Platform Engineer - Durable Agent Workflows

GEICO • Palo Alto (CA)

Presencial
USD 105.000 - 230.000
Competitive pay
Sponsorship potential
Senior AI Platform Engineer — Agent Harness & RAG
Senior AI Platform Engineer — Agent Harness & RAG

GEICO • New York (NY)

Presencial
USD 120.000 - 230.000