Senior AI Agent Engineer

FirstIgnite; Vitoria; Presencial; EUR 70.000 - 90.000; Jornada completa

Descripción de la vacante

FirstIgnite is hiring a Senior AI Agent Engineer to build long-running, multi-step agents and the evals measuring their improvements. You will craft tools, wrappers, and evaluations across diverse AI SDKs and APIs, with a strong emphasis on accuracy and traceability.

You will collaborate with product and the full-stack team to turn vague quality goals into measurable datasets, providing clear citations and safe handoffs to humans when needed.

Formación

• 3+ years engineering with LLM or agent systems used by real users.
• Experience evaluating agents, not just models, for multi-step runs.
• Integrated with external APIs and built reliable agent calls.
• Proficient with document pipelines: extraction, normalization, and truth matching.
• Experience with at least one LLM evaluation framework.

Responsabilidades

• Design and ship long-running, multi-step, tool-using agents on various AI SDKs and tooling, including the OpenAI Agents SDK, Anthropic Agent SDK, Vercel AI SDK, LangGraph, MCP, and Temporal Cloud.
• Wrap our APIs and our partners' APIs as tools an agent can call over MCP; translate old systems.
• Get structured data out of long documents and match against existing records; include batch processing.
• Stand up eval suites using frameworks like Promptfoo, Braintrust, LangSmith, DeepEval, LLM-as-judge methods, and custom harnesses to measure tool-use correctness and trajectory quality.
• Every agent produces a draft that a person signs off on; build citations and confidence signals that speed reviews and provide escape paths for the agent.
• Collaborate with product and domain experts to turn vague goals into measurable outcomes, even with tiny or confidential datasets.
• Instrument production traffic, convert real interactions into golden datasets, and run regression tests.
• Compare models and prompts across OpenAI, Anthropic, and open-weight options, assessing latency and quality.
• Bootstrap quality signals for features with no production traffic by generating synthetic documents and test cases.

Conocimientos

LLM/agent systems

Agent evaluation

API integration

Document pipelines

LLM evaluation frameworks

Descripción del empleo

About FirstIgnite

FirstIgnite makes software for university tech transfer offices. Those are the people who take research

coming out of a university lab and get it patented, licensed, or spun out into a company.

The role

We're hiring a Senior AI Agent Engineer. You'll build the agents in our product, and you'll build the

evals that tell us whether each change made them better or worse.

The work is document-heavy rather than chat. The agents run multi-step, call tools, read long and

inconsistently formatted source material, check it against existing records, and produce output that a

person reviews before anything happens with it.

Accuracy matters more here than speed or novelty. Most of the engineering effort goes into precision,

traceability, and getting the agent to hand off to a human at the right moment.

You'll report to the Head of Engineering and work with product and the full-stack team. If you've

shipped agents before, you've probably had the experience of changing a prompt and having no idea

whether you improved anything. That problem is most of this job.

What you'll do

• Design and ship long-running, multi-step, tool-using agents on various AI SDKs and tooling, included but not limited to the OpenAI Agents SDK, the Anthropic Agent SDK, the Vercel AI SDK, LangGraph, MCP, and Temporal Cloud.
• Wrap our APIs and our partners' APIs as tools an agent can call over MCP. Some of those systems are old, single-tenant, and outside our control, so a fair amount of the work is translation.
• Get structured data out of long documents and match it against records that already exist. Expect entity resolution and fuzzy matching, and expect much of it to run in batch.
• Stand up eval suites using various evaluation frameworks and tooling, included but not limited to Promptfoo, Braintrust, LangSmith, DeepEval, LLM-as-judge methods, and custom harnesses. Measure tool-use correctness, trajectory quality, and whether the agent finished the task.
• Every agent here produces a draft that a person signs off on. Build the citations and confidence signals that make that review fast, and give the agent a clear way to escape.
• Sit with product and domain experts and turn vague quality goals into something measurable. Sometimes the only dataset available for that is tiny, or confidential, or both.
• Instrument production traffic, turn real customer interactions into golden datasets, and run them as regression tests.
• Compare models against each other (OpenAI, Anthropic, open-weight), along with prompt strategies and agent designs, and know what each option costs in latency and quality.
• Bootstrap quality signal for features that have no production traffic yet. That usually means generating synthetic documents and test cases, including the ugly edge cases real customers will eventually send us, and knowing where synthetic data stops being a good proxy.
• Write the templates, docs, and tooling the rest of the team needs to run evals without coming to you.

Required Qualifications

• 3+ years of engineering experience, including hands-on work on LLM or agent systems that real users touched.
• You've evaluated agents, not only models, and you know why single-turn accuracy says little about a multi-step run.
• You've integrated against APIs you don't own, including old ones with bad documentation, and turned them into something an agent can call reliably.
• You're comfortable with document pipelines: pulling data out, normalizing it, and checking it against a structured source of truth.
• You've used at least one LLM evaluation framework, in-house tooling included.
• You know how