Senior Engineer, AI and Agents Remote / Hybrid (flexible)

Falcon Labs; United States; Hybrid; USD 150,000 - 210,000; Full time

Job summary

Falcon Labs seeks a senior engineer to build and run LLM agents, ensuring reliability and measurable improvement through experiments. You will own the agent platform, from runtimes to deployment and CI, and collaborate with AI Enablement to automate real tasks.

You’ll work with tracing, observability, and regression tests, write SQL to prove impact, and expand backend work across TypeScript or Python on AWS. This is a hands-on senior role shaping production AI.

Qualifications

• Must have built and run LLM agents in production with tool use, multi-step workflows, context management, cost and latency budgets.

Responsibilities

• Build and run the platform our agents live on: isolated runtimes, access control, cost limits, CI deployment.
• Instrument every agent with tracing and replay capability across prompts, tool calls, state and results.
• Analyze trajectories to identify errors and improve prompts, context, tools, models, or data.
• Create evals with gold datasets, step-level checks and regression suites that run in CI.
• Translate failures and feedback into new eval cases and monitor quality, latency and cost over time.

Skills

LLM agents

agent evaluation

tracing/observability

backend engineering

SQL writing

Tools

OpenTelemetry

Langfuse

LangSmith

Braintrust

Arize

Job description

Falcon Labs turns the post-transaction experience into an incremental revenue stream for online services and a rewarding moment for their customers. Our solution uses advanced technology and AI to deliver personalized, contextually relevant offers to consumers at their peak engagement moment, right after they complete a transaction.

The moment after a transaction is one of the most underutilized surfaces in commerce: intent is high, trust is established, and attention is real, yet for most brands it still ends with a static confirmation screen. We operate across major e-commerce ecosystems (Shopify, Wix, and a growing set of technology partnerships), earn a take rate on advertiser conversions, and run on real-time decisioning, Snowflake analytics, and an expanding push into agentic automation across the network. We are a team of ad-tech veterans from the mobile advertising world (ironSource/Unity).

The position

Full time. Senior individual contributor, reporting to the VP R&D and working closely with our Director of AI Enablement.

About the role

Agents already do real work across our company, and we keep adding more. We need a senior engineer who can build agents, run them reliably, and tell from the evidence whether a change made them better or worse. This is a senior engineering role first. Agents are where you start, and you pick up backend and platform work when the team needs it.

What you’ll do

Build and run the platform our agents live on: isolated runtimes, access control, cost limits, and the deployment and CI around them.

Instrument every agent with tracing, so any run can be replayed step by step, from prompts and tool calls to intermediate state and the final result.

Read trajectories to find where an agent went wrong, whether in the prompt, the context, a tool, the model or the data, and fix it.

Build evals that gate changes: golden datasets, step-level and end-to-end checks, LLM-as-judge graders calibrated against human review, and regression suites that run in CI.

Turn production failures and user feedback into new eval cases, and track quality, latency and cost per agent over time.

Work with our Director of AI Enablement and the teams who use the agents to turn manual work into automation people trust.

Later on

Customer-facing AI, where the bar for reliability, latency and cost is higher. Backend projects when the team needs another senior engineer.

Requirements

Must have:

LLM agents you built and ran in production: tool use, multi-step workflows, context management, cost and latency budgets.

Hands-on agent evaluation, including trajectory and task-success evals, LLM-as-judge and regression testing. You can explain how you know an eval measures the right thing.

Tracing and observability for agents, with OpenTelemetry or tools like Langfuse, LangSmith, Braintrust or Arize. You have debugged a failure from a trace.

Strong backend engineering in TypeScript or Python, and hands-on AWS: containers, IAM, secrets, infrastructure as code.

Comfort with data. You can write the SQL that shows whether an agent actually helped.

Nice to have:

Go, Terraform, MCP servers, LLM gateways and model routing, agent sandboxing, comparing models on accuracy, cost and latency, adtech or marketplace experience.

How we work with AI

AI is part of how we build every day. Coding agents write most of our code, and agents already handle real parts of our engineering and operations work. When a task comes in, we first ask how to automate it, where an agent fits in the loop, and how to get it to production through agents. We’re looking for people who already work this way. In this role, that is the job itself.