Senior AI Pipeline Architect — Scale & Observability

The Information; San Francisco (CA); Remote; USD 155,000 - 175,000; Full time

Benefits offered by this job

Medical coverage

Fertility care

Parental leave

PTO

401(k) matching

FSAs

Wellness stipend

Phone reimbursement

Lunch program

Inclusive workplace

Job summary

The Information in San Francisco seeks an AI Architect to lead the design, evolution, and scaling of its AI pipeline infrastructure. You’ll own an API-only backend platform that orchestrates AI workflows at scale and extend it to support high-fan-out features central to our product strategy.

Collaborating with product, editorial, and growth teams, you’ll migrate workflows, manage on-call responses, and mentor engineers on platform architecture, cost tracking, and observability to ensure

Qualifications

• Deep backend framework expertise in modern web frameworks.
• Experience designing idempotent, retryable, fan-out job pipelines.
• Hands-on work with multiple LLM providers and embeddings.
• Experience with vector databases for similarity matching at scale.
• Operations mindset for cloud-based services with observability tooling.
• Ability to author technical design documents and build platforms for others.

Responsibilities

• Maintain and operate all existing AI pipelines on the platform.
• Migrate remaining workflows from legacy framework to primary platform.
• Own on-call response for AI pipeline failures and retries.
• Manage LLM provider relationships, API keys, and cost tracking.
• Design and implement new high-fan-out AI pipelines.
• Collaborate with product and editorial teams to translate requirements into designs.
• Mentor engineers on pipeline patterns and platform architecture.

Skills

Backend framework

ORM usage

API-only design

Background jobs

LLM integration

Vector search

Observability

Cost tracking

Architecture design

Proposal development

Tools

Rails

Django

Ruby on Rails

Job description

The Information in San Francisco seeks an AI Architect to lead the design, evolution, and scaling of its AI pipeline infrastructure. You’ll own an API-only backend platform that orchestrates AI workflows at scale and extend it to support high-fan-out features central to our product strategy.

Collaborating with product, editorial, and growth teams, you’ll migrate workflows, manage on-call responses, and mentor engineers on platform architecture, cost tracking, and observability to ensure