Lead AI Platform Architect for Scalable Pipelines

theinformation-jobs; New York (NY); On-site; USD 155,000 - 175,000; Full time

Benefits offered by this job

Medical, dental, and vision coverage

Fertility care benefits

Parental leave

Generous PTO

401(k) with matching

FSAs

Wellness stipend

Cell phone reimbursement

Office lunches

Commuter benefits

Job summary

The Information is seeking an AI Architect to lead the design, evolution, and scaling of our AI pipeline infrastructure. You will own a backend platform that orchestrates AI workflows at scale and extend it to support a growing set of high-fan-out AI features central to our product strategy.

You will partner with product, editorial, and growth teams, mentor engineers, and drive architectural decisions on storage, queues, and cost controls.

Qualifications

• 5+ years building production applications in a modern web framework with ORM usage, API-only design, and background jobs.
• Experience designing idempotent, retryable, fan-out job pipelines (batching, concurrency controls, dead-letter handling).
• Hands-on work with multiple LLM providers, structured output, embeddings, prompt engineering, and cost tracking.

Responsibilities

• Maintain and operate all existing AI pipelines on the platform.
• Migrate remaining workflows from the legacy orchestration framework to the primary platform.
• Own on-call response for AI pipeline failures, including failed-job triage and retries.
• Manage LLM provider relationships, API key rotation, cost tracking, and model upgrades.
• Design and implement new high-fan-out AI pipelines on the existing platform.

Skills

Backend framework

Job orchestration

LLM integration

Vector search

Production operations

Architecture design

Tools

Ruby on Rails

Python

Job description

The Information is seeking an AI Architect to lead the design, evolution, and scaling of our AI pipeline infrastructure. You will own a backend platform that orchestrates AI workflows at scale and extend it to support a growing set of high-fan-out AI features central to our product strategy.

You will partner with product, editorial, and growth teams, mentor engineers, and drive architectural decisions on storage, queues, and cost controls.