Principal AI Data Platform Architect

ouryahoo; United States; Hybrid; USD 144,000 - 299,000; Full time

Benefits offered by this job

Healthcare

401(k)

Backup childcare

Education stipends

Job summary

Yahoo is seeking a Principal Engineer (IC5) to lead architecture and development of AI-powered data intelligence platforms at scale. You will shape technical direction, design petabyte-scale batch/streaming systems, and build RAG and multi-agent capabilities across engineering, governance, and analytics teams.

The role emphasizes correctness, reliability, security, and operational safety as you accelerate AI-enabled software development and drive adoption of advanced AI patterns in a hybrid work

Qualifications

• 10+ years building distributed systems and large-scale data platforms.
• Degree in CS, AI/ML, Engineering, Math, or related field.
• Deep Python expertise.
• Production experience with batch and streaming processing (Kafka, or equivalent).
• Shipped and operated RAG applications in production; search and reranking.
• Track record evaluating and validating AI-generated code correctness and operational safety.

Responsibilities

• Define technical vision and architecture for Yahoo's AI-powered data intelligence platform.
• Architect petabyte-scale batch and streaming systems with Beam, Dataflow, Spark, and Kafka.
• Build production RAG and multi-agent systems: semantic metadata integration, agent orchestration.
• Establish evaluation frameworks — retrieval quality, agent accuracy, hallucination rate, operational safety.
• Use AI development tools to accelerate design, implement the bar for how the org validates AI-generated code and agent plans.
• Drive adoption of RAG and agentic patterns across engineers.
• Partner with product, governance, infrastructure, and priority teams.

Skills

Distributed systems

Python

RAG

AI/ML concepts

Communication

Leadership

Education

Bachelor's degree in CS/AI/Engineering/Math

Tools

Kafka

Beam/Dataflow

Spark

Dataflow

Job description

Yahoo is seeking a Principal Engineer (IC5) to lead architecture and development of AI-powered data intelligence platforms at scale. You will shape technical direction, design petabyte-scale batch/streaming systems, and build RAG and multi-agent capabilities across engineering, governance, and analytics teams.

The role emphasizes correctness, reliability, security, and operational safety as you accelerate AI-enabled software development and drive adoption of advanced AI patterns in a hybrid work