Principal AI Data Intelligence Platform Architect

Yahoo; United States; Hybrid; USD 144,000 - 299,000; Full time

Job summary

Yahoo is seeking a Principal Engineer to lead the architecture and development of AI-powered data intelligence platforms at scale. You will shape technical direction, build reusable platform capabilities, and guide cross-functional teams across engineering, analytics, governance, and business units.

This role combines large-scale distributed data systems, retrieval-augmented generation, and agentic AI to detect anomalies, reason over datasets, and automate complex workflows while upholding

Qualifications

• 10+ years building distributed systems and large-scale data platforms.
• Degree in CS, AI/ML, Engineering, Math, or related field.
• Deep Python expertise and strong software engineering.
• Production experience with batch and streaming processing (Kafka, or equivalent).
• Shipped and operated RAG applications in production; search, and reranking.
• Track record evaluating and validating AI-generated code correctness and operational safety.
• Strong communication and technical leadership across cross-functional stakeholders.

Responsibilities

• Define technical vision and architecture for Yahoo's AI-powered data intelligence platform.
• Architect petabyte-scale batch and streaming systems with Beam, Dataflow, Spark, and Kafka.
• Build production RAG and multi-agent systems: semantic metadata integration, agent orchestration.
• Establish evaluation frameworks — retrieval quality, agent accuracy, hallucination rate, operational safety — with automated benchmarking.
• Use AI development tools to accelerate design, implement the bar for how the org validates AI-generated code and agent plans.
• Drive adoption of RAG and agentic patterns across engineers.
• Partner with product, governance, infrastructure, and exit-form priorities.

Skills

Distributed systems

Python

Technical leadership

Cross-functional collaboration

Education

Bachelor's degree in CS, AI/ML, Engineering, Math, or related field

Tools

Kafka

Beam/Dataflow

Spark

Vertex AI / GCP

BigQuery

Vector Search

Dataflow

LangGraph

CrewAI

AutoGen

Job description

Yahoo is seeking a Principal Engineer to lead the architecture and development of AI-powered data intelligence platforms at scale. You will shape technical direction, build reusable platform capabilities, and guide cross-functional teams across engineering, analytics, governance, and business units.

This role combines large-scale distributed data systems, retrieval-augmented generation, and agentic AI to detect anomalies, reason over datasets, and automate complex workflows while upholding