Principal Engineer (AI Production Readiness)

Workday; Toronto; Hybrid; CAD 150,000 - 230,000; Full time

Job summary

Workday’s Architecture & AI (AAI) organization is seeking a Principal AI Engineer to own the technical path for internal AI agents from prototype to production on Databricks and AWS.

You will design the architecture, build reference implementations, and run readiness reviews with security, SRE, and data teams while partnering with business stakeholders; this is a hybrid Toronto-based role with strong Python, IaC, and CI/CD expertise.

Qualifications

• 8+ years of Data & AI application development in Python, IaC and CI/CD environments.
• 8+ years modern data stack experience with Databricks, AWS, Snowflake, GCP.
• 6+ years of professional experience with LLM applications in cloud stacks.
• Excellent verbal and written communication skills.
• Ability to translate platform constraints to business decisions.
• Willing to report up to Toronto office 50%.

Responsibilities

• Define and enforce production-readiness standards for AI apps.
• Own architecture patterns across Databricks and AWS.
• Build prototypes and re-architect AI-generated apps into services with production state.
• Diagnose failures and drive blockers with cross-functional partners.
• Run design and readiness reviews with platform, data, security and SRE teams.
• Engage with business teams to translate needs into platform designs.

Skills

Data & AI Dev

Communication skills

LLM applications

Business collaboration

Tools

Databricks

AWS

Snowflake

GCP

Python

CI/CD

IaC

Job description

The Architecture & AI (AAI) organization is a dynamic and evolving team that is spearheading Workday’s growth through trusted data excellence, innovation, and architectural thought leadership. Equipped with an array of skills in data & AI platforms and architecture, data science, engineering, machine learning and AI strategy & product management, this team orchestrates the flow of data across our growing company while ensuring data accessibility, accuracy, and security With a relentless focus on innovation and efficiency, Workmates in AAI enable the transformation of complex data sets into actionable insights that fuel strategic decisions and position Workday at the forefront of the technology industry We are seeking a Principal AI Engineer to own the technical path that carries internally built AI agents from prototype to enterprise-grade production on Workday’s Databricks and AWS estate. This is a hands-on engineering role at the intersection of AI applications, data, identity and platform You will design the architecture and work in code to build the reference implementations and evaluation harnesses that business-built applications are re-architected onto, and hold the technical bar as those applications move into production

• Production readiness standards and automated enforcement: Define and enforce what production-ready means for an AI application at Workday — exit criteria against BT’s existing security, code, testing and documentation gates, plus the evaluation sets, graders and regression suites that run in the deployment pipeline to prove those criteria are met, drawing on representative data, production signals and human review
• Architecture and paved-road assets: Own the architecture patterns for AI applications across the Databricks and AWS stacks, and build and maintain the reference implementations and deployment templates for each recurring pattern, so a team building a known pattern inherits correct identity, data access and logging posture by construction
• Hands-on engineering: Work in code with the current stack; AWS Bedrock and AgentCore, Databricks Agent Bricks and Unity Catalog — to build prototypes, tool and system integrations, and to re-architect AI-generated applications into services that can hold a production state
• Technical diagnosis and resolution: Diagnose complex implementation challenges in non-deterministic systems, reproduce failures, bisect regressions across model version, prompt, retrieval index and tool schema, trace latency and token cost across multi-step agent runs, test hypotheses, and drive blockers to resolution with platform, data, security and SRE owners
• Design review and transition to operations: Run design and readiness reviews with platform, data, security and SRE partners, and own the handoff so no application enters production without an operator, a patch path and a rollback
• Business engagement: Work with the business teams building these applications to translate what they need into platform and data design, and translate platform constraints back into decisions they can act on, including the decision not to proceed
• Cross-functional technical leadership: Be the technical counterpart business owners consult before building, and carry direction across partner functions on influence rather than mandate

• 8+ years of experience in Data & AI application development in a Python, IaC and CI/CD; production environment
• 8+ years of modern data stack experience, ie; Databricks, AWS, Snowflake, GCP, or comparable technologies
• 6+ years of professional experience with LLM applications in a major cloud AI-native stack
• Excellent verbal and written communication skills
• Work with business counterparts and carry direction across partner functions on influence
• Open to 50% reporting to our Toronto office
• Ability to work in alignment with business teams to build applications and translate platform constraints back into decisions