Principal AI Engineer

Scotiabank; Dallas (TX); On-site; USD 150,000 - 210,000; Full time

Job summary

Scotiabank seeks a Principal Engineer to lead the discovery, architecture, and delivery of AI and agentic solutions across Canadian Banking, International Banking, GBM, GWM, Fraud and AML. This role spans data engineering, AI engineering, software engineering, and MLOps.

You will partner with business leaders to identify opportunities, translate workflows into practical solutions, and guide multidisciplinary teams from early experimentation through production deployment with a focus on security,

Qualifications

• Extensive software, data, or ML engineering leadership across large programs.
• Proven delivery of AI/ML solutions from discovery to production.
• Strong Python, SQL, API design, and distributed systems skills.
• Experience with Databricks, Spark, Delta Lake, and Azure.

Responsibilities

• Lead discovery with stakeholders to define AI use cases.
• Own end-to-end technical architecture across data, AI, and deployment.
• Guide teams through architectural trade-offs and reviews.
• Design agentic workflows and retrieval-augmented generation patterns.
• Establish AI governance, security, and responsible AI practices.

Skills

Python

SQL

API design

Distributed systems

Databricks

Apache Spark

Delta Lake

Azure

Docker

Kubernetes

MLOps

Leadership

Education

Bachelor’s or Master’s in CS/Eng/Math

Tools

Databricks

Apache Spark

Delta Lake

Azure

Docker

Kubernetes

MLflow

Job description

Select how often (in days) to receive an alert:

Please note that the Salary Range shown is a guideline only. Salary offered may vary based on factors, including, but not limited to, the successful candidate’s relevant knowledge, skills, and experience.

Join a purpose driven winning team, committed to results, in an inclusive and high-performing culture.

Purpose

We are seeking a Principal Engineer to lead the discovery, architecture, and delivery of AI and agentic solutions across Canadian Banking, International Banking, Global Banking and Markets (GBM), Global Wealth Management (GWM), Fraud, and Anti-Money Laundering (AML).

This is a hands-on technical leadership role spanning data engineering, AI engineering, software engineering, and MLOps. You will partner with business leaders to identify opportunities, translate complex workflows into practical solutions, and guide multidisciplinary teams from early experimentation through production deployment and ongoing operation.

The role requires deep engineering judgment: determining when to use traditional automation, machine learning, generative AI, or agentic workflows, and delivering solutions that meet enterprise expectations for security, reliability, explainability, and measurable business value.

What You'll Do

Business partnership and use case discovery

• Partner with line-of-business stakeholders, product owners, and domain experts to understand business processes, operational challenges, and opportunities for AI-driven improvement.
• Lead discovery workshops to identify and prioritize AI and agentic use cases based on business value, technical feasibility, data readiness, risk, and delivery effort.
• Translate opportunities into solution designs, delivery roadmaps, technical backlogs, and measurable acceptance criteria.
• Assess where autonomous execution is appropriate and where deterministic controls, human review, or approval are required.

Architecture and technical leadership

• Own end-to-end technical architecture across data ingestion, processing, model development, retrieval, agent orchestration, application integration, and production operations.
• Design agentic workflows incorporating tool use, enterprise APIs, retrieval-augmented generation, state management, and human oversight.
• Establish reusable engineering patterns, reference architectures, and shared components that accelerate delivery across business lines.
• Lead architecture and code reviews, resolve complex engineering challenges, and guide teams through technical trade-offs.
• Build critical components and reference implementations while mentoring engineers and influencing technical decisions across teams.

Data and AI Engineering

• Design scalable batch and streaming data pipelines using Databricks, Apache Spark, Delta Lake, SQL, Python, and Azure-native services.
• Establish data quality, lineage, metadata, access controls, and curated data products that support analytics, machine learning, and AI applications.
• Develop and integrate machine learning models, LLM-powered applications, retrieval services, and agentic workflows into enterprise systems.
• Define evaluation approaches for predictive performance, retrieval quality, response accuracy, tool execution, and end-to-end task completion.

MLOps DevOps and Production Delivery

• Establish lifecycle practices for experiments, models, prompts, agents, and evaluation datasets using MLflow and supporting engineering tools.
• Implement automated testing, CI/CD, infrastructure as code, deployment controls, rollback strategies, and environment management.
• Design and deploy containerized services using Docker and Kubernetes, including Azure Kubernetes Service (AKS).
• Define production monitoring for data and model drift, AI quality, latency, availability, cost, and operational failures.
• Partner with platform engineering, cybersecurity, and operations teams to deliver resilient services with clear ownership, support models, and service-level objectives.

Responsible AI and Enterprise Governance

• Embed privacy, security, auditability, explainability, and responsible AI requirements into solution design and delivery.
• Partner with model risk management, compliance, data governance, and cybersecurity teams throughout the delivery lifecycle.
• Implement appropriate controls for sensitive data access, prompt injection, unsafe tool execution, and unintended agent actions.
• Ensure production solutions maintain traceable decisions, evaluation evidence, and appropriate human accountability.

What You'll Bring

• Extensive experience in software, data, or machine learning engineering, with demonstrated technical leadership across complex enterprise initiatives.
• Proven ability to take AI or machine learning solutions from business discovery through production deployment and operation.
• Strong hands-on proficiency in Python, SQL, API design, distributed systems, and modern software engineering practices.
• Deep experience with Databricks, Apache Spark, Delta Lake, and enterprise data pipeline architecture.
• Practical experience building generative AI and agentic applications, including retrieval, tool integration, orchestration, and evaluation.
• Experience with MLflow and MLOps practices, including experiment tracking, model registration, deployment, monitoring, and lifecycle governance.
• Strong knowledge of Azure architecture, identity, networking, security, and cloud-native application delivery.
• Experience with Docker, Kubernetes, CI/CD pipelines, automated testing, and infrastructure as code.
• Ability to communicate technical trade-offs clearly to executives, business stakeholders, architects, and engineering teams.
• Demonstrated ability to lead through influence and raise engineering standards across organizational boundaries.

Preferred Qualifications

• Experience in banking or financial services, particularly banking operations, capital markets, wealth management, fraud detection, or AML.
• Experience delivering AI solutions within regulated environments and collaborating with model risk and governance functions.
• Familiarity with Unity Catalog, enterprise metadata platforms, semantic models, vector search, and knowledge graphs.
• Experience with event-driven architectures, secure agent tool interfaces, and Model Context Protocol (MCP).
• Bachelor's or master's degree in computer science, engineering, mathematics, or a related discipline, or equivalent practical experience.

Technology Environment

Data engineering

• Azure Databricks, Apache Spark, Delta Lake, Unity Catalog, Azure Data Lake Storage, Azure Data Factory, Azure Event Hubs.

AI and machine learning

• Azure-hosted AI models and services, MLflow, machine learning frameworks, agent orchestration frameworks, retrieval and vector search.

Application engineering

• Python, SQL, REST APIs, microservices, and event-driven integration.

Cloud and runtime

• Microsoft Azure, Docker, Kubernetes, AKS, Azure Functions, and Azure API Management.

Engineering operations

• Azure DevOps or GitHub Actions, Terraform or Bicep, Azure Monitor, Application Insights, Key Vault, and Microsoft Entra ID.

Scotiabank wants you to be able to bring your best self to work – and life, every day. With a focus on holistic well-being, our many flexible benefit programs are designed to help support your unique family, financial, physical, mental, and social health needs.

Scotiabank is a leading bank in the Americas. Guided by our purpose: "for every future", we help our customers, their families and their communities achieve success through a broad range of advice, products and services, including personal and commercial banking, wealth management and private banking, corporate and investment banking, and capital markets.

At Scotiabank, every employee is empowered to reach their fullest potential, respected for who they are and, embraced for their differences. That’s why we work to grow and diversify talent and engage employees in a performance-oriented culture.

At Scotiabank, we value the unique skills and experiences each individual brings to the Bank, and are committed to creating and maintaining an inclusive and accessible environment for everyone. If you require accommodation (including, but not limited to, an accessible interview site, alternate format documents, ASL Interpreter, or Assistive Technology) during the recruitment and selection process, please let our Recruitment team know.

Scotiabank is an equal opportunity employer. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or any other characteristic protected by federal, state, or local law.