AI Engineer

Scotiabank; Dallas (TX); On-site; USD 150,000 - 230,000; Full time

Job summary

Scotiabank is seeking a Senior AI Engineer to design, build, and operate AI and agentic solutions across Canadian Banking, International Banking, GBM, GWM, Fraud, and AML. You will work with stakeholders, data scientists, and platform teams to translate workflows into secure, production-grade AI services using Databricks, MLflow, Kubernetes, and the Azure stack.

The role involves building AI applications, data pipelines, and production-ready ML workflows, with a focus on governance, safety, and

Qualifications

• Strong professional experience in software, data, or machine learning engineering, including hands-on delivery of production AI or machine learning solutions.
• Proficiency in Python and SQL, API development, automated testing, Git, and modern software engineering practices.
• Hands-on experience building generative AI applications or agentic workflows, including retrieval, tool integration, orchestration, and evaluation.
• Experience with Databricks, Apache Spark, Delta Lake, and scalable data processing pipelines.
• Experience with MLflow and MLOps practices such as experiment tracking, model registration, versioning, deployment, and monitoring.
• Practical experience with Microsoft Azure services and cloud application security, identity, networking, and observability.
• Experience with Docker, Kubernetes, CI/CD pipelines, and infrastructure as code, with the ability to deploy and troubleshoot services.
• Ability to independently deliver well-defined engineering work, investigate ambiguous problems, and collaborate across business and technical teams.
• Understanding of AI evaluation, data privacy, secure tool use, human oversight, and the limitations of probabilistic model outputs.

Responsibilities

• Design, build, and operate AI and agentic solutions across banking domains.
• Participate in discovery sessions to understand workflows and AI-enabled improvement opportunities.
• Translate requirements into technical designs, tasks, estimates, and acceptance criteria.
• Assess data readiness, dependencies, and feasibility; build prototypes to validate assumptions.
• Build LLM-powered applications and agentic workflows with tool calling and orchestration.
• Develop batch and streaming data pipelines using Databricks, Spark, Delta Lake, Python, and SQL.
• Implement CI/CD pipelines, automated testing, and deployment with Docker, Kubernetes, and AKS.

Skills

Python

SQL

API development

Git

MLOps

Docker

Kubernetes

Education

Bachelor’s or Master’s degree in computer science, engineering, or related field

Tools

Databricks

Azure Kubernetes Service (AKS)

MLflow

Job description

Select how often (in days) to receive an alert:

Please note that the Salary Range shown is a guideline only. Salary offered may vary based on factors, including, but not limited to, the successful candidate’s relevant knowledge, skills, and experience.

Join a purpose driven winning team, committed to results, in an inclusive and high-performing culture.

Purpose:

We are seeking a Senior AI Engineer to design, build, and operate AI and agentic solutions supporting Canadian Banking, International Banking, Global Banking and Markets (GBM), Global Wealth Management (GWM), Fraud, and Anti-Money Laundering (AML). This is a hands-on engineering role spanning AI applications, data pipelines, software services, and MLOps. You will work with business stakeholders, product owners, data scientists, and platform teams to translate business workflows into secure, reliable production solutions using Databricks, MLflow, Kubernetes, and the Azure-native stack. You will own the design and delivery of assigned components and workflows, contribute to technical decisions with Principal Engineers and architects, and support solutions throughout their production lifecycle. The role requires sound judgment in choosing between deterministic automation, machine learning, generative AI, and agentic approaches.

What You'll Do

Business Discovery and Solution Design

• Participate in discovery sessions with business stakeholders to understand workflows, data sources, pain points, and opportunities for AI-enabled improvement.
• Translate requirements into technical designs, implementation tasks, estimates, and testable acceptance criteria in partnership with product owners and technical leads.
• Assess data readiness, integration dependencies, and technical feasibility; build focused prototypes to validate assumptions before production implementation.
• Identify appropriate boundaries for agent autonomy, human review, approvals, and deterministic business rules.

AI and Agentic Application Engineering

• Build LLM-powered applications and agentic workflows using tool calling, enterprise APIs, retrieval-augmented generation, structured outputs, and state management.
• Implement document ingestion, parsing, chunking, embeddings, indexing, retrieval, and reranking to provide relevant, permission-aware context.
• Integrate predictive models and business rules with AI workflows in collaboration with data scientists and domain experts.
• Implement validation, timeouts, retries, idempotency, and fallback paths to make workflows dependable and prevent unintended repeated actions.
• Compare model and orchestration options against task quality, latency, cost, and operational requirements.

Data and Software Engineering

• Develop and maintain batch and streaming data pipelines using Azure Databricks, Apache Spark, Delta Lake, Python, and SQL.
• Implement data quality checks, schema validation, lineage, and access controls using enterprise standards and Unity Catalog where applicable.
• Build tested APIs and services that connect AI applications to enterprise data, business systems, and approved tools.
• Write maintainable code with clear interfaces, unit and integration tests, documentation, and peer review; troubleshoot complex issues across application and data layers.

Evaluation MLOps and Production Delivery

• Use MLflow and supporting tools to track experiments, register models, version prompts and configurations, and retain evaluation results and deployment evidence.
• Build representative evaluation datasets and automated checks for model performance, retrieval relevance, grounded responses, tool execution, and end-to-end task completion.
• Implement CI/CD pipelines, automated regression testing, environment configuration, and controlled releases with rollback support.
• Containerize services with Docker and deploy to Kubernetes, including Azure Kubernetes Service (AKS), using approved infrastructure-as-code patterns.
• Instrument applications for tracing, logs, and metrics; monitor AI quality, data and model drift where applicable, latency, availability, and consumption costs.
• Investigate production incidents, perform root-cause analysis, maintain runbooks, and contribute to operational support under the team’s agreed support model.

Security and responsible AI

• Implement least-privilege access, secure secrets management, data protection, and audit logging using enterprise security and Azure identity standards.
• Apply safeguards for prompt injection, sensitive data disclosure, unsafe tool execution, and unauthorized agent actions.
• Implement human approval and escalation steps for sensitive decisions or actions, preserving traceability of inputs, outputs, and tool activity.
• Work with cybersecurity, model risk, compliance, and data governance partners to address required controls and remediation items.

Collaboration and Engineering Contribution

• Work with Principal Engineers and architects to align component designs with enterprise architecture and reusable platform capabilities.
• Collaborate with data engineers, data scientists, platform engineers, and business teams to resolve dependencies and deliver integrated solutions.
• Mentor engineers through code reviews, pairing, and knowledge sharing; contribute reusable components and practical engineering guidance.
• Communicate progress, blockers, technical trade-offs, and delivery risks clearly and early.

What You’ll Bring

Required Qualifications

• Strong professional experience in software, data, or machine learning engineering, including hands‑on delivery of production AI or machine learning solutions.
• Proficiency in Python and SQL, API development, automated testing, Git, and modern software engineering practices.
• Hands‑on experience building generative AI applications or agentic workflows, including retrieval, tool integration, orchestration, and evaluation.
• Experience with Databricks, Apache Spark, Delta Lake, and scalable data processing pipelines.
• Experience with MLflow and MLOps practices such as experiment tracking, model registration, versioning, deployment, and monitoring.
• Practical experience with Microsoft Azure services and cloud application security, identity, networking, and observability.
• Experience with Docker, Kubernetes, CI/CD pipelines, and infrastructure as code, with the ability to deploy and troubleshoot services.
• Ability to independently deliver well‑defined engineering work, investigate ambiguous problems, and collaborate across business and technical teams.
• Understanding of AI evaluation, data privacy, secure tool use, human oversight, and the limitations of probabilistic model outputs.

Preferred Qualifications

• Experience in financial services, particularly banking, capital markets, wealth management, fraud detection, or AML.
• Experience implementing retrieval and vector search, document processing, semantic models, or knowledge graph integrations.
• Familiarity with Model Context Protocol (MCP), event-driven architectures, and agent orchestration frameworks.
• Experience working in regulated environments with model risk, audit, and data governance requirements.
• Bachelor’s or master’s degree in computer science, engineering, mathematics, or a related discipline, or equivalent practical experience.

Technology Environment

• Data engineering: Azure Databricks, Apache Spark, Delta Lake, Unity Catalog, Azure Data Lake Storage, Azure Data Factory, and Azure Event Hubs.
• AI and machine learning: Azure-hosted AI models and services, MLflow, machine learning frameworks, agent orchestration frameworks, retrieval, and vector search.
• Software and cloud: Python, SQL, REST APIs, Microsoft Azure, Docker, Kubernetes, AKS, Azure Functions, and Azure API Management.
• Engineering operations: Azure DevOps or GitHub Actions, Terraform or Bicep, Azure Monitor, Application Insights, Key Vault, and Microsoft Entra ID.

What's in it for you?

Scotiabank wants you to be able to bring your best self to work – and life, every day. With a focus on holistic well-being, our many flexible benefit programs are designed to help support your unique family, financial, physical, mental, and social health needs.
#Dallas

Scotiabank is a leading bank in the Americas. Guided by our purpose: "for every future", we help our customers, their families and their communities achieve success through a broad range of advice, products and services, including personal and commercial banking, wealth management and private banking, corporate and investment banking, and capital markets.

At Scotiabank, we value the unique skills and experiences each individual brings to the Bank, and are committed to creating and maintaining an inclusive and accessible environment for everyone. If you require accommodation (including, but not limited to, an accessible interview site, alternate format documents, ASL Interpreter, or Assistive Technology) during the recruitment and selection process, please let our Recruitment team know. We thank all applicants for their interest in a career at Scotiabank; however, only those candidates who are selected for an interview will be contacted.

Scotiabank is an equal opportunity employer. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, veteran status, or any other characteristic protected by federal, state, or local law.