Principal AI/ML Engineer

Fidelity Investments Inc.; North Carolina; On-site; USD 180,000 - 230,000; Full time

Job summary

Fidelity Investments Inc. seeks a Principal AI/ML Engineer to design and deploy secure, scalable AI systems across cloud platforms. You will build robust pipelines for training, inference, and monitoring, with emphasis on governance, safety, and compliance.

The role involves mentoring engineers, advancing federated learning and multi-agent safety concepts, and collaborating with cybersecurity and data science teams to embed AI safety into enterprise architectures. Onsite work model applies.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, IT, or related field (or foreign equivalent).
• Five years of experience as a Principal AI/ML Engineer developing ML platform applications for Cloud infrastructures using Agile methods.
• Master’s degree option with three years of experience as Principal AI/ML Engineer.
• Experience designing secure, scalable, distributed AI/ML systems with IAM and encryption.
• Proficient with IaC and cloud-native platforms (Terraform, AWS, Azure, GCP) and container orchestration (Kubeflow).

Responsibilities

• Architect AI/ML systems for training, inference, observability, and monitoring across distributed clouds.
• Develop secure AI frameworks, including adversarial robustness and governance.
• Build agentic AI workflows automating data pipelines and model lifecycle operations.
• Support R&D of advanced AI/ML methods including federated learning and multi-agent safety.
• Evaluate performance, reliability, and robustness under real-world conditions.
• Collaborate with cybersecurity, cloud engineering, and data science teams to ensure governance.
• Mentor teams on ML algorithms, security, and best practices.
• Provide technical guidance on ML infrastructure and deployment patterns.

Skills

AI/ML system design

Cloud architecture

MLOps and CI/CD

Python/Java

Security awareness

Education

Bachelor’s degree in Computer Science or related field

Five years of experience as Principal AI/ML Engineer

Tools

Terraform

AWS

Azure

GCP

Kubeflow

Jenkins

Git

MLflow

Job description

Job Description:

Note: Fidelity will not provide immigration sponsorship for this position.

Position Description:

Designs and develops advanced Machine Learning (ML) and trustworthy AI systems that support large-scale, enterprise-wide platforms. Focuses on secure model development, AI safety, multi-agent orchestration, and cloud-scale ML infrastructure, enabling reliable, transparent, and high-assurance deployment of AI across critical business functions. Builds robust pipelines for model training, inference, and continuous monitoring, ensuring compliance and resilience under real-world conditions. Designs model lifecycle operations (MLOps) infrastructure including SageMaker and Kubernetes for reproducible, scalable, and secure ML deployment. Responsible for research and prototyping of cutting-edge AI methodologies, including federated learning, adversarial robustness, and multi-agent safety mechanisms.

Primary Responsibilities:

• Architects and implements AI/ML systems that support model training, inference, observability, and continuous monitoring across distributed cloud environments.
• Develops secure and trustworthy AI frameworks, including adversarial robustness pipelines, anomaly detection models, and model governance mechanisms that ensure compliance, transparency, and risk mitigation.
• Builds and optimizes agentic AI workflows that automate data pipelines, model lifecycle operations, and system self-diagnostics.
• Supports research and prototyping of advanced AI/ML methodologies, including hybrid neural architectures, federated learning, adversarial learning, and multi-agent AI safety mechanisms.
• Conducts performance, reliability, and robustness evaluations of AI systems under real-world constraints, high-throughput workloads, and adversarial conditions.
• Collaborate with cybersecurity, cloud engineering, and data science teams to integrate AI safety and model assurance into enterprise architectures.
• Collaborates with cross-functional teams to integrate AI safety and governance into enterprise architectures while mentoring engineering teams on advanced ML algorithms and secure development practices.
• Delivers technical guidance and mentorship to cross-functional engineering teams on advanced ML algorithms, infrastructure patterns, and secure development practices.

Education and Experience:

Bachelor’s degree in Computer Science, Engineering, Information Technology, Information Systems, or a closely related field (or foreign education equivalent) and five (5) years of experience as a Principal AI/ML Engineer (or closely related occupation) developing ML platform applications for Cloud infrastructures (Amazon Web Services (AWS), Azure, Google, and IBM) using agile methodologies.

Or, alternatively, Master’s degree in Computer Science, Engineering, Information Technology, Information Systems, or a closely related field (or foreign education equivalent) and three (3) years of experience as a Principal AI/ML Engineer (or closely related occupation) developing ML platform applications for Cloud infrastructures (Amazon Web Services (AWS), Azure, Google, and IBM) using agile methodologies.

Skills and Knowledge:

Candidate must also possess:

• Demonstrated Expertise (“DE”) delivering scalable, secure, and distributed applications with robust Identity and Access Management (IAM) and encryption (Knowledge Management System (KMS)), by architecting and deploying enterprise-scale AI/ML systems and auto-ML infrastructure through Infrastructure as code (IAC) and using Cloud-native platforms (Terraform, AWS, Azure, or GCP) and container orchestration frameworks (Kubeflow).
• DE architecting and engineering high-performance big data applications (AWS Glue, EMR, Kinesis, Athena, and Dynamo DB) and autonomous multi-agent systems; designing batch processing jobs and Extract, Transform, Load (ETL) pipelines to support predictive analytics using Hadoop, MongoDB, AWS, and PostgreSQL; developing agentic workflows using frameworks including Strands, CrewAI, LangGraph, and OpenAI Swarm with protocols -- Model Context Protocol (MCP) Server and Accelerated Graphics Port (AGP); and driving context aware predictive analytics and optimization models by implementing multi threaded, asynchronous solutions in Python and Java, supported by short term and long term memory management architectures for Retrieval Augmented Generation (RAG) pipelines using vector databases, OpenSearch, and high performance caching solutions (Redis or Memcached).
• DE automating end-to-end application deployment and MLOps via workflow orchestration tools (Airflow and AWS Step Functions) and CI/CD pipelines (Jenkins and Git); performing model metadata processing to deliver lineage tracking, version control, auditability, reproducibility, and real time analysis of model performance, parameters, and deployment history across distributed ML workflows, using AWS DynamoDB, AWS RDS, MLflow, and AWS Athena; enabling reproducible builds, integrity checks, and scalable CI/CD by securely handling, versioning, and distributing container images, ML models, and software dependencies using JFrog Artifactory; and accelerating model development, monitoring, drift detection, and interpretability within Agile environments by engineering bridge solutions, using Go and FastAPI.
• DE implementing secure AI through adversarial robustness, federated learning, and governance across distributed ML systems using AI Generative Adversarial Network (AIGAN), Google Federated Learning Framework, SageMaker Clarify, and MLflow; enhancing low latency, high throughput model serving and maximizing central processing unit (CPU) or graphics processing unit (GPU) utilization through deployment on accelerated inference servers, using Deep Java Library (DJL), Triton, and Flask; and evaluating ML model inference performance using statistical analysis, monitoring tools (CloudWatch, Datadog, and Splunk), and dashboards including Streamlit and Gradio.

[Experience and/or expertise may be gained during doctoral program.]

#PE1M2

#LI-DNI

Fidelity’s Onsite Working Model
Fidelity is transitioning to a full-time onsite working model through a phased rollout across regions and roles. Currently, some roles and locations require 100% onsite presence, while others require less. Onsite expectations are likely to evolve as the rollout continues. This transition does not apply to fully remote roles.

Category:

Information Technology

Please be advised that Fidelity’s business is governed by the provisions of the Securities Exchange Act of 1934, the Investment Advisers Act of 1940, the Investment Company Act of 1940, ERISA, numerous state laws governing securities, investment and retirement-related financial activities and the rules and regulations of numerous self-regulatory organizations, including FINRA, among others. Those laws and regulations may restrict Fidelity from hiring and/or associating with individuals with certain Criminal Histories.