Principal AI/ML Engineer

Habitat For Humanity Of Durham; Durham (NC); On-site; USD 140,000 - 180,000; Full time

Job summary

Fidelity Investments seeks a Principal AI/ML Engineer to design and deploy secure, scalable AI systems across enterprise cloud environments. You will lead model training, inference, and monitoring while advancing governance and risk mitigation.

Responsibilities span federated learning, adversarial robustness, and multi-agent safety with collaboration across cybersecurity and cloud teams. Onsite role with modern AI tooling.

Qualifications

• Demonstrated Expertise delivering scalable, secure distributed applications with IAM and encryption.
• Architecting AI/ML systems on cloud-native platforms with CI/CD.
• Auto-ML infrastructure via IaC and container orchestration.
• Secure AI methods including adversarial robustness and federated learning.

Responsibilities

• Architects and implements AI/ML systems for training, inference, observability, and monitoring across distributed clouds.
• Develops secure AI frameworks, including adversarial robustness pipelines and model governance.
• Builds agentic AI workflows automating data pipelines and model lifecycle operations.
• Supports research on hybrid architectures, federated learning, and multi-agent safety.
• Performs performance, reliability, and robustness evaluations under real-world constraints.
• Collaborates with cybersecurity, cloud engineering, and data science to integrate safety.
• Mentors teams on advanced ML algorithms and secure development practices.

Skills

IAM & encryption

KMS

Terraform

Cloud platforms

Kubeflow

MLOps

Python

Java

Adversarial robustness

Education

Bachelor's in CS/Eng

Tools

SageMaker

Kubernetes

Airflow

MLflow

DynamoDB

Job description

Job Description:

Note: Fidelity will not provide immigration sponsorship for this position.

Position Description:

Designs and develops advanced Machine Learning (ML) and trustworthy AI systems that support large-scale, enterprise-wide platforms. Focuses on secure model development, AI safety, multi-agent orchestration, and cloud-scale ML infrastructure, enabling reliable, transparent, and high-assurance deployment of AI across critical business functions. Builds robust pipelines for model training, inference, and continuous monitoring, ensuring compliance and resilience under real-world conditions. Designs model lifecycle operations (MLOps) infrastructure including SageMaker and Kubernetes for reproducible, scalable, and secure ML deployment. Responsible for research and prototyping of cutting-edge AI methodologies, including federated learning, adversarial robustness, and multi-agent safety mechanisms.

Primary Responsibilities:

• Architects and implements AI/ML systems that support model training, inference, observability, and continuous monitoring across distributed cloud environments.




• Develops secure and trustworthy AI frameworks, including adversarial robustness pipelines, anomaly detection models, and model governance mechanisms that ensure compliance, transparency, and risk mitigation.




• Builds and optimizes agentic AI workflows that automate data pipelines, model lifecycle operations, and system self-diagnostics.




• Supports research and prototyping of advanced AI/ML methodologies, including hybrid neural architectures, federated learning, adversarial learning, and multi-agent AI safety mechanisms.




• Conducts performance, reliability, and robustness evaluations of AI systems under real-world constraints, high-throughput workloads, and adversarial conditions.




• Collaborate with cybersecurity, cloud engineering, and data science teams to integrate AI safety and model assurance into enterprise architectures.




• Collaborates with cross-functional teams to integrate AI safety and governance into enterprise architectures while mentoring engineering teams on advanced ML algorithms and secure development practices.




• Delivers technical guidance and mentorship to cross-functional engineering teams on advanced ML algorithms, infrastructure patterns, and secure development practices.

Education and Experience:

Bachelor’s degree in Computer Science, Engineering, Information Technology, Information Systems, or a closely related field (or foreign education equivalent) and five (5) years of experience as a Principal AI/ML Engineer (or closely related occupation) developing ML platform applications for Cloud infrastructures (Amazon Web Services (AWS), Azure, Google, and IBM) using agile methodologies.

Or, alternatively, Master’s degree in Computer Science, Engineering, Information Technology, Information Systems, or a closely related field (or foreign education equivalent) and three (3) years of experience as a Principal AI/ML Engineer (or closely related occupation) developing ML platform applications for Cloud infrastructures (Amazon Web Services (AWS), Azure, Google, and IBM) using agile methodologies.

Skills and Knowledge:

Candidate must also possess:

• Demonstrated Expertise (“DE”) delivering scalable, secure, and distributed applications with robust Identity and Access Management (IAM) and encryption (Knowledge Management System (KMS)), by architecting and deploying enterprise-scale AI/ML systems and auto-ML infrastructure through Infrastructure as code (IAC) and using Cloud-native platforms (Terraform, AWS, Azure, or GCP) and container orchestration frameworks (Kubeflow).




• DE architecting and engineering high-performance big data applications (AWS Glue, EMR, Kinesis, Athena, and Dynamo DB) and autonomous multi-agent systems; designing batch processing jobs and Extract, Transform, Load (ETL) pipelines to support predictive analytics using Hadoop, MongoDB, AWS, and PostgreSQL; developing agentic workflows using frameworks including Strands, CrewAI, LangGraph, and OpenAI Swarm with protocols -- Model Context Protocol (MCP) Server and Accelerated Graphics Port (AGP); and driving context aware predictive analytics and optimization models by implementing multi threaded, asynchronous solutions in Python and Java, supported by short term and long term memory management architectures for Retrieval Augmented Generation (RAG) pipelines using vector databases, OpenSearch, and high performance caching solutions (Redis or Memcached).




• DE automating end-to-end application deployment and MLOps via workflow orchestration tools (Airflow and AWS Step Functions) and CI/CD pipelines (Jenkins and Git); performing model metadata processing to deliver lineage tracking, version control, auditability, reproducibility, and real time analysis of model performance, parameters, and deployment history across distributed ML workflows, using AWS DynamoDB, AWS RDS, MLflow, and AWS Athena; enabling reproducible builds, integrity checks, and scalable CI/CD by securely handling, versioning, and distributing container images, ML models, and software dependencies using JFrog Artifactory; and accelerating model development, monitoring, drift detection, and interpretability within Agile environments by engineering bridge solutions, using Go and FastAPI.




• DE implementing secure AI through adversarial robustness, federated learning, and governance across distributed ML systems using AI Generative Adversarial Network (AIGAN), Google Federated Learning Framework, SageMaker Clarify, and MLflow; enhancing low latency, high throughput model serving and maximizing central processing unit (CPU) or graphics processing unit (GPU) utilization through deployment on accelerated inference servers, using Deep Java Library (DJL), Triton, and Flask; and evaluating ML model inference performance using statistical analysis, monitoring tools (CloudWatch, Datadog, and Splunk), and dashboards including Streamlit and Gradio.

[Experience and/or expertise may be gained during doctoral program.]

#PE1M2

#LI-DNI

Fidelity’s Onsite Working Model
Fidelity is transitioning to a full-time onsite working model through a phased rollout across regions and roles. Currently, some roles and locations require 100% onsite presence, while others require less. Onsite expectations are likely to evolve as the rollout continues. This transition does not apply to fully remote roles.

Certifications:

Category:

Information Technology

Please be advised that Fidelity’s business is governed by the provisions of the Securities Exchange Act of 1934, the Investment Advisers Act of 1940, the Investment Company Act of 1940, ERISA, numerous state laws governing securities, investment and retirement-related financial activities and the rules and regulations of numerous self-regulatory organizations, including FINRA, among others. Those laws and regulations may restrict Fidelity from hiring and/or associating with individuals with certain Criminal Histories.