Senior AI Ops Engineer - Infrastructure

USAA Life Insurance Company; San Antonio (TX); Hybrid; USD 150,000 - 190,000; Full time

Job summary

USAA Life Insurance Company is seeking a Senior AI Ops Engineer - Infrastructure in San Antonio, TX. You will build pipelines ingesting infrastructure telemetry, train models to baseline normal behavior, and automate remediation with safe rollback.

This role operates in a regulated financial context with strict audit controls and an on-site requirement five days a week. You’ll work with Terraform, Ansible, Kubernetes, and model governance to ensure reliability across compute, network, storage,

Qualifications

• Bachelor’s degree or 4 years of relevant education/experience.
• 5+ years of hands-on infrastructure, platform, or site-reliability engineering experience with strong programming and scripting (Python + Go or similar).
• Experience building and productionizing ML models for operational data (anomaly detection, forecasting, capacity modeling).
• Deep, hands-on experience with infrastructure-as-code and orchestration tooling for production compute, network, storage.

Responsibilities

• Build and operate high-throughput ingestion and feature pipelines for infrastructure telemetry into the AIOps feature store.
• Train, validate, and tune unsupervised and time-series models to baseline normal infrastructure behavior and detect anomalies.
• Create dynamic topology graphs and event correlation logic to reduce alert volume reaching humans.
• Write and test remediation automation with rollback and safety checks within governance frameworks.
• Monitor and retrain models and automations; participate in on-call rotation.

Skills

Python

Go

TensorFlow

PyTorch

scikit-learn

Terraform

Ansible

Kubernetes

on-call engineering

Education

Bachelor’s degree; OR 4 years of relevant education and/or experience

Tools

CMDB

Terraform

Kubernetes operators

Moogsoft

Job description

Why USAA?

At USAA, our mission is to empower our members to achieve financial security through highly competitive products, exceptional service and trusted advice. We seek to be the #1 choice for the military community and their families.

Embrace a fulfilling career at USAA, where our core values - honesty, integrity, loyalty and service - define how we treat each other and our members. Be part of what truly makes us special and impactful.

We are proud to support active-duty military spouses. USAA roles may offer remote or hybrid flexibility for active-duty military spouses consistent with applicable policy and business needs.

The Opportunity

The Infrastructure AI Ops team owns AI-enabled operations for everything the application estate runs on - compute, network, storage, and the cloud and on-prem platforms underneath it.

As a Senior AI Ops Engineer - Infrastructure on this team, you'll build the pipelines that ingest infrastructure telemetry at real scale; train the models that baseline normal capacity and resource behavior and catch the anomaly before it becomes an outage - a disk filling, a node drifting out of healthy state, a network path degrading; and then write the automation that acts on it - a safe auto-scale, a graceful node drain and replace, a volume expansion with a tested rollback. This is a from-scratch build inside a regulated financial institution, so the reliability and audit bar is real: every automated action you ship needs a kill switch, a rollback path, and a reason a risk examiner would accept.

This position offers a flexible work environment with an onsite requirement of 5 days per week. This position is based in San Antonio, TX. Relocation assistance is not available for this position.

What you’ll do:

• Build and operate high-throughput ingestion and feature pipelines that consume infrastructure telemetry - compute, network, storage, container, and cloud platform signals collected by the Observability & Telemetry team - into the AIOps feature store.
• Train, validate, and tune unsupervised and time-series models - isolation forests, autoencoders, LSTMs, or comparable techniques - to baseline normal infrastructure behavior and detect capacity and resource anomalies before they become incidents.
• Build the dynamic service and infrastructure topology graph from CMDB, discovery, and live signal, and design the event correlation logic that reduces raw infrastructure alert volume reaching a human.
• Write, test, and version remediation automation - Terraform, Ansible, or equivalent - for infrastructure failure modes, with a tested rollback or compensating action for every change, operating within the approval-gate, blast-radius, and kill-switch framework.
• Instrument and monitor your own models and automations in production, retraining on a defined cadence, and participate in the on-call rotation for any live action at autonomy level 3 or above.

What you have:

• Bachelor’s degree; OR 4 years of relevant education and/or experience.
• 5+ years of hands‑on infrastructure, platform, or site-reliability engineering experience, with strong programming and scripting ability - Python plus Go or a comparable systems language.
• Practical experience building and productionizing ML models for operational data - anomaly detection, forecasting, or capacity modeling - using frameworks such as TensorFlow, PyTorch, or scikit-learn.
• Deep, hands‑on experience with infrastructure-as-code and orchestration tooling - Terraform, Ansible, Kubernetes operators, or equivalent - for production compute, network, and storage environments.
• Experience writing and testing operational automation with rollback, compensating actions, and approval-gated or policy-controlled execution for infrastructure changes specifically, not just application deploys.
• Comfort operating in a regulated environment, with demonstrated ability to debug and reason about a live production infrastructure incident under time pressure.

What sets you apart:

• Demonstrated experience designing, deploying, and operationalizing model-driven capacity management or anomaly-detection capabilities that support infrastructure operations and on-call engineering teams.
• Experience developing guard railed automation for production infrastructure, including automated scaling, node replacement, or storage volume expansion, with tested kill-switch and safety-control mechanisms.
• Experience designing and implementing evaluation frameworks for operational AI, including test cases, regression suites, and drift detection to identify model or threshold degradation prior to production deployment.
• Experience building or maintaining runtime service or infrastructure topology graphs and reconciling topology data with systems of record, such as aConfiguration Management Database (CMDB), with familiarity consuming outputs from enterprise AIOps and observability platforms such as Moogsoft, BigPanda, or Dynatrace.
• Experience partnering directly with model governance or model risk