Senior Staff Machine Learning Engineer

Zscaler; Washington (District of Columbia); Remote; USD 158,000 - 225,000; Full time

Benefits offered by this job

Time off plans

Parental leave

Retirement options

In-office perks

Job summary

Zscaler is seeking a Senior Staff Machine Learning Engineer to join our AI Platform and Data Science team. This remote (USA) role reports to the Manager AI Platform and Data Science and focuses on high-fidelity risk identification in customer data using AI, ML, and data engineering to automate security analysis.

You'll deliver production-ready LLM-powered agents, collaborate with threat-research, and own end-to-end pipelines, ensuring quality and reliability in a CI/CD-driven environment.

Qualifications

• Experience building LLM-powered agents in production with tool and function calling, prompt/context engineering, multi-step orchestration frameworks, and evaluation of non-deterministic output against ground truth.
• 8+ years of professional Python development with production-quality systems, validated inputs, data contracts, and tests; SQL and Python data analytics libraries (pandas, NumPy).
• Ownership of production reliability: CI/CD, containerization, logging and metrics, observability and on-call incident debugging in AWS.
• Sr. Staff level ownership and leadership of emergent requirements, architecture, and complex projects.

Responsibilities

• Translate risk identification methods into agent logic and ensure quality across risk analysis and recommendations.
• Collaborate with threat-research to understand data and threats for security heuristics.
• Identify data requirements and work with data engineering for pipelines and enrichments.
• Apply data-driven quality approach to threat detection, backtesting, precision/recall tuning, and QC.
• Deploy and monitor solutions in production within CI/CD.

Skills

LLM-powered agents

Python development

SQL

Data analytics (Pandas/NumPy)

CI/CD & AWS

Leadership

Tools

LangChain

LangGraph

OpenSearch/Elasticsearch

Job description

Remote - USA; Santa Clara, California, USA

Zscaler (NASDAQ: ZS) accelerates digital transformation so customers can be more agile, efficient, resilient, and secure. The Zscaler Zero Trust Exchange platform protects thousands of customers from cyberattacks and data loss by securely connecting users, devices, and applications in any location. Distributed across 160+ public exchanges globally and thousands of private exchanges at the edge, the SASE-based Zero Trust Exchange is the world’s largest in-line cloud security platform.

We believe the future of work is Human + AI and are building an AI-native enterprise where human potential is amplified by machine intelligence to solve the world’s hardest security challenges. Driven by deep customer obsession, we are committed to the mission, outcome, and to each other. We bring these commitments to life through three core behaviors: ownership and collaboration, trust through outcomes and impact, and a challenge culture with ongoing feedback. Ready to make an impact at the company pioneering security transformation in the AI era? Join us at Zscaler.

Role

We are looking for a Senior Staff Machine Learning Engineer to join our team. This is a remote (USA) role, reporting to the Manager AI Platform and Data Science in the AI Platform and Data Science department. This team’s mission is high-fidelity risk identification in customer data, with goals to catch all threats while minimizing noise. To achieve this, you will develop solutions applying data analysis and threat-research while leveraging AI, machine learning, and data engineering to build quality, data-driven components to automate security analysis.

What you’ll do (Role Expectations)

• Translate risk identification methods into agent logic, understanding the benefits and limitations of agents and ensuring quality across risk analysis, explanations, and recommendations
• Collaborate closely with threat-research to understand data, threats, and their approach to developing security heuristics
• Identify and solve data requirements for analysis, developing and collaborating with data engineering teams for pipelines, enrichments, and aggregations
• Follow a data-driven quality approach to threat detection, including backtesting, balancing precision vs recall, tuning, and quality control
• Deploy and monitor your solutions in production within our CI/CD framework

Who You Are (Success Profile)

• You thrive in ambiguity and build the path as you walk it, seeing unstructured challenges as the raw material to construct meaningful systems.
• You act like an owner with a deep passion for the mission, operating with integrity and navigating seamlessly between high-level strategy and hands-on execution.
• You are a continuous learner with a growth mindset who actively seeks feedback to develop yourself, elevate your partners, and execute with purpose.
• You are a positive force who approaches complex technical challenges with contagious, constructive energy and a focus on solutions.
• You are data-driven, using analytics and concrete evidence over assumptions to find the truth, measure what matters, and guide informed decisions.

What We’re Looking for (Minimum Qualifications)

• Experience building LLM-powered agents in production: tool and function calling, prompt and context engineering, multi-step orchestration frameworks (LangGraph, LangChain, or equivalent), and evaluation of non-deterministic output against ground truth
• 8+ years of professional Python development with demonstrated ability to design and maintain production-quality systems with validated inputs, data contracts, and comprehensive unit and integration tests, alongside hands-on expertise with SQL and Python data analytics libraries (e.g., pandas, Polars, NumPy)
• Demonstrated ownership of production reliability: CI/CD, containerization, structured logging and metrics, observability and alerting, on-call participation, incident debugging in live distributed systems, and familiarity with cloud and infrastructure-as-code in an AWS environment
• Sr. Staff level ownership and leadership of emergent requirements, architecture, and complex engineering projects

What Will Make You Stand Out (Preferred Qualifications)

• Experience with production AI or ML systems where cost, latency, and accuracy are competing constraints, including handling model selection, routing, and regression testing
• Background in cyber threat research, threat modeling, threat hunting, detection engineering, or previous experience in data-driven risk analysis, fraud detection, adversary profiling and targeting, actuarial risk, or close collaboration with risk analysis teams
• Data engineering experience building and operating pipelines over large-volume event data using SQL, columnar or search-backed stores (OpenSearch/Elasticsearch, Athena/Presto), schema evolution, backfills, and data quality validation

Base Pay Range

$157,500 - $225,000 USD

At Zscaler, we are committed to building a team that reflects the communities we serve and the customers we work with. We foster an inclusive environment that values all backgrounds and perspectives, emphasizing collaboration and belonging. Join us in our mission to make doing business seamless and secure.

Our Benefits program is one of the most important ways we support our employees. Zscaler proudly offers comprehensive and inclusive benefits to meet the diverse needs of our employees and their families throughout their life stages, including:

• Time off plans for vacation and sick time
• Parental leave options
• Retirement options
• In-office perks, and more!

Learn more about Zscaler's hybrid working model and benefits here .

By applying for this role, you adhere to applicable laws, regulations, and Zscaler policies, including those related to security and privacy standards and guidelines.

Zscaler is committed to providing equal employment opportunities to all individuals. We strive to create a workplace where employees are treated with respect and have the chance to succeed. All qualified applicants will be considered for employment without regard to race, color, religion, sex (including pregnancy or related medical conditions), age, national origin, sexual orientation, gender identity or expression, genetic information, disability status, protected veteran status, or any other characteristic protected by federal, state, and local laws. See more information by clicking on the Know Your Rights: Workplace Discrimination is Illegal link.

Pay Transparency

Zscaler complies with all applicable federal, state, and local pay transparency rules.

Zscaler is committed to providing reasonable support (called accommodations or adjustments) in our recruiting processes for candidates who are differently abled, have long term conditions, mental health conditions or sincerely held religious beliefs, or who are neurodivergent or require pregnancy-related support.