AI Safety Engineer: LLM Guardrails & Compliance

DataJobs; Austin (TX); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Medical coverage

13 Paid Holidays

401(k) with match

Job summary

Realtor.com is seeking a senior AI safety engineer to own LLM safety guardrails for RealAssist, an AI-powered real-estate assistant. You will build classifiers for Fair Housing compliance and content moderation, tuning evaluation pipelines and runtime screening layers across cloud guardrail services.

Collaborate with ML, backend, product, and legal teams to drive safe, scalable AI. The role emphasizes balancing safety, latency, and user experience, with onsite collaboration and strong tooling

Qualifications

• Experience building AI safety and evaluation pipelines for production ML systems.
• Strong Python coding skills and hands-on ML engineering background.
• Familiarity with prompt-injection defense and security testing concepts.
• Exposure to regulated domains (fair housing, compliance) is a plus.
• Ability to design labeled eval datasets from seed ideas and synthesize data.

Responsibilities

• Own end-to-end LLM-as-a-judge classifiers for Fair-Housing compliance and content moderation.
• Design guardrail evaluation pipelines with labeled dataset curation, train/test/validate splits, and offline prod-replay.
• Use defensible metrics by reporting confusion matrices and precision/recall/F1 per category.
• Integrate and operate cloud guardrail services as a runtime screening layer with fail-open behavior and alerting.
• Red-team the assistant and wire safety regression checks into CI to prevent guardrail degradation.
• Manage guardrail infrastructure as code using Terraform and participate in design reviews.
• Partner with ML, backend, product, and legal/compliance to define risk tiering and safety reviews.
• Independently design, implement, and tune guardrail and evaluation systems balancing safety, latency, and UX.
• Keep eval datasets calibrated to real traffic and evolving attack patterns; stay current with AI-safety research.

Skills

AI Safety

Python

ML Engineering

Red-Teaming

Evaluation Metrics

Prompt Security

Security Testing

Education

Bachelor's Degree

Tools

Terraform

Vertex AI

AWS Bedrock Guardrails

Azure AI Content Safety

New Relic

OWASP LLM Top 10

DeepEval

LangSmith

Arize Phoenix

OpenAI Evals

Job description

Realtor.com is seeking a senior AI safety engineer to own LLM safety guardrails for RealAssist, an AI-powered real-estate assistant. You will build classifiers for Fair Housing compliance and content moderation, tuning evaluation pipelines and runtime screening layers across cloud guardrail services.

Collaborate with ML, backend, product, and legal teams to drive safe, scalable AI. The role emphasizes balancing safety, latency, and user experience, with onsite collaboration and strong tooling