AI Safety Engineer: Guardrails & Evaluation

Realtor.com; Austin (TX); On-site; USD 150,000 - 185,000; Full time

Benefits offered by this job

Medical coverage

Family forming benefits

13 Paid Holidays

Flexible Time Off

401(k) match

Tuition Reimbursement

Volunteer time off

Job summary

Realtor.com is seeking an AI Engineer to join our AI Integrations Team to build and tune classifiers that keep our AI assistant compliant and safe. You will own the guardrail layer, including evaluation pipelines, cloud guardrails, and infrastructure as code while partnering with ML, backend, product and legal teams.

You will work on Fair-Housing compliance, content moderation, and safety testing for RealAssist, with a focus on high recall without overblocking legitimate users.

Qualifications

• AI Safety & Evaluation expertise with real-world ML experience.
• 4+ years of software/ML engineering with LLM work.
• Bachelor’s degree or equivalent experience.
• Metric literacy: precision/recall/F1, confusion matrices, calibration.
• Experience building or tuning LLM prompts/classifiers and evaluating with eval frameworks.
• Familiarity with prompt-injection/jailbreak defense concepts.

Responsibilities

• LLM-as-a-judge classifiers for Fair-Housing compliance and content moderation.
• Guardrail evaluation harness development and maintenance.
• Build, tune, and own LLM-as-a-judge classifiers for safety and content filtering.
• Design/run guardrail evaluation pipelines with labeled data and splits.
• Report confusion matrices and precision/recall/F1 per category.
• Integrate cloud guardrails as runtime screening with fail-open behavior.
• Red-team the assistant and integrate safety checks into CI pipelines.
• Manage guardrail infrastructure as code with Terraform.
• Collaborate with ML, backend, product and legal teams for risk tiering.

Skills

AI Safety

Evaluation Expertise

4+ years experience

LLM applications

Prompt classifiers

Dataset curation

Red-teaming

Terraform / IaC

Google Cloud Vertex AI

Observability tools

Compliance awareness

Fair housing knowledge

Education

Bachelor's degree or equivalent

Tools

Google Cloud Vertex AI

Terraform

Model Armor

LangSmith/OpenAI Evals

Arize Phoenix

Job description

Realtor.com is seeking an AI Engineer to join our AI Integrations Team to build and tune classifiers that keep our AI assistant compliant and safe. You will own the guardrail layer, including evaluation pipelines, cloud guardrails, and infrastructure as code while partnering with ML, backend, product and legal teams.

You will work on Fair-Housing compliance, content moderation, and safety testing for RealAssist, with a focus on high recall without overblocking legitimate users.