AI Safety Engineer — Guardrails & LLM Evaluation

News Corp; Austin, Northern (TX, KY); Hybrid; USD 140,000 - 210,000; Full time

Job summary

Realtor.com® is seeking an AI Engineer to join the AI Integrations Team in Austin, TX. You will build and tune classifiers that keep the AI assistant compliant, safe, and helpful, working at the intersection of AI and consumer-facing products.

You will own the AI safety layer end-to-end, design guardrail evaluation pipelines, and collaborate with ML, backend, product, and legal teams to ensure safe launches.

Qualifications

• AI Safety & Evaluation Expertise (Required)
• 4+ years of professional software/ML engineering experience with hands-on LLM application work
• Bachelor’s degree or equivalent experience
• Strong Python proficiency
• Real classification-metrics literacy – precision/recall/F1, confusion matrix debugging, dataset curation, and production distribution calibration
• Experience building or tuning LLM prompts/classifiers and evaluating them with an eval framework
• Experience turning seed sets into robust labeled eval datasets
• Familiarity with prompt‑injection / jailbreak defense concepts
• Hands-on with cloud guardrail services and IaC
• Exposure to regulated/compliance‑sensitive domains
• Red‑teaming or AI‑security background

Responsibilities

• Independently design, implement, and tune guardrail and evaluation systems.
• Make architectural decisions balancing safety, latency, and user experience.
• Write clean, maintainable, well-tested code that sets team standards.
• Keep eval datasets calibrated to real traffic and emerging attack patterns.
• Stay current with AI-safety research, jailbreak techniques, and guardrail tooling
• Own the AI safety layer end-to-end from design through deployment.
• Anticipate problems and address them through red-teaming and regression checks.
• Make data-driven decisions grounded in eval metrics.
• Deliver high-quality work consistently on schedule.

Skills

AI Safety & Evaluation

Python

LLM prompts/classifiers

Dataset synthesis

Evaluation frameworks

Prompt-injection defense

Guardrails infra (IaC)

Cloud guardrail services

Terraform / IaC

Google Cloud Vertex AI / Gemini

Monitoring / observability

Regulatory/compliance domain knowledge

Red-teaming / AI-security

Education

Bachelor’s degree or equivalent experience

Tools

DeepEval

RAGAS

Arize Phoenix

LangSmith

OpenAI Evals

Job description

Realtor.com® is seeking an AI Engineer to join the AI Integrations Team in Austin, TX. You will build and tune classifiers that keep the AI assistant compliant, safe, and helpful, working at the intersection of AI and consumer-facing products.

You will own the AI safety layer end-to-end, design guardrail evaluation pipelines, and collaborate with ML, backend, product, and legal teams to ensure safe launches.