AI Engineer

DataJobs; Austin (TX); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Medical coverage

13 Paid Holidays

401(k) with match

Job summary

Realtor.com is seeking a senior AI safety engineer to own LLM safety guardrails for RealAssist, an AI-powered real-estate assistant. You will build classifiers for Fair Housing compliance and content moderation, tuning evaluation pipelines and runtime screening layers across cloud guardrail services.

Collaborate with ML, backend, product, and legal teams to drive safe, scalable AI. The role emphasizes balancing safety, latency, and user experience, with onsite collaboration and strong tooling

Qualifications

• Experience building AI safety and evaluation pipelines for production ML systems.
• Strong Python coding skills and hands-on ML engineering background.
• Familiarity with prompt-injection defense and security testing concepts.
• Exposure to regulated domains (fair housing, compliance) is a plus.
• Ability to design labeled eval datasets from seed ideas and synthesize data.

Responsibilities

• Own end-to-end LLM-as-a-judge classifiers for Fair-Housing compliance and content moderation.
• Design guardrail evaluation pipelines with labeled dataset curation, train/test/validate splits, and offline prod-replay.
• Use defensible metrics by reporting confusion matrices and precision/recall/F1 per category.
• Integrate and operate cloud guardrail services as a runtime screening layer with fail-open behavior and alerting.
• Red-team the assistant and wire safety regression checks into CI to prevent guardrail degradation.
• Manage guardrail infrastructure as code using Terraform and participate in design reviews.
• Partner with ML, backend, product, and legal/compliance to define risk tiering and safety reviews.
• Independently design, implement, and tune guardrail and evaluation systems balancing safety, latency, and UX.
• Keep eval datasets calibrated to real traffic and evolving attack patterns; stay current with AI-safety research.

Skills

AI Safety

Python

ML Engineering

Red-Teaming

Evaluation Metrics

Prompt Security

Security Testing

Education

Bachelor's Degree

Tools

Terraform

Vertex AI

AWS Bedrock Guardrails

Azure AI Content Safety

New Relic

OWASP LLM Top 10

DeepEval

LangSmith

Arize Phoenix

OpenAI Evals

Job description

Own the AI safety layer for Realtor.com’s RealAssist, an LLM-powered real-estate assistant used across web and mobile. On the AI Integrations Team at News Corp, you will build and tune LLM safety guardrails that act as a launch gate for Fair Housing compliance and content moderation, with runtime prompt-injection and jailbreak screening designed for production.

What you’ll do

• Build, tune, and take end-to-end ownership of LLM-as-a-judge classifiers for Fair-Housing compliance and content moderation, targeting high recall on disallowed content without overblocking legitimate users
• Design and run guardrail evaluation pipelines, including labeled dataset curation, train/test/validate splits, and offline prod-replay to evaluate overblocking
• Use defensible metrics by reporting confusion matrices and precision/recall/F1 per category to guide safety decisions
• Integrate and operate cloud guardrail services as a runtime prompt-injection/jailbreak screening layer, including fail-open behavior and alerting
• Red-team the assistant and wire safety regression checks into CI to prevent silent guardrail degradation as the product grows
• Manage guardrail infrastructure as code using Terraform (templates, IAM, and project shape), and participate in technical design reviews and architecture discussions
• Partner with ML, backend, product, and legal/compliance stakeholders to define risk tiering and get new AI capabilities through safety review before launch
• Independently design, implement, and tune guardrail and evaluation systems with a focus on balancing safety, latency, and user experience; write clean, maintainable, well-tested code
• Keep eval datasets calibrated to real traffic and emerging attack patterns, staying current with AI-safety research and new guardrail tooling

What you bring

• AI Safety & Evaluation Expertise (required)
• 4+ years of professional software/ML engineering experience with hands‑on LLM application work
• Bachelor’s degree or equivalent experience
• Strong Python proficiency
• Real classification-metrics literacy: precision/recall/F1 (macro vs weighted), confusion matrix debugging, dataset curation, and calibration to production distribution
• Experience evaluating LLM prompts/classifiers with an eval framework such as DeepEval, RAGAS, Arize Phoenix, LangSmith, OpenAI Evals, or a homegrown harness
• Ability to turn small seed sets into robust labeled eval datasets using dataset-synthesis and prompt-optimization loops
• Familiarity with prompt-injection/jailbreak defense concepts, including OWASP LLM Top 10, input/output filtering, least-privilege tool access, and adversarial testing
• Hands‑on experience with cloud guardrail services: Google Cloud Model Armor, AWS Bedrock Guardrails, or Azure AI Content Safety
• Terraform/IaC for cloud infrastructure
• Experience with Google Cloud Vertex AI / Gemini
• Understanding of monitoring and observability tools (for example, New Relic or similar)
• Exposure to regulated, compliance-sensitive domains (fair housing, fair lending, healthcare, finance, trust & safety)
• Red‑teaming or AI‑security background

Tools you’ll work with

Python, DeepEval, RAGAS, Arize Phoenix, LangSmith, OpenAI Evals, OWASP LLM Top 10, Google Cloud Model Armor, AWS Bedrock Guardrails, Azure AI Content Safety, Terraform, Google Cloud Vertex AI / Gemini, New Relic

Onsite role with strong support and collaboration

The AI Integrations Team is a cross-functional squad building how Realtor.com integrates AI across its products. The work blends creativity and innovation with in-person collaboration. For most roles, employees work four or more days in office.

Benefits

• Inclusive and competitive medical, Rx, dental, and vision coverage
• Family forming benefits
• 13 Paid Holidays
• Flexible Time Off
• 8 hours of paid volunteer time off
• Immediate eligibility into the Company 401(k) plan with 3.5% company match
• Tuition Reimbursement for degreed and non-degreed programs
• 1:1 personalized Financial Planning Sessions
• Free snacks and refreshments in each office location
• Student Debt Retirement Savings Match program