Senior AI Safety Engineer - Guardrails & LLM Eval

Realtor.com; Austin (TX); In loco; USD 140.000 - 190.000; Tempo pieno

Vantaggi offerti da questo lavoro

Health benefits

401(k) company match

Tuition reimbursement

Paid holidays

Volunteer time off

Snacks in offices

Descrizione del lavoro

Realtor.com is seeking an AI Engineer to join the AI Integrations Team in Austin, TX. You will own the AI safety layer for the RealAssist assistant, building classifiers and guardrails to keep the system compliant, safe, and helpful across web and mobile platforms.

The role requires hands-on ML/LLM work, expertise in evaluation metrics, and experience with cloud guardrails and IaC. You will collaborate with ML, backend, product, and legal teams to drive safe, scalable AI features.

Competenze

• AI safety and evaluation expertise; 5+ years in software/ML with LLM work.
• Strong grasp of precision/recall/F1 and confusion matrices for production models.
• Experience building/tuning LLM prompts and eval frameworks.
• Ability to curate labeled eval datasets and calibrate to production distributions.
• Familiarity with prompt-injection/jailbreak defense concepts and guardrails.
• Experience with cloud guardrails services and IaC (Terraform).
• Knowledge of monitoring tools and compliance-driven domains.

Mansioni

• Design, implement, and tune guardrail and evaluation systems end-to-end.
• Build LLM-as-a-judge classifiers for fair housing compliance and content moderation.
• Develop guardrail evaluation pipelines with labeled datasets and eval runs.
• Integrate cloud guardrail services into runtime prompt screening with fail-open behavior.
• Conduct red-team activities and wire safety checks into CI/CD pipelines.
• Manage guardrail infra as code; participate in design reviews and risk tiering.
• Collaborate with ML, backend, product, and legal teams to vet new capabilities.

Conoscenze

AI Safety

LLM evaluation

Classification metrics

Prompt engineering

Dataset curation

Prompt-injection concepts

Red-teaming

Guardrails infra

IaC / Terraform

Cloud guardrails (Model Armor/Bedrock)

Vertex AI / Gemini

Monitoring/observability

Regulatory/compliance exposure

AI-security background

Formazione

Bachelor’s degree in CS or related field

Equivalent experience

Strumenti

Terraform

Google Cloud Vertex AI

Google Cloud Model Armor

AWS Bedrock Guardrails

Azure AI Content Safety

New Relic

CI/CD tooling

Descrizione del lavoro

Realtor.com is seeking an AI Engineer to join the AI Integrations Team in Austin, TX. You will own the AI safety layer for the RealAssist assistant, building classifiers and guardrails to keep the system compliant, safe, and helpful across web and mobile platforms.

The role requires hands-on ML/LLM work, expertise in evaluation metrics, and experience with cloud guardrails and IaC. You will collaborate with ML, backend, product, and legal teams to drive safe, scalable AI features.