Senior AI Evaluation Engineer: LLM Benchmarking & Safety

Helpware Inc.; México; Remote; MXN 900,000 - 1,500,000; Full time

Job summary

Helpware Inc. seeks a Senior/Staff Evaluation AI Engineer to anchor benchmarking, validation, and safety for our Agentic AI platform used in enterprise real estate workflows.

You will design automated evaluation pipelines, define domain metrics, and partner with Engineering, Product, Data, and Quality teams to ensure measurable, reliable AI capabilities. This 6-month engagement with potential prolongation focuses on defensive alignment testing, tool integration, and scalable regression suites

Qualifications

• 5–7 years in software/QA/AI roles (software engineering, quality engineering, test automation, or related).
• Hands‑on experience evaluating or testing LLM, NLP, ML, or AI applications.
• Strong Python skills and experience building automated test/evaluation workflows.
• Experience creating and maintaining golden/benchmark datasets.
• Experience with adversarial testing.
• Familiarity with AI safety, security & privacy, and responsible AI evaluation.
• Deep LangChain, LangGraph (including sub‑graphs & conditional routing) and LangSmith expertise.
• Hands‑on experience using LangSmith for tracing, experiments, datasets, evaluations.
• Strong GitHub experience: repos, branching, PRs, code reviews, CI/CD workflows.
• Experience with Claude Code or similar AI‑assisted development tools.
• Experience with AWS and Amazon Bedrock.
• Experience with REST APIs, JSON, SQL, and modern app architectures.
• Strong understanding of LLMs, prompt engineering, embeddings, vector search, RAG, and AI agents.
• Experience with automated, regression, integration, and end-to-end testing.
• Hands‑on experience evaluating LangGraph‑based or multi‑agent apps at node/state‑transition level.
• Excellent analytical, problem‑solving, and communication skills.
• Ability to own ambiguous technical problems independently.

Responsibilities

• Architect Evaluation Frameworks: design, implement, and scale automated evaluation pipelines for multi‑agent LLM systems.
• Define Domain‑Specific Metrics: develop advanced metrics with G‑Eval and LLM‑as‑a‑judge concepts.
• Establish Guardrails & Safety Protocols: implement defensive alignment testing and prompt injection mitigation.
• Build the Benchmarking Infrastructure: create robust datasets, golden regression sets, and synthetic data pipelines.
• Monitor System Drift & Observability: track production‑grade agent workflows for drift and latency.
• Build Automated Evaluation Infrastructure: develop scalable frameworks using Python, SQL, APIs, and CI/CD for hundreds/thousands of cases.
• Develop Evaluation Metrics & Graders: implement automated grading using deterministic validation and human‑in‑the‑loop methods.

Skills

Python

AI evaluation

LLM testing

CI/CD

LangChain

LangSmith

LangGraph

GitHub Actions

RAG

Adversarial testing

Data analysis

Tools

LangChain

LangSmith

LangGraph

Claude_Code

AWS_Bedrock

GitHub

CI/CD tooling

Job description

Helpware Inc. seeks a Senior/Staff Evaluation AI Engineer to anchor benchmarking, validation, and safety for our Agentic AI platform used in enterprise real estate workflows.

You will design automated evaluation pipelines, define domain metrics, and partner with Engineering, Product, Data, and Quality teams to ensure measurable, reliable AI capabilities. This 6-month engagement with potential prolongation focuses on defensive alignment testing, tool integration, and scalable regression suites