AI Reliability & Evaluation Engineer — Remote

Techventus; United States; Remote; USD 8,900 - 18,000; Full time

Benefits offered by this job

Remote-first working arrangements

Opportunity to specialize in AIReliabl

Exposure to production AI systems

Performance reviews

Job summary

Techventus is seeking an AI Reliability & Evaluation Engineer for full remote work. You will design evaluation frameworks for LLM applications, create test datasets, measure latency and cost, and implement monitoring and recovery procedures to ensure production reliability.

The role requires 2 years of QA automation or backend engineering, a BS in CS/SE/Data Science, and familiarity with AI evaluation tools. Remote-first environment with ongoing professional development.

Qualifications

• Demonstrated experience with automated testing or AI evaluation.
• Strong analytical and documentation skills.
• Familiarity with monitoring/tracing tools preferred.

Responsibilities

• Develop evaluation frameworks for LLM applications and AI agents.
• Create test datasets and automated regression tests for AI outputs.
• Measure response quality, retrieval accuracy, latency, and operating cost.
• Test tool calling, workflow execution, and integration failure scenarios.
• Implement monitoring, tracing, logging, and performance dashboards.
• Identify hallucinations, prompt-injection risks, and unsafe model behavior.
• Design fallback mechanisms, human approval steps, and recovery procedures.
• Collaborate with application engineers to improve production reliability.
• Document evaluation results and communicate risks to technical stakeholders.

Skills

Python & testing

LLM prompts & RAG

API testing & automation

LLM evaluation metrics

Logging & monitoring

Security & data protection

Reproducible tests & analysis

Education

Bachelor's degree in Computer Science, Software Engineering, Data Science, or related field

QA automation or backend engineering experience

Tools

LangSmith

OpenTelemetry

Monitoring tooling

Job description

Techventus is seeking an AI Reliability & Evaluation Engineer for full remote work. You will design evaluation frameworks for LLM applications, create test datasets, measure latency and cost, and implement monitoring and recovery procedures to ensure production reliability.

The role requires 2 years of QA automation or backend engineering, a BS in CS/SE/Data Science, and familiarity with AI evaluation tools. Remote-first environment with ongoing professional development.