AI Reliability & Evaluation Engineer

Techventus; United States; Remote; USD 8,900 - 18,000; Full time

Benefits offered by this job

Remote-first working arrangements

Opportunity to specialize in AIReliabl

Exposure to production AI systems

Performance reviews

Job summary

Techventus is seeking an AI Reliability & Evaluation Engineer for full remote work. You will design evaluation frameworks for LLM applications, create test datasets, measure latency and cost, and implement monitoring and recovery procedures to ensure production reliability.

The role requires 2 years of QA automation or backend engineering, a BS in CS/SE/Data Science, and familiarity with AI evaluation tools. Remote-first environment with ongoing professional development.

Qualifications

• Demonstrated experience with automated testing or AI evaluation.
• Strong analytical and documentation skills.
• Familiarity with monitoring/tracing tools preferred.

Responsibilities

• Develop evaluation frameworks for LLM applications and AI agents.
• Create test datasets and automated regression tests for AI outputs.
• Measure response quality, retrieval accuracy, latency, and operating cost.
• Test tool calling, workflow execution, and integration failure scenarios.
• Implement monitoring, tracing, logging, and performance dashboards.
• Identify hallucinations, prompt-injection risks, and unsafe model behavior.
• Design fallback mechanisms, human approval steps, and recovery procedures.
• Collaborate with application engineers to improve production reliability.
• Document evaluation results and communicate risks to technical stakeholders.

Skills

Python & testing

LLM prompts & RAG

API testing & automation

LLM evaluation metrics

Logging & monitoring

Security & data protection

Reproducible tests & analysis

Education

Bachelor's degree in Computer Science, Software Engineering, Data Science, or related field

QA automation or backend engineering experience

Tools

LangSmith

OpenTelemetry

Monitoring tooling

Job description

Techventus is looking for an AI Reliability & Evaluation Engineer to help make AI applications dependable, measurable, secure, and suitable for production environments. The role focuses on systematic evaluation, testing, observability, error handling, and controls for AI-powered applications and agents.

No. of Vacancies

2

Specific Skills

• Python and automated testing.
• Understanding of LLM applications, prompts, and RAG architectures.
• API testing and test automation.
• Familiarity with LLM evaluation methods and quality metrics.
• Logging, tracing, monitoring, and debugging.
• Understanding of application security and data protection.
• Ability to design reproducible tests and analyze results.

Responsible For

• Develop evaluation frameworks for LLM applications and AI agents.
• Create test datasets and automated regression tests for AI outputs.
• Measure response quality, retrieval accuracy, latency, and operating cost.
• Test tool calling, workflow execution, and integration failure scenarios.
• Implement monitoring, tracing, logging, and performance dashboards.
• Identify hallucinations, prompt-injection risks, and unsafe model behavior.
• Design fallback mechanisms, human approval steps, and recovery procedures.
• Collaborate with application engineers to improve production reliability.
• Document evaluation results and communicate risks to technical stakeholders.

Additional Requirements

• Demonstrable experience with automated testing or AI evaluation.
• Strong analytical and documentation skills.
• Familiarity with tools such as LangSmith, OpenTelemetry, or equivalent monitoring and tracing systems is preferred.
• Understanding of human-in-the-loop workflows and AI governance is an advantage.
• Ability to explain technical risks and recommend practical mitigations.

Job Nature

Full Time

Educational Requirements

• Bachelor's degree in Computer Science, Software Engineering, Data Science, or a related field.
• Relevant experience in QA automation or backend engineering is acceptable.

Experience Requirements

2 Years

Job Location

Full Remote

PKR 220,000–450,000 monthly, depending on experience and specialist knowledge.

Other Benefits

• Remote-first working arrangements.
• Opportunity to specialize in AI reliability engineering.
• Exposure to production AI systems and enterprise deployments.
• Professional development and performance reviews, subject to company policy.

Job Level

Sr. Position