LLM Observability & Evaluation Engineer
KTek Resourcing LLC
Get a reply from this employer
Your resume and cover letter, tailored to exactly what they’re hiring for.
Job summary
KTek Resourcing LLC is seeking a GenAI / LLM Observability & Evaluation Engineer to ensure reliability and performance of production AI agents and LLM-powered apps. You will implement evaluation frameworks, tracing, telemetry, and dashboards to monitor production systems, driving incident readiness and operational excellence.
The role requires hands-on experience with LLMs, evaluation tools, Python automation, and strong observability skills to optimize performance and reliability in a
Qualifications
- Hands-on work with LLMs, Generative AI, or Agentic AI.
- Experience evaluating LLMs, testing agents, and analyzing performance.
- Strong Python and test automation skills.
- Experience with observability, tracing, telemetry and monitoring.
- Ability to build dashboards and analyze application performance.
- Knowledge of SLOs, production operations, incident response, and reliability.
Responsibilities
- Implement LLM evaluation frameworks and agent testing workflows.
- Design and operate observability, tracing, and telemetry solutions.
- Build and maintain dashboards to monitor production AI agents.
- Collaborate with teams to improve reliability and release quality.
Skills
Tools
Job description
KTek Resourcing LLC is seeking a GenAI / LLM Observability & Evaluation Engineer to ensure reliability and performance of production AI agents and LLM-powered apps. You will implement evaluation frameworks, tracing, telemetry, and dashboards to monitor production systems, driving incident readiness and operational excellence.
The role requires hands-on experience with LLMs, evaluation tools, Python automation, and strong observability skills to optimize performance and reliability in a
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
