Founding Technical Lead for AI EvalOps

SmartDev; gmina Osie; On-site; PLN 240,000 - 420,000; Full time

Job summary

SmartDev seeks a founding technical lead for Veris EvalOps, building a platform answering whether a system is safe to launch and if it remains correct after deployment. You’ll take it from first line of code to first paying clients in 6–7 months.

You will architect two commercial modules: a pre‑production Release Gate and a Knowledge Health Monitor, while hiring and leading engineers across three streams.

Qualifications

• 5+ years in engineering, including 2+ years leading a team of 3–8 through a full build cycle.
• LLM evaluation methodology including LLM‑as‑judge design, RAGAS/DeepEval metrics, and regression testing.
• LLM observability and tracing with OpenTelemetry across LLM calls and multi‑step trajectories.
• RAG system architecture with vector stores and retrieval, re‑ranking.
• AI agent systems using patterns like ReAct, Plan‑and‑Execute, guardrails.
• Backend platform engineering with production‑grade async Python and multi‑tenant SaaS.
• Ability to weigh integrating external tooling vs building from scratch during onboarding.

Responsibilities

• Own the evaluation engine: scoring, groundedness checks, hallucination detection, and regression comparison.
• Build tracing and observability for LLM calls, RAG retrievals, and agent workflows.
• Ship the knowledge health pipeline: ingestion, stale content detection, and coverage mapping.
• Own platform core and integrations: multi‑tenant architecture, RBAC, API connectors, dashboards, and CI/CD hooks.
• Build and lead the team: hire and run 5–7 engineers across three streams and own architecture decisions end‑to‑end.

Skills

Engineering leadership

LLM evaluation

LLM observability

RAG system architecture

AI agent systems

Backend platform engineering

Multi-tenant SaaS

Python async (FastAPI)

CI/CD integration

Client-facing role

Tools

OpenTelemetry

Pinecone

Weaviate

Qdrant

pgvector

FastAPI

Celery

PostgreSQL

Redis

Job description

SmartDev seeks a founding technical lead for Veris EvalOps, building a platform answering whether a system is safe to launch and if it remains correct after deployment. You’ll take it from first line of code to first paying clients in 6–7 months.

You will architect two commercial modules: a pre‑production Release Gate and a Knowledge Health Monitor, while hiring and leading engineers across three streams.