Technical Lead

Smartdev1; Poland; On-site; PLN 320,000 - 520,000; Full time

Job summary

Veris is seeking a founding technical lead to architect and ship two commercial modules: a Release Gate readiness engine and a Knowledge Health Monitor. You will lead 5–7 engineers across three streams, own architecture decisions, and deliver the first pilot client by month 3.

This role requires 5+ years in engineering with leadership experience, expertise in LLM evaluation, observability, and multi-tenant backend platforms.

Qualifications

• 5+ years in engineering with 2+ years leading a team through a full build cycle.
• Experience designing and shipping production ML/AI evaluation systems.
• Hands-on with observability, tracing, and cost/latency attribution.
• Strong system design for multi-tenant SaaS platforms.
• Experience with enterprise knowledge sources and AI governance is a plus.

Responsibilities

• Own the evaluation engine: LLM-as-judge, rules, and regression checks to generate a readiness score.
• Build tracing and observability across LLM calls, retrievals, and agent workflows.
• Develop the knowledge health pipeline: ingestion, analysis, and gap detection.
• Own platform core and integrations: multi-tenant architecture, RBAC, APIs, dashboards, and CI/CD hooks.
• Recruit and lead 5–7 engineers across three streams and drive architecture decisions end to end.

Skills

Engineering leadership

LLM evaluation methodology

LLM observability

RAG system architecture

AI agent systems

Backend platform engineering

Build-vs-integrate judgment

OpenTelemetry

FastAPI

PostgreSQL

Redis

CI/CD integration

Tools

OpenTelemetry

FastAPI

PostgreSQL

Redis

CI/CD

Job description

You'll be the founding technical lead for Veris EvalOps, building the platform that answers the two questions every AI-deploying business needs answered: is this system safe to launch, and is it still working correctly a month later. You'll take it from first line of code to first paying clients in 6-7 months.

Role Summary

This is a zero-to-one build, not a maintenance role. You'll architect and ship two commercial modules - a pre-production Release Gate that turns "looks good" into a reproducible readiness score, and a Knowledge Health Monitor that continuously audits the knowledge base an AI draws from - while hiring and leading the engineers who build them alongside you. There's no principal architect above you to upscale to: you make the calls and live with them, with the first pilot client live by month 3.

Key Responsibilities

• Own the evaluation engine. LLM-as-judge scoring, rule-based checks, groundedness verification, hallucination detection, and regression comparison - every readiness score comes from here.
• Build the tracing and observability layer. Distributed tracing across LLM calls, RAG retrievals, and agent workflows, built on OpenTelemetry, capturing every token, tool call, cost, and latency metric.
• Ship the knowledge health pipeline. Ingestion and continuous analysis of enterprise knowledge sources - stale-content detection, contradiction analysis, and coverage-gap mapping.
• Own platform core and integrations. Multi-tenant architecture, RBAC, API connectors, dashboards, and the CI/CD hooks that let the Release Gate plug into client engineering workflows.
• Build and lead the team. Hire and run 5-7 engineers across three streams - Platform Core, Release Gate, Knowledge Health - and own every architecture decision end to end.

Must-have

• 5+ years in engineering. Including 2+ years leading a team of 3-8 through a complete build cycle - architecture to shipping to paying users. Not a first-time lead role.
• LLM evaluation methodology. LLM-as-judge design, RAGAS/DeepEval-style metrics, golden dataset construction, regression testing for AI systems, and hallucination detection - not just the library calls, the mechanics behind them.
• LLM observability and tracing. OpenTelemetry-based tracing across LLM calls, RAG retrievals, and multi-step agent trajectories; cost/latency attribution; drift and anomaly detection.
• RAG system architecture. Production experience across the full pipeline - chunking, embeddings, a vector store (Pinecone, Weaviate, Qdrant, or pgvector), retrieval, re-ranking.
• AI agent systems. Production experience with agent patterns (ReAct, Plan-and-Execute, supervisor/sub-agent), tool-call evaluation, and guardrails.
• Backend platform engineering. Production-grade async Python (FastAPI, Celery), multi-tenant SaaS architecture, PostgreSQL/Redis, and CI/CD integration.
• Build-vs-integrate judgment and client-facing comfort. Can weigh integrating Langfuse/Braintrust vs. building from scratch, and work directly with pilot clients during onboarding and results review.

Nice-to-have

• LLM APIs and model ecosystem. Multi-provider experience (OpenAI, Anthropic, Azure OpenAI, Bedrock) and routing/prompt-management at scale.
• MLOps and experiment tracking. Background with MLflow, Weights & Biases, or equivalent experiment-tracking tooling.
• Security, compliance, and AI governance. EU AI Act and NIST AI RMF awareness, PII handling in AI pipelines, and red-teaming basics - increasingly a qualification question in enterprise security reviews.