Technical Lead

SmartDev; gmina Osie; On-site; PLN 240,000 - 420,000; Full time

Job summary

SmartDev seeks a founding technical lead for Veris EvalOps, building a platform answering whether a system is safe to launch and if it remains correct after deployment. You’ll take it from first line of code to first paying clients in 6–7 months.

You will architect two commercial modules: a pre‑production Release Gate and a Knowledge Health Monitor, while hiring and leading engineers across three streams.

Qualifications

• 5+ years in engineering, including 2+ years leading a team of 3–8 through a full build cycle.
• LLM evaluation methodology including LLM‑as‑judge design, RAGAS/DeepEval metrics, and regression testing.
• LLM observability and tracing with OpenTelemetry across LLM calls and multi‑step trajectories.
• RAG system architecture with vector stores and retrieval, re‑ranking.
• AI agent systems using patterns like ReAct, Plan‑and‑Execute, guardrails.
• Backend platform engineering with production‑grade async Python and multi‑tenant SaaS.
• Ability to weigh integrating external tooling vs building from scratch during onboarding.

Responsibilities

• Own the evaluation engine: scoring, groundedness checks, hallucination detection, and regression comparison.
• Build tracing and observability for LLM calls, RAG retrievals, and agent workflows.
• Ship the knowledge health pipeline: ingestion, stale content detection, and coverage mapping.
• Own platform core and integrations: multi‑tenant architecture, RBAC, API connectors, dashboards, and CI/CD hooks.
• Build and lead the team: hire and run 5–7 engineers across three streams and own architecture decisions end‑to‑end.

Skills

Engineering leadership

LLM evaluation

LLM observability

RAG system architecture

AI agent systems

Backend platform engineering

Multi-tenant SaaS

Python async (FastAPI)

CI/CD integration

Client-facing role

Tools

OpenTelemetry

Pinecone

Weaviate

Qdrant

pgvector

FastAPI

Celery

PostgreSQL

Redis

Job description

Company Description

SmartDev is an AI‑powered software development company headquartered in Vietnam, part of the Verysell Group. We help global businesses deliver faster and build smarter — combining AI‑driven development practices with deep expertise across fintech, healthcare, retail, and enterprise technology.

Our team of engineers, architects, and AI specialists works across the full stack: from custom software and cloud solutions to generative AI, MLOps, and intelligent automation. At SmartDev, we believe that great architecture and AI‑first thinking are a competitive advantage — and we build our teams accordingly.

The company is at a pivotal point in our journey: transitioning from a pure IT outsourcing provider to an AI‑enabled solutions partner.

We have over 200 talented employees working both on‑site and remote/hybrid at locations:

• Danang: 81 Quang Trung, Hai Chau 1 ward, Hai Chau District, Danang
• Hanoi: Zodiac building, 19 Duy Tan Street, Dich Vong Hau ward, Cau Giay District, Hanoi

Job Description

You’ll be the founding technical lead for Veris EvalOps, building the platform that answers the two questions every AI‑deploying business needs answered: is this system safe to launch, and is it still working correctly a month later. You’ll take it from first line of code to first paying clients in 6–7 months.

Role Summary

This is a zero-to-one build, not a maintenance role. You’ll architect and ship two commercial modules — a pre‑production Release Gate that turns “looks good” into a reproducible readiness score, and a Knowledge Health Monitor that continuously audits the knowledge base an AI draws from — while hiring and leading the engineers who build them alongside you. There’s no principal architect above you to escalation to: you make the calls and live with them, with the first pilot client live by month 3.

Key Responsibilities

• Own the evaluation engine. LLM‑as‑judge scoring, rule‑based checks, groundedness verification, hallucination detection, and regression comparison — every readiness score comes from here.
• Build the tracing and observability layer. Distributed tracing across LLM calls, RAG retrievals, and agent workflows, built on OpenTelemetry, capturing every token, tool call, cost, and latency metric.
• Ship the knowledge health pipeline. Ingestion and continuous analysis of enterprise knowledge sources — stale‑content detection, contradiction analysis, and coverage‑gap mapping.
• Own platform core and integrations. Multi‑tenant architecture, RBAC, API connectors, dashboards, and the CI/CD hooks that let the Release Gate plug into client engineering workflows.
• Build and lead the team. Hire and run 5–7 engineers across three streams — Platform Core, Release Gate, Knowledge Health — and own every architecture decision end to end.

Qualifications

Must‑have

• 5+ years in engineering. Including 2+ years leading a team of 3–8 through a complete build cycle — architecture to shipping to paying users. Not a first‑time lead role.
• LLM evaluation methodology. LLM‑as‑judge design, RAGAS/DeepEval‑style metrics, golden dataset construction, regression testing for AI systems, and hallucination detection — not just the library calls, the mechanics behind them.
• LLM observability and tracing. OpenTelemetry‑based tracing across LLM calls, RAG retrievals, and multi‑step agent trajectories; cost/latency attribution; drift and anomaly detection.
• RAG system architecture. Production experience across the full pipeline — chunking, embeddings, a vector store (Pinecone, Weaviate, Qdrant, or pgvector), retrieval, re‑ranking.
• AI agent systems. Production experience with agent patterns (ReAct, Plan‑and‑Execute, supervisor/sub‑agent), tool‑call evaluation, and guardrails.
• Backend platform engineering. Production‑grade async Python (FastAPI, Celery), multi‑tenant SaaS architecture, PostgreSQL/Redis, and CI/CD integration.
• Build‑vs‑integrate judgment and client‑facing comfort. Can weigh integrating Langfuse/Braintrust vs. building from scratch, and work directly with pilot clients during onboarding and results review.

Nice‑to‑have

• LLM APIs and model ecosystem. Multi‑provider experience (OpenAI, Anthropic, Azure OpenAI, Bedrock) and routing/prompt‑management at scale.
• MLOps and experiment tracking. Background with MLflow, Weights & Biases, or equivalent experiment‑tracking tooling.
• Security, compliance, and AI governance. EU AI Act and NIST AI RMF awareness, PII handling in AI pipelines, and red‑teaming basics — increasingly a qualification question in enterprise security reviews.