Principal Data Scientist - Agent Builder

United States Digital Space LLC; Sweden; Hybrid; SEK 831,700 - 1,315,600; Full time

Benefits offered by this job

Health coverage for you and your family

Flexible locations and schedules

Generous vacation days

Parental leave of 16 weeks

Job summary

United States Digital Space LLC is looking for a Principal Data Scientist to lead the technical direction in evaluating and improving chat quality for their agentic platform. This role combines applied leadership with strong engineering collaboration.

The ideal candidate will have 8+ years of experience with a focus on conversational AI, a solid understanding of AI/ML systems, and the ability to mentor others. The role offers competitive compensation and numerous employee benefits.

Qualifications

• 8+ years of applied DS/ML experience.
• Hands-on ability with Python and PyTorch.
• Strong understanding of retrieval systems.

Responsibilities

• Define evaluation strategy for conversational AI.
• Lead design of quality metrics and frameworks.
• Productionize evaluation pipelines with engineering.

Skills

Applied DS/ML experience

Expertise in IR, NLP, ranking

Python

Strong communication

Hands-on experience with PyTorch

Experience with retrieval systems

Tools

Elasticsearch

Pandas

Job description

the company, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale — unleashing the potential of businesses and people. The Search AI Platform, used by more than 50% of the Fortune 500, brings together the precision of search and the intelligence of AI to enable everyone to accelerate the results that matter. By taking advantage of all structured and unstructured data — securing and protecting private information more effectively — the company’s complete, cloud-based solutions for search, security, and observability help organizations deliver on the promise of AI.

What is The Role

The Search Conversational Experiences team builds the company’s new conversational and agentic platform that lets customers chat with their own data in Elasticsearch. We build the core quality layer for RAG, agents and tools, retrieval and citations, streaming, memory, and the evaluation signals that turn open-ended questions into grounded, reliable answers.

As a Principal Data Scientist, you will help set the technical direction for how we evaluate, improve, and scale chat quality across the company’s agentic platform. You will define the evaluation strategy that guides product decisions, including which models we standardize on, how we route requests across agents, which tools we enable and when, and how we tailor agents to different company use cases in search and beyond. You will work closely with backend engineering, product, UX, and other data scientists to turn ambiguous, cutting‑edge problems into measurable product improvements.

You’ll help lead work on frontier problems such as folding RAG and vector search into an agent’s knowledge base, dynamically enriching model context to improve groundedness, shaping reasoning strategies and tool‑selection policies, lighting up agent‑driven visualizations on top of Elasticsearch data, and exploring multimodality where it can create meaningful user value. This is an applied leadership role: you will prototype, evaluate, influence roadmap direction, and help teams ship improvements that customers can feel.

What You Will Be Doing

• Define the evaluation strategy for conversational and agentic search, including offline and online evaluation, golden datasets, rubrics, LLM‑as‑judge calibration, groundedness and citation checks, and A/B testing.
• Lead the design of quality metrics and decision frameworks for RAG, agents, tools, model selection, agent routing, prompt behavior, and cost/latency trade‑offs.
• Build, compare, and guide improvements across retrieval and re‑ranking approaches, including sparse and dense retrieval, vector search, query understanding, semantic rewrites, and context enrichment.
• Turn experimental results into product and business decisions: which models to use, how to route requests efficiently, which tools should be exposed, and how agents should be customized for different company use cases.
• Partner with engineering to productionize evaluation pipelines, telemetry, dashboards, CI guardrails, and regression detection for chat quality, helpfulness, dedication, latency, and cost.
• Influence the roadmap by identifying the highest‑leverage quality gaps, proposing practical solutions, and communicating trade‑offs clearly to product, engineering, and leadership.
• Mentor other data scientists and engineers in experiment design, evaluation methodology, statistical rigor, and practical approaches to improving LLM‑powered systems.
• Share outcomes through clear docs, notebooks, PRs, dashboards, technical proposals, and cross‑functional reviews.

What You Bring

• 8+ years of applied DS/ML experience, with deep expertise in IR, NLP, ranking, semantic search, RAG, or LLM‑powered product experiences.
• Strong track record defining and leading evaluation for production AI/ML systems, including offline metrics, online experimentation, LLM‑as‑judge approaches, groundedness, citation quality, and model comparison.
• Experience influencing product and technical strategy through data, especially in ambiguous or emerging domains where the “right” metric or approach is not obvious at the start.
• Hands‑on ability with Python, PyTorch/Transformers, Pandas, notebooks, reproducible experiments, versioned datasets, and clean, reviewable code.
• Strong understanding of retrieval systems, including dense and sparse retrieval, re‑ranking, vector search, query understanding, and evaluation metrics such as nDCG, MRR, Recall@k, precision, and latency/cost trade‑offs.
• Experience collaborating closely with engineering teams to move from prototype to production, including telemetry design, dashboards, CI guardrails, and quality regression tracking.
• Practical Elasticsearch experience, or experience with similar search and distributed data systems. ES|QL familiarity is a plus.
• Excellent written and verbal communication, with the ability to explain complex scientific and technical trade‑offs to engineering, product, design, and leadership audiences.
• A collaborative, low‑ego style and a strong ability to mentor, raise standards, and develop transparency for others in a distributed team.

Compensation for this role is in the form of base salary. This role does not have a variable compensation component.

At the company, our compensation philosophy aims to provide fair, competitive and transparent remuneration. Salary ranges are established based on a combination of external market benchmarks, internal pay equity considerations, and the responsibilities and complexity associated with each role. This approach helps ensure consistency across comparable roles while remaining competitive within the relevant labour markets.

The final compensation offered within the applicable range will be determined based on several objective factors, including relevant professional experience, level of skills and expertise, alignment with the role requirements, and the overall scope and complexity of the position.

The typical starting salary range for this role is: 831 700 kr — 1 315 600 kr SEK

Additional Information - We Take Care of Our People

We strive to have parity of benefits across regions and while regulations differ from place to place, we believe taking care of our people is the right thing to do.

• Competitive pay based on the work you do here and not your previous salary
• Health coverage for you and your family in many locations
• Ability to craft your calendar with flexible locations and schedules for many roles
• Generous number of vacation days each year
• Increase your impact — We match up to $2000 (or local currency equivalent) for financial donations and service
• Up to 40 hours each year to use toward volunteer projects you love
• Embracing parenthood with minimum of 16 weeks of parental leave

The company is an equal opportunity employer and is committed to creating an inclusive culture that celebrates different perspectives, experiences, and backgrounds.