Senior AI Data Engineer

Veeva Systems; Portland (OR); On-site; USD 85,000 - 225,000; Full time

Benefits offered by this job

Medical, dental, and vision insurance

Flexible PTO

Retirement programs

Charitable giving program

Job summary

A leading SaaS company is seeking an AI Evaluation Specialist to ensure the reliability and safety of their AI systems. This role involves defining evaluation strategies, assessing LLM outputs, and developing automated evaluation pipelines. Candidates should have a strong background in data integrity, programming expertise in Python, and a bachelor's degree in a related field. This position offers a flexible working environment in Portland, Oregon with competitive compensation and benefits.

Qualifications

• Strong understanding of data quality principles.
• Demonstrated skill in advanced prompt engineering techniques.
• Proficiency in designing automated evaluation pipelines.
• Comfortable with debugging agentic systems.
• Experience integrating with CI/CD systems.

Responsibilities

• Define evaluation strategies for new AI Agents.
• Evaluate quality of LLM‑generated content against metrics.
• Design and generate high-quality test data sets.
• Develop and maintain automated evaluation pipelines.
• Perform root cause analysis on identified defects.

Skills

Data Integrity & Validation

Prompt Engineering

Automated Evaluation Implementation

Debugging Agentic Systems

Programming in Python

Education

Bachelor’s degree in Data Science or related field

Tools

Python

Pytest

Job description

Veeva Systems is a mission‑driven organization and pioneer in industry cloud, helping life sciences companies bring therapies to patients faster. As one of the fastest‑growing SaaS companies in history, we surpassed $2B in revenue in our last fiscal year with extensive growth potential ahead. At the heart of Veeva are our values: Do the Right Thing, Customer Success, Employee Success, and Speed. We're not just any public company – we made history in 2021 by becoming a public benefit corporation (PBC), legally bound to balancing the interests of customers, employees, society, and investors. As a Work Anywhere company, we support your flexibility to work from home or in the office, so you can thrive in your ideal environment. Join us in transforming the life sciences industry, committed to making a positive impact on its customers, employees, and communities.

The Role

This role is responsible for ensuring the reliability, accuracy, and safety of our Veeva AI Agents through rigorous evaluation and systematic validation methodologies. We're looking for experienced candidates with a meticulous, critical, and curious mindset, exceptional analytical and systematic problem‑solving capabilities, excellent technical communication skills, and the ability to learn application areas quickly.

What You’ll Do

• Evaluation Strategy & Planning: Define and establish comprehensive evaluation strategies for new AI Agents, prioritizing the integrity and coverage of test data sets to reflect real‑world usage and potential failure modes.
• LLM Output Integrity Assessment: Programmatically and manually evaluate the quality of LLM‑generated content against predefined metrics (factual accuracy, contextual relevance, coherence, and safety standards).
• Creating High‑Fidelity Datasets: Design, curate, and generate diverse, high‑quality test data sets, including challenging prompts and scenarios; evaluate LLM outputs to proactively identify system biases, unsafe content, hallucinations, and critical edge cases.
• Automation of Evaluation Pipelines: Develop, implement, and maintain scalable automated evaluations to ensure efficient, continuous validation of agent behavior and prevent regressions with new features and model updates.
• Root Cause Analysis: Understand model behaviors and assist in the trace and root‑cause analysis of identified defects or performance degradations.
• Reporting & Performance Metrics: Clearly document, track, and communicate performance metrics, validation results, and bug status to the broader development and product teams.

Requirements

• Data Integrity & Validation: Strong, specialized understanding of data quality principles, methods for validating datasets against bias, integrity concerns, and quality standards; ability to craft diverse and adversarial test data to uncover AI edge cases.
• Prompt Engineering & Model Expertise: Demonstrated skill in advanced prompt engineering techniques; deep knowledge of LLM common failure modes (hallucination, incoherence, jailbreaking).
• Automated Evaluation Implementation: Proficiency in designing and deploying automated evaluation pipelines; familiarity with quality metrics such as task success rate, semantic similarity, and sentiment analysis for output measurement.
• Debugging Agentic Systems: Comfortable with debugging agentic systems, including tracing and interpreting an agent’s internal reasoning, tool use, and action sequence.
• Programming & Frameworks: Proficiency in Python, Pytest, modern web automation tools; experience integrating pipelines with CI/CD systems.
• Bachelor’s degree in Data Science, Machine Learning, Computer Science, or related field with experience in Gen AI / LLMs.
• High work ethic—Veeva is a hard‑working company.
• High integrity and honesty—Veeva expects that from all employees.
• Applicants must have the unrestricted right to work in the United States or Canada; Veeva will not provide sponsorship at this time.

Learn More

• Engineer Perspective: 3 Reasons to Consider Veeva
• Engineering at Veeva

Perks & Benefits

• Medical, dental, vision, and basic life insurance
• Flexible PTO and company‑paid holidays
• Retirement programs
• 1% charitable giving program

Compensation

• Base pay: $85,000 - $225,000
• Salary range provided to comply with local regulations; actual salaries may vary based on experience and location. The position may also be eligible for variable bonus and/or stock bonus.

Veeva’s headquarters is located in the San Francisco Bay Area with offices in more than 15 countries around the world.

Veeva is an equal‑opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, sex, sexual orientation, gender identity or expression, religion, national origin or ancestry, age, disability, marital status, pregnancy, protected veteran status, protected genetic information, political affiliation, or any other characteristics protected by local laws, regulations, or ordinances. If you need assistance or accommodation due to a disability or special need when applying for a role or in our recruitment process, please contact us at [e-mail].