QA Analyst - LOIS for Word

Filevine

San Francisco (CA)

On-site

USD 105,000 - 150,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Medical, Dental & Vision insurance
Maternity & paternity leave
Short & long-term disability
Company swag

Job summary

Filevine, located in San Francisco, is seeking a Quality Assurance (QA) professional to lead the QA function for their AI‑powered legal drafting product. This role involves defining the test strategy, evaluating AI output, and ensuring quality is integrated into features from the planning stage.

Candidates should have a minimum of 2 years of QA experience and a track record in building QA processes in fast-moving environments. Benefits include competitive pay and comprehensive insurance plans.

Qualifications

  • 2+ years of QA experience, owning test strategy for a product.
  • Proven track record of building QA process and automation.
  • Ability to operate in ambiguous environments.

Responsibilities

  • Define and own the test strategy and QA process.
  • Evaluate AI output quality across multiple dimensions.
  • Design and maintain evaluation sets for AI outputs.

Skills

QA process ownership
Automation skills
Analytical skills
Bug tracking tools

Job description

Role Summary

This role owns quality for LOIS, Filevine’s AI‑powered legal drafting product. You’ll help build and own the QA function: the test strategy, the process, and the automation. You’ll work at the intersection of legal AI and product quality, evaluating everything from AI output accuracy to UI behavior across the product. A big part of this role is moving quality upstream, so it’s built into how features are planned and shipped from the start.

What You’ll Do
  • Define and own the test strategy, process, and QA playbook for the product
  • Move QA upstream by partnering with product and engineering early, so test plans are ready before features ship
  • Evaluate AI output quality across dimensions like accuracy, citation reliability, and appropriately scoped reasoning
  • Design and maintain evaluation sets that account for non‑deterministic output and catch regressions over time
  • Test how AI‑generated edits apply to documents, including accuracy and formatting fidelity across document structures
  • Build automated guardrails integrated into CI/CD that cover critical flows
  • Write structured, reproducible bug reports
  • Partner with legal domain experts to understand how different customer types use the product, and translate that into test coverage
  • Approach QA with a product and design mindset. If something works but could be better, we want to hear it
  • Compare behavior across product surfaces and document discrepancies
What You’ll Need
  • 2+ years of QA experience, including ownership of test strategy for a product or major feature area
  • Track record of building QA process and automation from scratch in a fast‑moving environment
  • Comfort operating in high ambiguity, where features ship quickly and you figure out how to test them fast
  • Strong analytical and problem‑solving skills, with attention to both functional correctness and AI output quality
  • Ability to reason about non‑deterministic systems and define what "correct" means when there’s no single right answer
  • Excellent written communication, with the ability to write clear, concise, actionable bug reports backed by evidence
  • Experience with bug tracking tools and test management workflows
Nice To Have
  • Experience testing AI or LLM‑based products, especially for output accuracy and eval design
  • CI/CD experience setting up pipelines and release gates for automated tests
  • A design or UX sensibility, with an eye for where product workflows could be smoother
  • Prior experience in legal technology or comfort reading legal contracts
  • Programming experience beyond test automation
Compensation Information

$105,000 - $150,000. The salary range represents the low and high end of the salary range for this position. The total compensation package for this position will be determined by each individual’s location, qualifications, education, work experience, skills and performance. An individual’s final offer will be weighted by their primary work location and expertise.

Benefits
  • A dynamic, rapidly growing company, focused on helping organizations thrive
  • Medical, Dental & Vision insurance for full‑time employees
  • Competitive & Fair Pay
  • Maternity & paternity leave for full‑time employees
  • Short & long‑term disability
  • Opportunity to learn from a dedicated leadership team
  • Top‑of‑the‑line company swag
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Quality Assurance Analyst
Quality Assurance Analyst

RecruiterForge • United States

Remote
USD 105,000 - 150,000
Medical insurance
Dental insurance
Vision insurance
+5
Remote QA Analyst for AI Legal Drafting
Remote QA Analyst for AI Legal Drafting

RecruiterForge • United States

Remote
USD 105,000 - 150,000
Medical insurance
Dental insurance
Vision insurance
+5
QA Architect for AI-Powered Legal Drafting (LOIS)
QA Architect for AI-Powered Legal Drafting (LOIS)

Filevine • San Francisco (CA)

On-site
USD 105,000 - 150,000
Medical, Dental & Vision insurance
Maternity & paternity leave
Short & long-term disability
+1
Quality Assurance Analyst II
Quality Assurance Analyst II

Filevine • New York (NY)

Remote
USD 120,000 - 140,000
Medical, Dental, & Vision Insurance
Competitive & Fair Pay
Maternity & paternity leave
+3
QA Engineer
QA Engineer

Storm3 • New York (NY)

On-site
USD 160,000 - 300,000
Visa sponsorship available
Relocation support
Unlimited PTO
+3
Senior QA Engineer — Remote Testing & Automation
Senior QA Engineer — Remote Testing & Automation

Filevine • New York (NY)

Remote
USD 120,000 - 140,000
Medical, Dental, & Vision Insurance
Competitive & Fair Pay
Maternity & paternity leave
+3
AI Solutions QA Analyst
AI Solutions QA Analyst

McDermott • United States

Remote
USD 93,000 - 142,000
Discretionary bonus
Competitive benefits package
Manager, Software Quality Assurance
Manager, Software Quality Assurance

Wolters Kluwer • New York (NY)

On-site
USD 108,000 - 188,000
Medical, Dental, & Vision Plans
401(k)
FSA/HSA
+4
QA Engineer
QA Engineer

Neuroscale AI • United States

Hybrid
USD 90,000 - 115,000
Medical, dental, and vision benefits
Professional growth opportunities
Remote-first / hybrid flexibility
AI Solutions QA Analyst
AI Solutions QA Analyst

McDermott Will & Schulte • United States

Remote
USD 90,000 - 130,000
Remote-first