Production Insurance AI Evaluation Specialist

Jda Tsg; United States; Remote; USD 75,000 - 90,000; Full time

Benefits offered by this job

Healthcare coverage

Employee Assistance Program

401K with company match

Paid time off

Paid parental leave

Volunteer Day Off

Life insurance

Business travel accident insurance

Job summary

JDA TSG seeks a Production Insurance AI Evaluation Specialist to audit live AI outputs against source records across commercial lines workstreams. You’ll verify submissions, ACORD forms, quotes, and policy documents, ensuring accurate premium allocations and data integrity.

You will escalate truly ambiguous cases, contribute to weekly calibration sessions, and deliver consistent weekly workbooks while meeting SLA expectations, helping refine models and improve policy accuracy.

Qualifications

• Active Property & Casualty license in good standing.
• 2+ years of commercial insurance brokerage/agency operations.
• Experience analyzing complex email threads, quotes, ACORD forms and policy schedules.
• Ability to compare unstructured documents with JSON extraction workbooks.

Responsibilities

• Audit submissions and AI extractions across emails, forms, quotes, and policy documents to verify premiums and policy status.
• Flag ambiguous edge cases for escalation and contribute to weekly calibration with workstream owners.
• Maintain weekly workbook delivery and ensure SLA compliance.

Skills

Active P&C License

Commercial Lines Expertise

Submission & Email Thread Parsing

Analytical Precision

Tools

Applied Epic

EZLynx

Carrier portals

Job description

Overview

At JDA TSG, we equip world-class brands with top-tier talent and operational expertise, because we know our people are our true differentiator. As a Production Insurance AI Evaluation Specialist, you will run an expert quality review function for active AI agents performing commercial insurance tasks. We believe AI operational success relies on domain expertise. By auditing live AI outputs against source records, catching discrepancies, and reporting error patterns, your work protects policy accuracy, refines machine learning models, and ensures reliable performance across enterprise insurance workflows.

In this role, you will establish trusted ground-truth accuracy baselines across two primary Commercial lines workstreams prior to and following General Availability (GA) releases: Submissions Manager (auditing email threads and multi-line submission attachments) and Commercial Lines Autofill / Risk Checker (auditing form schemas and quote extractions)

Title: Production Insurance AI Evaluation Specialist

Location: Remote, USA

Salary: $75,000 to $90,000 USD

What You Will Do

• Ground-Truth Queue Auditing & Discrepancy Isolation: You will review assigned submissions and AI extractions across multi-file email threads, ACORD forms, quote schedules, and attached policy documents. You will audit AI outputs against target schemas to verify total submission premium, line-of-business (LOB) premium allocations, LOB selections, email thread attribution, extracted schema fields, policy status, and decline reasons, catching both missing and incorrect values across commercial forms and binders. Success is measured by audit completion volume, extraction accuracy rates, and error-catching precision.
• Disciplined Escalation & Pattern Calibration: You will strictly adhere to client quality standards by flagging genuinely ambiguous edge cases for workstream owner escalation rather than guessing, preserving ground-truth baseline integrity. You will aggregate extraction errors into structured weekly workbooks and participate in peer calibration sessions with workstream owners to align on reference-label standards and agreement targets. Success is measured by baseline data integrity, workbook accuracy, and calibration alignment across workstreams.
• SLA & Weekly Workbook Execution: Maintain a consistent review pace while meeting established productivity expectations and delivering verified weekly workbooks on schedule. Success is measured by review throughput, SLA compliance, accuracy, and timely workbook delivery.

Who You Are

• Active P&C License (Mandatory): Active Property & Casualty (P&C) Producer / Agent License in good standing.
• Commercial Lines Expertise: 2+ years of recent operational experience in commercial insurance brokerage, agency, MGA, or carrier operations. Direct experience with Commercial General Liability, Commercial Property, Commercial Auto, Workers' Compensation, and Umbrella/Excess lines is required (personal-lines-only backgrounds will not be considered).
• Submission & Email Thread Parsing: Proven ability to analyze complex agency email threads, multi-line carrier quotes, ACORD forms, and policy schedules.
• Analytical Precision & Systems Familiarity: Exceptional visual rigor in comparing unstructured documents against JSON extraction workbooks to isolate missing vs. incorrect values. Practical experience using agency management systems such as Applied Epic, EZLynx, or carrier management portals.

What We Offer

• Healthcare- Comprehensive coverage for you and your family
• Employee Assistance Program - Get support when you or your family need it with counseling and coaching
• 401K with company match
• Paid time off
• Paid parental leave
• Volunteer Day Off
• Life insurance - Protect your loved ones and their future
• Business travel accident insurance

The specific pay rate offered for this role depends on candidate qualifications, relevant experience, technical expertise, and role requirements. Final individual offers are determined through candidate evaluations during the interview process

USD $75,000.00 - USD $90,000.00 /Yr.

We are an equal opportunity employer committed to building an inclusive workplace. To view our Equal Employment Opportunity (EEO) policy, please click here.