Applied AI Data Analyst: LLM Evaluation & Gold Datasets

Socket.dev; New York (NY); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Equity

Health coverage

Dental and vision coverage

Fertility/Family-forming support

Parental leave (12 weeks)

Unlimited PTO

401(k)

Relocation assistance

Workspace stipend

Team perks

Job summary

Confido, based in New York City, is seeking an Applied AI Data Analyst to bridge massive unstructured financial and operational data with our R&D team. You will own the evaluation platform of agentic forecasting pipelines and improve customer-facing insights for mission-critical decisions.

This role sits in the Applied AI Research department, with opportunities to advance to senior data science and AI/ML engineering roles. In-person in NYC with relocation support.

Qualifications

• Graduate degree in STEM fields (MS/PhD) required.
• 2–4 years of experience as a quantitative data analyst or AI evaluation professional.
• Proficiency in Python and hands-on work with AI/ML models.
• Strong SQL skills and enterprise data infra (PostgreSQL, Snowflake, dbt, AWS).
• Ability to translate unstructured document data into rigorous evaluation insights for engineers.

Responsibilities

• Analyze edge cases, model predictions, hallucination rates and extraction accuracy for LLM-driven document processing systems and agentic workflows.
• Root cause analysis of complex datasets to explain why an extraction or time-series model succeeded or failed.
• Define, curate, and maintain gold-standard evaluation datasets for automated backtesting of AI agents and demand forecasting models.
• Build dashboards, data transformations, and evaluation harnesses to monitor system health, model drift, and AI performance across release cycles.
• Collaborate with applied AI engineers, product managers and business operations managers to translate real-world problems into quantitative product insights.

Skills

Python

SQL

PostgreSQL

Snowflake

dbt

AWS

Analytical mindset

Education

Graduate degree in STEM (MS/PhD)

Tools

LangSmith

Arize AI

TruLens

Job description

Confido, based in New York City, is seeking an Applied AI Data Analyst to bridge massive unstructured financial and operational data with our R&D team. You will own the evaluation platform of agentic forecasting pipelines and improve customer-facing insights for mission-critical decisions.

This role sits in the Applied AI Research department, with opportunities to advance to senior data science and AI/ML engineering roles. In-person in NYC with relocation support.