Analyst, Applied Validations

present of AI; New York (NY); On-site; USD 120,000 - 180,000; Full time

Job summary

Simile seeks an analyst to verify that AI-driven simulations of human behavior mirror real-world data. You will design validation studies, run backtests in Python, and compare results against ground truth using statistical metrics, clearly describing what the evidence supports and its limits.

The role emphasizes solid statistics, careful data quality judgment, and turning findings into actionable recommendations for modeling and data teams within an AI product context.

Qualifications

• Strong statistics and data quality judgment.
• Ability to translate findings into recommendations for modeling teams.
• Experience validating simulations against real-world data.

Responsibilities

• Design validation studies to compare simulated vs. real data.
• Run backtests in Python and evaluate with statistical metrics.
• Explain evidence and limitations clearly to stakeholders.
• Collaborate with data and modeling teams to improve simulations.

Skills

Python

Statistical analysis

Backtesting

Data quality

Tools

APIs

Python tools

Job description

The role centers on proving that simulated human behavior matches real world data. A strong candidate designs validation studies, runs backtests in Python, compares simulated results against ground truth with statistical metrics, and explains clearly what the evidence does and does not support. Solid statistics, careful judgment about data quality, and the ability to turn findings into recommendations for modeling and data teams matter more than any single tool.

How it fits the AI era

The work sits inside an AI product, so the analyst spends time checking model outputs and judging whether comparisons are fair. Coding agents speed up pipelines and experiments, but choosing the right study, spotting misleading comparisons, and explaining limits to customers stay human. The role grows as simulations become a way enterprises test decisions before launch.

How it builds AI experience

The job offers direct practice evaluating generative agents against real human responses, including holdout discipline and diagnosing where gaps come from. Building this means getting comfortable with backtests, distributional comparisons, and survey replication, plus reviewing outputs from coding agents rather than trusting them. Prior work evaluating LLM outputs or running survey based studies helps.

Worth showing

• Python and API experiments
• Backtest and holdout design
• Statistical comparison metrics
• Survey replication methods
• Clear customer readouts

This is our own read, written from the employer's posting and the company's Spotlight. The employer's posting is the source of truth for the role.

About Simile

Simile builds artificial intelligence driven simulations of human behavior so enterprise teams can test strategic decisions before launch, according to the company.