TS/SCI Data Scientist - AI Benchmark Architect

Precision Solutions Ag; Fort Meade (MD); On-site; USD 130,000 - 285,000; Full time

Job summary

Precision Solutions is seeking a Data Scientist to design and maintain AI model benchmark suites, curate evaluation datasets, and develop scoring methods for rigorous model assessment at Ft. Meade, MD.

The role emphasizes translating evaluation results for stakeholders and collaborating to deploy stable evaluation tools. Requirements include strong Python, InspectAI, MongoDB, and analytics skills, with experience in AI benchmarking and cloud/container environments.

Qualifications

• Experience with Python and data analysis.
• Experience designing or supporting AI model evaluation frameworks, LLM benchmarking, and agentic workflow evaluation.
• Experience evaluating tool use, multi-step task performance, task completion, and agent reliability.
• Strong knowledge of statistical analysis, experimental design, data cleaning, data validation, and data visualization.
• Experience curating, validating, and versioning evaluation datasets and test scenarios.
• Experience documenting dataset lineage, traceability, benchmark assumptions, scoring rationale, and evaluation results.
• Experience working within cloud-based or containerized environments.
• Experience using Git-based version control.
• Familiarity with Jira, Confluence, or similar collaboration and documentation tools.
• Ability to operate effectively within airgapped or otherwise constrained environments.

Responsibilities

• Design, implement, and maintain AI model benchmark suites that measure model performance against defined evaluation criteria.
• Curate evaluation datasets, develop scoring methods, and produce clear and repeatable results reporting.
• Develop benchmarks and test scenarios that evaluate agentic workflows and multi-step task performance.
• Measure tool use, task completion, reliability, and other performance indicators across agentic systems.
• Curate, validate, and version evaluation datasets and test scenarios.
• Develop adversarial and edge-case datasets designed to stress-test model and agent behavior.
• Produce scoring frameworks, statistical analyses, visualizations, and written findings.
• Translate raw evaluation results into clear and actionable information for technical and nontechnical stakeholders.
• Document dataset lineage, scoring rationale, benchmark assumptions, evaluation methodology, and evaluation outputs.
• Collaborate with Researchers and Developers to operationalize evaluation workflows into stable and repeatable tools.

Skills

Python

InspectAI

MongoDB

Jupyter notebooks

Data visualization

Statistical analysis

Data cleaning

LLM benchmarking

Agentic workflow evaluation

Experimental design

Tools

Git

Jira

Confluence

Job description

Precision Solutions is seeking a Data Scientist to design and maintain AI model benchmark suites, curate evaluation datasets, and develop scoring methods for rigorous model assessment at Ft. Meade, MD.

The role emphasizes translating evaluation results for stakeholders and collaborating to deploy stable evaluation tools. Requirements include strong Python, InspectAI, MongoDB, and analytics skills, with experience in AI benchmarking and cloud/container environments.