Artificial Intelligence Researcher

Aavalar Consulting, Inc; Philadelphia (Philadelphia County); On-site; USD 110,000 - 160,000; Full time

Job summary

Aavalar Consulting, Inc is seeking a technically adept evaluator to research and benchmark emerging AI platforms, LLMs, and agents. You will design structured evaluation plans, build proofs‑of‑concepts, and create reusable methods for future IT assessments across enterprise environments.

The role requires hands‑on testing, cloud familiarity, and strong communication to present actionable recommendations to both technical and business stakeholders.

Qualifications

• Bachelor's degree or equivalent experience in a related field.
• Experience with evaluating AI platforms, agents, and enterprise technologies.

Responsibilities

• Research and evaluate emerging AI platforms, LLMs, coding assistants, agents, MCP-enabled solutions, and related enterprise technologies.
• Design and execute structured evaluations, benchmarks, test scenarios, and scorecards to assess capabilities, reliability, security, interoperability, scalability, and value.
• Experiment, prototype, and build proofs-of-concept to technical capabilities and limitations.
• Evaluate agent configuration, integrations, data flows, authentication, permissions, security risks, and governance considerations.
• Analyze token usage, context windows, rate limits, licensing, consumption patterns, and other cost drivers.
• Compare competing technologies, document findings, and present clear recommendations to technical and business stakeholders.
• Develop reusable evaluation methods, standards, guidance, and decision criteria for future IT assessments.
• Build scripts, test harnesses, integrations, or sample applications as needed to support evaluations.

Skills

Generative AI concepts

LLMs & prompting

Agent orchestration

Evaluation methodologies

Software engineering

Cloud technologies

Security testing

CI/CD

APIs & integrations

Python

Research & hands-on testing

Communication skills

Education

Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or related discipline

Tools

GitHub Copilot

Claude Code

M365 Copilot

Azure AI

Job description

• Research and evaluate emerging AI platforms, LLMs, coding assistants, agents, MCP-enabled solutions, and related enterprise technologies
• Design and execute structured evaluations, benchmarks, test scenarios, and scorecards to assess capabilities, reliability, security, interoperability, scalability, and value
• Experiment, prototype, and build proofs-of-concept to technical capabilities and limitations
• Evaluate agent configuration, integrations, data flows, authentication, permissions, security risks, and governance considerations
• Analyze token usage, context windows, rate limits, licensing, consumption patterns, and other cost drivers
• Compare competing technologies, document findings, and present clear recommendations to technical and business stakeholders
• Develop reusable evaluation methods, standards, guidance, and decision criteria for future IT assessments
• Build scripts, test harnesses, integrations, or sample applications as needed to support evaluations

Required Skills & Experience

• Bachelor's degree in Computer Science, Engineering, Information Systems, Data Science, or related discipline, or equivalent experience
• Working proficiency with Generative AI concepts and technologies, including LLMs, prompting, context, RAG, tool use, agents, orchestration, and evaluation
• Experience evaluating M365 Copilot, GitHub Copilot, Claude Code, AI agents, MCP servers, or comparable technologies
• Familiarity with prompt/agent evaluation, automated/adversarial/security/AI safety testing
• Strong technical foundation in software engineering, enterprise platforms, cloud technologies, integrations, or solution architecture
• Experience with Microsoft 365, Entra ID, Azure AI, AWS AI services, or hybrid enterprise environments
• Experience creating technical scorecards, comparison matrices, architecture assessments, security reviews, or technology standards
• Knowledge of software architecture, CI/CD, source control, observability, or enterprise data integration
• Ability to independently research unfamiliar technologies, conduct hands‑on testing, and reach evidence‑based conclusions
• Develop Python or another modern language to build prototypes, integrations, and test assets
• Experience with APIs, authentication, cloud services, developer tooling, or technical prototyping
• Understanding of enterprise security, identity/access management, permissions, data protection, and risks
• Strong written and verbal communication skills and a self‑directed approach to working in an agile innovation environment