Applied AI Engineer - Developer Experience

Spectrum IT Recruitment; California (MO); On-site; USD 150,000 - 210,000; Full time

Job summary

Spectrum IT Recruitment seeks an experienced Applied AI Engineer to enhance AI use across the software-development lifecycle. You will design experiments, build evaluation pipelines, and translate results into practical AI capabilities for development teams.

You will partner with data, platform, and engineering groups to influence testing, code review, migrations, security remediations, and compliance evidence, measuring business impact along the way.

Qualifications

• Experience designing and executing experiments to evaluate AI capabilities.
• Strong Python and SQL skills with practical ML/AI tooling.
• Familiarity with CI/CD workflows and software-delivery impact.

Responsibilities

• Design and run experiments to evaluate AI configurations and coverage.
• Build classification and evaluation pipelines for software teams.
• Develop LLM-evaluation systems with ground truth and metrics.
• Translate findings into practical AI tools and workflows.
• Collaborate with data, platform, and engineering teams to drive adoption.

Skills

Python

SQL

Experiment design

Analytical skills

Communication skills

CI/CD knowledge

Tools

GitLab

Jira

Confluence

dbt

Airflow

LangChain

LangGraph

Bedrock

Job description

We are seeking an experienced Applied AI Engineer to help evaluate and improve how artificial intelligence is used throughout the software-development lifecycle.

This role goes beyond AI-assisted code generation. You will explore how AI can improve testing, code review, documentation, migrations, incident response, security remediation, certification, and other stages that commonly constrain software delivery.

You will own both the analytical and applied sides of this work. This includes designing experiments, developing classification and evaluation pipelines, measuring business and engineering outcomes, and translating findings into practical AI capabilities that development teams can use.

You will partner with data, platform, and engineering teams. While data engineers will support extraction, identity resolution, and metric pipelines, you will be responsible for interpreting the results, establishing what the evidence supports, and determining how those findings should influence engineering practices.

Key Responsibilities

• Design and run experiments to evaluate model selection, context quality, permissions, MCP coverage, budget allocation, and other AI-development configurations.
• Use randomized and quasi-experimental methods to determine which AI capabilities meaningfully improve software delivery.
• Build classification pipelines for understanding how engineering teams use AI tools.
• Develop and validate LLM-as-a-judge systems using appropriate sampling, human-labelled ground truth, and published precision and recall measurements.
• Revalidate evaluation systems whenever models, prompts, or classification taxonomies change.
• Build AI evaluation frameworks, including golden datasets, regression suites, groundedness measures, answer-quality scoring, and cost and latency monitoring.
• Analyse adoption and performance over time using cohort analysis, longitudinal methods, and staggered-adoption estimates.
• Extend AI capabilities into testing, environment setup, data preparation, migration, modernisation, code review, security remediation, and compliance evidence.
• Work directly with engineering teams to identify the actual constraints affecting software delivery.
• Translate analytical findings into practical tools, workflows, and recommendations.
• Identify the behaviours and working practices associated with successful AI adoption and teach those practices across teams.
• Partner with platform engineers on coding-assistant configuration, MCP servers, model gateways, telemetry, and model registries.
• Communicate findings to engineering and business leadership, clearly explaining what is working, what is not, and the limitations of each conclusion.
• Handle engineering-productivity and personnel-adjacent data responsibly, using aggregated reporting by default.

Required Experience

• At least seven years of combined experience across software engineering, data science, applied machine learning, or quantitative analysis.
• Production experience building or deploying LLM-powered applications.
• Practical knowledge of prompting, context management, tool or function calling, model evaluation, and common LLM failure modes.
• Strong experience with experimental design and causal inference.
• Familiarity with randomized experiments, quasi-experimental methods, difference-in-differences, instrumental variables, or hierarchical models.
• Strong Python and SQL skills.
• Experience with statistical and machine-learning tools such as pandas, statsmodels, scikit-learn, or R.
• Experience collecting and instrumenting data from operational systems, APIs, and software-development platforms.
• Ability to design defensible sampling and validation methodologies.
• Experience resolving identities and joining data from systems that were not originally designed to work together.
• Strong analytical skills, including exploratory analysis, distributions, cohorts, and time-series analysis.
• Good understanding of the software-development lifecycle, including code review, testing, CI/CD, release management, and change management.
• Working knowledge of Git and Git-based development workflows.
• Experience analysing repository, merge-request, commit, diff, SHA, and CI/CD pipeline data.
• Experience integrating with systems such as GitLab, Jira, or Confluence through APIs.
• Strong written and verbal communication skills.
• Ability to explain analytical findings to executives while also presenting methodologies that can withstand technical scrutiny from engineers.

Preferred Experience

• Building MCP servers or clients, or working with comparable connector frameworks.
• Experience with agent frameworks such as LangGraph, LangChain, Amazon Bedrock Agents, Strands, or similar technologies.
• Enterprise deployment of AI coding assistants and analysis of their usage telemetry.
• Experience with server-side Git hooks, GitLab CI, webhooks, or event-driven data capture.
• Knowledge of permissions, scoped credentials, and audit logging for MCP-connected enterprise systems.
• Experience with LLM evaluation tools such as Ragas, DeepEval, or Amazon Bedrock Model Evaluation.
• Familiarity with Amazon Bedrock, AWS infrastructure, or cloud cost-and-usage data.
• Knowledge of engineering-productivity frameworks such as DORA, SPACE, or DX Core 4.
• Experience with program analysis, test generation, developer tooling, or software-engineering research.
• Familiarity with transformation and orchestration tools such as dbt, Airflow, or Dagster.
• Experience with business-intelligence and data-visualisation tools.
• Knowledge of queueing, flow analysis, utilisation, batch economics, or constraint identification.

What Success Looks Like

Your initial focus will be establishing a credible measurement programme for AI-assisted software development. You will help the organisation understand which AI investments produce meaningful outcomes, where adoption is being constrained, and which capabilities should be developed next.

Success will not be measured by the number of AI tools introduced. It will be measured by the quality of the evidence produced, the reliability of the evaluation systems, and the demonstrable improvements made to software delivery.

What We Offer

• A global and multicultural working environment.
• The opportunity to solve complex, high-impact applied AI problems.
• Direct ownership of the systems and capabilities you develop.
• Close collaboration with engineering, platform, data, and business leaders.
• A fast-moving environment that values curiosity, evidence, and practical outcomes.
• The opportunity to influence how AI is applied across the entire software-development lifecycle.