Medior AI Engineer

Keystone Solutions; Brussel Hoofdstad; On-site; EUR 70,000 - 95,000; Full time

Job summary

Keystone Solutions is seeking a Medior AI Engineer to join our consultancy mission at a client site in Belgium. You will join the AI Center of Excellence, focusing on Run phase production operations with a hybrid Azure/open-source stack.

You will monitor performance, reliability, and security, manage ingestion and integrations, and drive cost efficiency and safe change control across DEV to PROD environments. Excellent Dutch/French and English proficiency required.

Qualifications

• Proven experience deploying AI services in production.
• Strong knowledge of Azure cloud and open-source stack.
• Experience with cost monitoring and optimization.

Responsibilities

• Operate AI services in production: monitor availability, latency, errors; handle incidents and requests; analyze root causes and fix or escalate.
• Maintain evaluation sets with business content owners; track quality, non-answer rate, and user feedback; detect regressions.
• Manage ingestion pipelines: incremental refresh, failed documents, index health, onboarding of new content sources.
• Monitor tool and integration calls to business systems; handle failures and timeouts; review logs and exceptions.
• Manage model/version upgrades, configuration changes, and security updates; validate against eval sets before release.
• Monitor token consumption and cost per service; alert on drift; propose optimizations.
• Maintain Terraform, Helm, and Azure DevOps artifacts across DEV/TST/ACC/PROD environments.
• Apply data protection rules: trace retention, erasure, access restrictions per service DPIA.
• Maintain runbooks and documentation; report service health to owners and AI CoE.
• Deliver small enhancements and fixes; larger features handled by Build team.

Skills

Azure DevOps

AKS

Azure OpenAI

LangChain

LLM

Python

RAG

Terraform

Vector databases

Job description

Mission Overview:

Keystone Solutions is seeking a Medior AI Engineer to join our consultancy mission at a client site. You will be part of the AI Center of Excellence, focusing on the Run phase of AI services in production. This role involves ensuring the reliability, security, and cost-efficiency of AI services built on a hybrid operating model, utilizing both managed Azure services and self-hosted open-source components.

Key Responsibilities:

• Operate the AI services in production: monitor availability, latency, and errors, handle incidents and service requests, analyze root causes, then fix or elevate.
• Keep answer quality under control: run and maintain evaluation sets with business content owners, track answer quality, non-answer rate, and user feedback, and detect and investigate quality regressions.
• Operate the ingestion pipelines: manage incremental refresh, failed documents, index health, and onboarding of new content sources or business domains.
• Operate agent integrations: monitor tool and integration calls to business systems, handle failures and timeouts, and review action logs and exceptions.
• Manage change safely: model version upgrades and retirements, prompt and configuration changes, and dependency and security updates, validated against evaluation sets before release.
• Follow run cost: monitor token consumption and cost per service, alert on drift, and propose optimizations.
• Maintain Terraform, Helm, and Azure DevOps artifacts, ensuring consistency across DEV, TST, ACC, and PROD environments.
• Apply data protection rules: manage trace retention, erasure, and access restrictions as defined in each service’s data protection impact assessment.
• Maintain runbooks and documentation, and report service health to service owners and the AI CoE regularly.
• Deliver small enhancements and fixes; structural changes and new features will be handled by the Build team.

Why Join Keystone Solutions?

As a consultant with Keystone Solutions, you will experience:

• Consultancy Nature: Engage in on-site work, embodying the consultant identity.
• Dynamic Projects: Tackle diverse challenges across various client environments.
• Turbo-Charged Learning: Benefit from professional development and broad learning experiences.
• Skyrocketing Ambition: Enjoy a commitment to career growth within our framework.
• Values: Being a "K-Stone" means bringing our core values to every engagement.

Duration:

04/01/2027 - 31/12/2027 12 months (full time)

Skills required:

• Azure DevOps - Level: Intermediate - Most recent: This year
• Azure Kubernetes Service - AKS - Level: Intermediate - Most recent: This year
• Azure OpenAI Service - Level: Intermediate - Most recent: This year
• LangChain - Level: Intermediate - Most recent: Any time
• LLM / Large Language Models - Level: Intermediate - Most recent: This year
• Python - Level: Intermediate - Most recent: This year
• RAG (Retrieval-Augmented Generation) - Level: Intermediate - Most recent: This year
• Terraform - Level: Intermediate - Most recent: This year
• Vector databases - Level: Intermediate - Most recent: This year

Language requirements:

• Dutch or French Level Native or bilingual proficiency
• English Level Full professional proficiency