CDI - GenAI R&D Automation Testing Senior Software Quality Engineer - SBS - Paris

SBS; Courbevoie; Sur place; EUR 50 000 - 75 000; Plein temps

Résumé du poste

Une entreprise innovante à Courbevoie recherche un Ingénieur QA Senior en automatisation de tests pour garantir la qualité de leur plateforme AI. Le candidat sera responsable de la conception de frameworks de tests automatisés, de la collaboration avec des équipes AI, et de la mise en place de stratégies complètes pour les agents AI basées sur RAG. Ce poste offre un environnement de travail enrichissant et l'opportunité de contribuer à des projets de pointe dans le domaine de l'IA.

Prestations

Travail à distance 2 jours par semaine

Bénéfices attrayants : mutuelle, CSE, titres restaurants, primes de vacances

Qualifications

• 5+ ans d'expérience dans les tests logiciels, dont au moins 2 ans sur des systèmes AI/ML.
• Maîtrise des applications LLM, chatbots, ou IA conversationnelle.
• Compréhension des architectures RAG et bases de données vectorielles.

Responsabilités

• Concevoir et implémenter des frameworks de tests automatisés pour les pipelines RAG.
• Développer des suites de tests spécialisées pour les composants AI/ML.
• Collaborer avec des ingénieurs AI pour définir des métriques de qualité.

Connaissances

Tests de systèmes AI/ML

Automatisation des tests

Python

NLP

Développement de frameworks de test

Formation

Licence en Informatique, ingénierie ou domaine similaire

Outils

pytest

LangChain

MLflow

Description du poste

Social network you want to login/join with:

CDI - GenAI R&D Automation Testing Senior Software Quality Engineer - SBS - Paris, Courbevoie

col-narrow-left

Client:

SBS

Location:

Courbevoie, France

Job Category:

Other

-

EU work permit required:

Yes

col-narrow-right

Job Reference:

c605d2e09e75

Job Views:

3

Posted:

30.06.2025

Expiry Date:

14.08.2025

col-wide

Job Description:

As a GenAI QA Engineer, you will ensure the quality and reliability of our RAG-based AI agent platform. Your responsibilities include:

Design and implement automated testing frameworks for RAG pipelines, including:

• Vector database performance and accuracy testing
• Retrieval quality metrics and relevance scoring
• LLM response validation and hallucination detection
• End-to-end agent conversation flow testing

Develop specialized test suites for AI/ML components:

• Knowledge base ingestion and chunking strategies
• Embedding quality and semantic search accuracy
• Prompt injection and security vulnerability testing
• Multi-modal content handling (documents, tables, images)

Create automated evaluation frameworks for:

• Agent response accuracy and consistency
• Contextual understanding and reasoning capabilities
• Performance benchmarking across different LLMs
• A/B testing for prompt engineering optimization

Collaborate with AI engineers to:

• Define quality metrics for RAG architectures
• Establish ground truth datasets for evaluation
• Design test scenarios for edge cases and failure modes

Build testing infrastructure for:

• Knowledge base versioning and rollback testing
• API rate limiting and scalability testing
• Integration testing with customer systems

Ensure compliance and safety:

• Test for bias and fairness in AI responses
• Validate data privacy and security measures
• Implement guardrails testing for harmful content
• Document AI system limitations and failure modes

Develop comprehensive test strategies for RAG-based AI agents.

Create automated benchmarks for retrieval quality and response accuracy.

Build dashboards for monitoring AI system performance in production.

Collaborate with customers to understand their AI agent requirements.

Contribute to AI safety and alignment best practices.

Description du poste

As a GenAI QA Engineer, you will ensure the quality and reliability of our RAG-based AI agent platform. Your responsibilities include:

Design and implement automated testing frameworks for RAG pipelines, including:

• Vector database performance and accuracy testing
• Retrieval quality metrics and relevance scoring
• LLM response validation and hallucination detection
• End-to-end agent conversation flow testing

Develop specialized test suites for AI/ML components:

• Knowledge base ingestion and chunking strategies
• Embedding quality and semantic search accuracy
• Prompt injection and security vulnerability testing
• Multi-modal content handling (documents, tables, images)

Create automated evaluation frameworks for:

• Agent response accuracy and consistency
• Contextual understanding and reasoning capabilities
• Performance benchmarking across different LLMs
• A/B testing for prompt engineering optimization

Collaborate with AI engineers to:

• Define quality metrics for RAG architectures
• Establish ground truth datasets for evaluation
• Implement continuous monitoring for model drift
• Design test scenarios for edge cases and failure modes

Build testing infrastructure for:

• Multi-tenant agent deployments
• Knowledge base versioning and rollback testing
• API rate limiting and scalability testing
• Integration testing with customer systems

Ensure compliance and safety:

• Test for bias and fairness in AI responses
• Validate data privacy and security measures
• Implement guardrails testing for harmful content
• Document AI system limitations and failure modes

Develop comprehensive test strategies for RAG-based AI agents.

Create automated benchmarks for retrieval quality and response accuracy.

Design adversarial testing scenarios to identify system vulnerabilities.

Build dashboards for monitoring AI system performance in production.

Collaborate with customers to understand their AI agent requirements.

Contribute to AI safety and alignment best practices.

Qualifications

Required Skills:

Education: Bachelor's degree in Computer Science, Engineering, AI/ML, or related field.

Experience: 5+ years in software testing with at least 2 years focused on AI/ML systems.

AI/ML Testing Expertise:

• Experience testing LLM applications, chatbots, or conversational AI
• Understanding of RAG architectures and vector databases (Pinecone, Weaviate, Qdrant)
• Familiarity with embedding models and similarity search concepts
• Knowledge of prompt engineering and LLM evaluation metrics

Technical Skills:

• Proficiency in Python for test automation and AI/ML frameworks
• Experience with LLM frameworks (LangChain, LlamaIndex, Haystack)
• API testing for RESTful services and streaming endpoints
• Familiarity with ML testing tools (MLflow, Weights & Biases, Neptune)

Automation Frameworks:

• pytest, unittest for Python-based testing
• Experience with async testing for streaming responses
• Load testing tools for AI endpoints (Locust, K6)
• CI/CD integration with model deployment pipelines

Domain Knowledge:

• Understanding of NLP concepts and evaluation metrics (BLEU, ROUGE, BERTScore)
• Knowledge of information retrieval metrics (precision, recall, MRR)
• Familiarity with financial services use cases for AI agents
• Understanding of responsible AI principles

Preferred Qualifications:

• Experience with cloud AI services (AWS Bedrock, Azure OpenAI, Google Vertex AI)
• Knowledge of vector database optimization and indexing strategies
• Familiarity with fine-tuning and model evaluation workflows
• Experience with multilingual AI systems testing
• Understanding of regulatory requirements for AI in financial services (EU AI Act, GDPR)
• Contributions to open-source AI/ML testing frameworks

Informations supplémentaires

Les avantages à nous rejoindre :

• Un accord télétravail pour télétravailler jusqu'à 2 jours par semaine selon vos missions.
• Un package avantages intéressants : une mutuelle, un CSE, des titres restaurants, un accord d'intéressement, des primes vacances.