Copilot Agent QA Engineer for 365 & Dynamics

ITC WORLDWIDE; Karachi Division; On-site; PKR 2,000,000 - 4,000,000; Full time

Job summary

ITC WORLDWIDE is seeking a Microsoft 365/Dynamics Copilot Agent Developer to design, test, and optimize enterprise-grade Copilot agents. You will build automated evaluation pipelines, assess grounded Ness and response quality, and ensure governance and security in a complex Microsoft ecosystem.

The role emphasizes multi-turn conversational testing, prompt engineering, and KPI monitoring using Copilot Studio Kit, Azure AI, Dataverse, and Direct Line APIs.

Qualifications

• Experience in designing and developing enterprise-grade Copilot agents.
• Experience with AI agent testing, evaluation, and automation.
• Hands-on with Copilot Studio, Azure AI services, and Dataverse.

Responsibilities

• Design, develop, test and optimize Copilot agents for enterprise use.
• Build scalable test frameworks and automated evaluation pipelines.
• Analyze transcripts and telemetry to measure agent KPIs and quality.

Skills

Prompt Engineering

RAG

Multi-Turn Testing

Generative AI Evaluation

Agent Benchmarking

Grounded Ness Testing

Hallucination Detection

Response Match Testing

Topic Match Testing

Attachment Match Testing

Conversation Transcript Analysis

Agent KPI Reporting

AI Governance

Tools

Microsoft Copilot Studio

Microsoft 365 Copilot Extensibility

Dynamics 365 Copilot

Power Platform (Power Apps, Power Automate, Dataverse)

Azure AI Services

Copilot Studio Kit

AI Builder

Direct Line API Integration

Azure Application Insights

Power BI Analytics & Reporting

Microsoft Entra ID Security

Azure DevOps / GitHub CI/CD

Test Automation Framework Development

Job description

ITC WORLDWIDE is seeking a Microsoft 365/Dynamics Copilot Agent Developer to design, test, and optimize enterprise-grade Copilot agents. You will build automated evaluation pipelines, assess grounded Ness and response quality, and ensure governance and security in a complex Microsoft ecosystem.

The role emphasizes multi-turn conversational testing, prompt engineering, and KPI monitoring using Copilot Studio Kit, Azure AI, Dataverse, and Direct Line APIs.