Copilot Agent Developer — 365 & Dynamics AI Testing

ITC WORLDWIDE; Karachi Division; On-site; PKR 1,500,000 - 2,500,000; Full time

Job summary

ITC WORLDWIDE is seeking a highly skilled Microsoft 365 / Dynamics 365 Copilot Agent Developer to design, develop, test, and optimize enterprise Copilot agents. The role focuses on building scalable test frameworks, automated evaluation pipelines, and benchmark datasets to ensure quality in Generative AI solutions.

You will leverage Copilot Studio Kit, Azure AI, Dataverse, Direct Line APIs, and AI governance practices to evaluate grounding, accuracy, and KPI-driven performance across enterprise

Qualifications

• Hands-on experience developing and evaluating multi-turn conversational agents.
• Experience with prompt engineering, Retrieval-Augmented Generation (RAG), and LLM evaluation.
• Ability to design automated regression testing frameworks for Microsoft 365 and Dynamics 365 Copilot solutions.
• Experience analyzing conversation transcripts, telemetry, knowledge retrieval effectiveness, and agent KPIs.

Responsibilities

• Design, develop, test, and optimize enterprise-grade Copilot agents.
• Build scalable test frameworks, automated evaluation pipelines, benchmark datasets, and quality validation processes.
• Validate agent accuracy, grounded Ness, response quality, relevance, safety, and overall business effectiveness.

Skills

Microsoft Copilot Studio

365 Copilot Extensibility

Dynamics 365 Copilot

Power Platform

Azure AI Services

Copilot Studio Kit

AI Builder

Prompt Engineering

RAG

Multi‑Turn Testing

Generative AI Evaluation

Agent Benchmark

Grounded Ness Testing

Hallucination Detection

Response & Attachment Testing

Azure Application Insights

Power BI Analytics

Direct Line API

Azure Entra ID Security

Azure DevOps / GitHub CI/CD

Test Automation Framework

AI Governance

Transcript Analysis

Agent KPI Reporting

Dynamics 365 Sales / CS / Insights

Job description

ITC WORLDWIDE is seeking a highly skilled Microsoft 365 / Dynamics 365 Copilot Agent Developer to design, develop, test, and optimize enterprise Copilot agents. The role focuses on building scalable test frameworks, automated evaluation pipelines, and benchmark datasets to ensure quality in Generative AI solutions.

You will leverage Copilot Studio Kit, Azure AI, Dataverse, Direct Line APIs, and AI governance practices to evaluate grounding, accuracy, and KPI-driven performance across enterprise