Copilot Agent Engineer for 365/Dynamics: AI QA & Automation

ITC WORLDWIDE; Karachi Division; On-site; PKR 2,000,000 - 3,000,000; Full time

Job summary

ITC WORLDWIDE is seeking a Microsoft 365 / Dynamics 365 Copilot Agent Developer in Karachi to design, develop, test, and optimize enterprise Copilot agents. You will build scalable test frameworks, evaluation pipelines, and benchmark datasets to validate grounded Ness, response quality, and business impact.

The ideal candidate has hands-on experience with multi-turn conversations, prompt engineering, RAG, and AI governance, collaborating with solution architects and QA teams to ensure

Qualifications

• Experience designing and evaluating enterprise-grade Copilot agents.
• Hands-on with multi-turn conversational agents and test automation.
• Strong understanding of AI governance, Responsible AI, and safety testing.
• Proven ability to build automated evaluation pipelines and benchmarks.

Responsibilities

• Design, develop, test, and optimize Copilot agents for Microsoft 365 and Dynamics 365.
• Build scalable test frameworks, evaluation pipelines, and KPI reporting.
• Analyze transcripts, telemetry, and knowledge retrieval effectiveness.
• Collaborate with architects, developers, and QA to meet reliability and security standards.

Skills

Microsoft Copilot Studio

Microsoft 365 Copilot Extensibility

Dynamics 365 Copilot

Power Platform

Azure AI Services

Copilot Studio Kit

AI Builder

Prompt Engineering

Retrieval-Augmented Generation

Multi-Turn Conversation Testing

Generative AI Evaluation

Grounded Ness Testing

Hallucination Detection

Azure DevOps / GitHub CI/CD

Test Automation Framework Development

Conversation Transcript Analysis

Agent KPI Reporting

Tools

Dataverse

Direct Line API

Azure OpenAI

Azure Application Insights

Power BI

Azure AI

Copilot Studio Kit

Job description

ITC WORLDWIDE is seeking a Microsoft 365 / Dynamics 365 Copilot Agent Developer in Karachi to design, develop, test, and optimize enterprise Copilot agents. You will build scalable test frameworks, evaluation pipelines, and benchmark datasets to validate grounded Ness, response quality, and business impact.

The ideal candidate has hands-on experience with multi-turn conversations, prompt engineering, RAG, and AI governance, collaborating with solution architects and QA teams to ensure