Copilot Agent Engineer - AI QA & Automation (365/D365)

ITC WORLDWIDE; Karachi Division; On-site; PKR 2,000,000 - 4,000,000; Full time

Job summary

ITC WORLDWIDE is seeking a highly skilled Microsoft 365 / Dynamics 365 Copilot Agent Developer to design, develop, test, and optimize enterprise Copilot agents using Microsoft Copilot Studio, Power Platform, and Azure AI. The role focuses on scalable test frameworks, evaluation pipelines, and automated regression testing to ensure high-quality GenAI solutions.

The ideal candidate will have hands-on experience with multi-turn conversational agents, prompt engineering, and governance practices,

Qualifications

• Hands-on experience building and evaluating multi-turn conversational agents (LLMs).
• Experience with Copilot Studio Kit and Azure AI services.
• Ability to design automated regression testing frameworks for Copilot solutions.
• Experience analyzing conversation transcripts, telemetry, knowledge retrieval, and agent KPIs.
• Knowledge of AI governance and Responsible AI practices.

Responsibilities

• Design, develop, test, and optimize enterprise Copilot agents.
• Build scalable test frameworks and evaluation pipelines.
• Validate agent accuracy, grounded ness, relevance, safety, and business impact.
• Collaborate with architects, developers, and QA teams to ensure enterprise standards.

Skills

Copilot Studio

Microsoft Copilot Studio

Dynamics 365 Copilot

Power Platform

Azure AI

AI Builder

Prompt Engineering

RAG

Multi-turn Testing

Test Automation

Agent Benchmark

Grounded Ness

Hallucination Detection

Response Testing

KPI Reporting

Direct Line API

Entra ID

DevOps / CI-CD

Conversation Analysis

Telemetry Analysis

Tools

Azure DevOps

GitHub CI/CD

Power BI

Dataverse

Direct Line API

Job description

ITC WORLDWIDE is seeking a highly skilled Microsoft 365 / Dynamics 365 Copilot Agent Developer to design, develop, test, and optimize enterprise Copilot agents using Microsoft Copilot Studio, Power Platform, and Azure AI. The role focuses on scalable test frameworks, evaluation pipelines, and automated regression testing to ensure high-quality GenAI solutions.

The ideal candidate will have hands-on experience with multi-turn conversational agents, prompt engineering, and governance practices,