Given input, expect output works for a plugin. Agent responses depend on prompt, retrieval and context interpretation, so you measure response quality and task alignment, not correctness.
Test sets. Built into Copilot Studio. Up to 100 cases per set — hand-written, spreadsheet import, or AI-generated. Quick set: 10 questions from the agent's description and instructions. Full set: up to 100 from knowledge sources or topics.
The generator reaches knowledge sources using the connected account's credentials, so generated cases can contain sensitive data that account can see. Review before sharing the set.
Grading. Every method except general quality needs expected responses or keywords — writing down what a good answer looks like is the actual work. Custom Graders (classification method) encode your own policies where built-in dimensions don't fit.
Test user profiles. Evaluations run under a designated test account, and that account connects to knowledge sources and tools during the run. Testing as yourself proves nothing about what a sales rep sees. Simulate profiles to check behaviour across roles and access levels.
Limits: GCC can't add user profiles to test sets and doesn't support the similarity method. User-auth evaluations need the Copilot Studio connector enabled.
Automation. Results live 89 days — export CSV for anything auditable. REST API triggers evaluations programmatically for release validation and CI/CD regression runs.
Copilot Studio Kit goes deeper: tests via Direct Line API, enriched from Application Insights and Dataverse transcripts, so you get triggered topic and intent recognition scores behind each pass or fail. Multi-turn tests, and pipeline gating — deploy pauses, tests run, thresholds checked, then promote.
What goes in the set:
- Questions users actually ask, badly phrased ones included
- Every bug that ever shipped, permanently
- Questions the agent should refuse or escalate
- The same question from two profiles where correct answers differ
- Anything near column-secured fields
Correction to the ALM post. I said keep a manual list of twenty questions and run it after each deploy. Right instinct, wrong implementation — put them in a test set, grade them, run them from the pipeline.
Related: Copilot in Dynamics 365 — what it actually does
Thanks for reading this article. Hope this Article will help you. Cheers!!!
#Dynamics365 #PowerPlatform #MicrosoftCopilot #DynamicsCRM #CopilotStudio #Dataverse #AIDeveloper #DynamicsAIEngineer #HireAIEngineer #AIEngineer #D365Consultant #CRMDeveloper #AIAgents #MSDyn365
