Testing a Copilot agent: evaluation, not "it seemed to work"

Given input, expect output works for a plugin. Agent responses depend on prompt, retrieval and context interpretation, so you measure response quality and task alignment, not correctness.

Test sets. Built into Copilot Studio. Up to 100 cases per set — hand-written, spreadsheet import, or AI-generated. Quick set: 10 questions from the agent's description and instructions. Full set: up to 100 from knowledge sources or topics.

The generator reaches knowledge sources using the connected account's credentials, so generated cases can contain sensitive data that account can see. Review before sharing the set.

Grading. Every method except general quality needs expected responses or keywords — writing down what a good answer looks like is the actual work. Custom Graders (classification method) encode your own policies where built-in dimensions don't fit.

Test user profiles. Evaluations run under a designated test account, and that account connects to knowledge sources and tools during the run. Testing as yourself proves nothing about what a sales rep sees. Simulate profiles to check behaviour across roles and access levels.

Limits: GCC can't add user profiles to test sets and doesn't support the similarity method. User-auth evaluations need the Copilot Studio connector enabled.

Automation. Results live 89 days — export CSV for anything auditable. REST API triggers evaluations programmatically for release validation and CI/CD regression runs.

Copilot Studio Kit goes deeper: tests via Direct Line API, enriched from Application Insights and Dataverse transcripts, so you get triggered topic and intent recognition scores behind each pass or fail. Multi-turn tests, and pipeline gating — deploy pauses, tests run, thresholds checked, then promote.

What goes in the set:

• Questions users actually ask, badly phrased ones included
• Every bug that ever shipped, permanently
• Questions the agent should refuse or escalate
• The same question from two profiles where correct answers differ
• Anything near column-secured fields

Correction to the ALM post. I said keep a manual list of twenty questions and run it after each deploy. Right instinct, wrong implementation — put them in a test set, grade them, run them from the pipeline.

Related: Copilot in Dynamics 365 — what it actually does

Thanks for reading this article. Hope this Article will help you. Cheers!!!

#Dynamics365 #PowerPlatform #MicrosoftCopilot #DynamicsCRM #CopilotStudio #Dataverse #AIDeveloper #DynamicsAIEngineer #HireAIEngineer #AIEngineer #D365Consultant #CRMDeveloper #AIAgents #MSDyn365

Packaging and Deploying Web Resources: A Practical ALM Workflow for Dynamics 365 JavaScript

You've written the script. You've unit-tested the logic and stepped through it in dev tools. It works perfectly on your form. Now comes the part that trips up more Dynamics 365 projects than any bug ever does: getting that script safely out of your dev environment and into production without breaking something else on the way.

This post is about the plumbing — solutions, environment variables, and a repeatable release process — rather than more JavaScript syntax. If you've been following the series, think of it as the missing link between "my script works" and "my script is live, and I can update it again next sprint without a two-hour outage window."

Web resources live in a solution — always

A web resource that exists only in your dev environment isn't deployable. The first rule of ALM in Dynamics 365 is simple: every JavaScript file, every image, every HTML web resource you build has to sit inside an unmanaged solution in dev, so it can be exported and carried forward.

Create one solution per project or product area — not the Default Solution — with your own publisher and a prefix:

Publisher: Contoso CRM Team
Prefix: con_

Web resource name: con_/scripts/opportunity.form.js
Web resource name: con_/scripts/lib/discountLogic.js

Notice the folder-style naming. Dynamics doesn't enforce folders, but the "/scripts/", "/styles/", "/images/" convention inside the name keeps a solution with 40+ web resources from turning into an unreadable flat list. Pick a convention on day one — renaming a web resource later means updating every form event handler that references it by name.

Stop hardcoding URLs and GUIDs

This is the single most common thing I see go wrong in a promotion from dev to test to production: a script with a hardcoded environment URL, flow trigger endpoint, or record GUID baked directly into the code.

// don't do this
const flowUrl = "https://prod52a1.environment.api.powerplatform.com/powerautomate/...";
const escalationTeamId = "3f9a1c20-...-...-...-prodonly";

That script works in production and breaks the moment someone imports the same managed solution into UAT, because the flow URL and team GUID are different there. Use environment variables instead — a Dataverse table type built exactly for this. Define the environment variable in your solution, give it a default value for dev, and set the current value per environment after import (test, UAT, and prod each get their own).

// do this instead
async function getEnvVarValue(schemaName) {
  const result = await Xrm.WebApi.retrieveMultipleRecords(
      "environmentvariabledefinition",
          `?$filter=schemaname eq '${schemaName}'&$expand=environmentvariabledefinition_environmentvariablevalue($select=value)`
            );
              const def = result.entities[0];
                const values = def.environmentvariabledefinition_environmentvariablevalue;
                  return values && values.length ? values[0].value : def.defaultvalue;
                  }
                  
                  const flowUrl = await getEnvVarValue("con_DiscountApprovalFlowUrl");

Yes, it's a couple more lines than a hardcoded string. It's also the difference between a 30-second solution import and a support ticket at 5pm on a Friday because someone forgot to swap a URL by hand.

A release path you can actually repeat

The environments-and-arrows diagram is familiar to anyone who's worked on a real project, but the part people skip is making it boring — the same steps, every time, ideally without a human retyping anything:

Dev (unmanaged)
  → export as MANAGED solution
      → import into Test/UAT
            → validate, set environment variable values
                    → export the same version as MANAGED
                              → import into Production
                                          → set production environment variable values

A few rules that keep this from going sideways:

Only your dev environment is unmanaged. Every downstream environment gets a managed solution. Managed solutions can't be casually edited in the target environment — which is exactly what stops someone from "just quickly" fixing a bug directly in production and having it silently overwritten by the next real release.

Bump the solution version every release (1.0.0.3, 1.0.0.4...) so you can tell at a glance what's actually deployed in each environment, and so a re-import doesn't get silently skipped as "no changes detected."

Never hand-edit a web resource inside the target environment's UI. If test or production needs a change, it goes back through dev, gets re-exported, and flows down the same path. The moment someone edits a web resource directly in production "just this once," your source of truth splits in two.

Keep the actual JavaScript in source control

The solution zip is not source control. Treat the Dataverse web resource as a deployment target, not the place your code lives. The workflow that scales:

1. Write/edit the .js file in your repo (VS Code, real linting, real diffs)
2. Use the Power Platform CLI to push it straight to your dev web resource:

   pac webresource push --file ./src/opportunity.form.js `
        --solution ConCrmTeamSolution --publisher con
        
        3. Commit the .js file to git like any other source file
        4. When it's time to release, export/pack the SOLUTION (not the .js file) via:
        
           pac solution export --name ConCrmTeamSolution --managed
              pac solution unpack --zipfile ConCrmTeamSolution.zip --folder ./solution --packagetype Managed

Once the unpacked solution folder is in git alongside your web resource source, a pull request actually shows you a meaningful diff — not a base64 blob — and code review on a form script becomes possible for the first time.

Cache is the silent killer of "but it worked in dev"

You imported the solution, the web resource clearly has your new code in the customizations, and the form still runs the old logic. Nine times out of ten, that's the browser serving a cached copy of the .js file. A few things that actually fix it, in order of how often I reach for them:

Do a hard refresh (Ctrl+Shift+R / Cmd+Shift+R) on the form before assuming your deployment failed. If it's happening to your whole team repeatedly after every release, increment a version query string convention in how the web resource is referenced, or use the "Publish All Customizations" step deliberately — a solution import doesn't always auto-publish everything, and an unpublished web resource will serve the old version even though the record itself shows the new content.

A short pre-flight checklist

Before you ship a form script anywhere past dev, it's worth running down a short list rather than trusting memory:

Is the web resource actually included in the solution (added web resources don't auto-include just because they exist)? Are there any hardcoded URLs, GUIDs, or environment-specific values left in the code? Does the target environment already have its environment variable values set, or will the script silently fail on first load after import? Did you bump the solution version? And — the one everyone forgets at least once — did you publish?

Where this leaves you

None of this is exciting the way a working Copilot Studio integration is, but it's the reason that integration keeps working three months from now, across three environments, maintained by more than one person. A script that only exists correctly in your head and your dev environment isn't shipped — it's a demo.

Next in the series, we'll go back to the form itself and look at ribbon and command bar customization with JavaScript — enable rules, custom buttons, and the display rules that decide when a command even shows up.

Ribbon and Command Bar Customization with JavaScript in Dynamics 365

Everything so far in this series has lived inside the form body — fields, lookups, Web API calls, agent responses. This post moves up one level, to the command bar sitting above all of it: the Save button, the ribbon buttons, the custom "Send for Approval" or "Escalate to Manager" buttons that show up (or don't) depending on the state of the record. That behavior is driven by JavaScript too, and it trips people up in a different way than form scripting does, because the wiring is less obvious.

Two ways to control a button, and why this post is about the JS one

Dynamics 365 gives you two paths to command bar logic. The modern command bar designer lets you write simple Power Fx-style formulas directly against a button's Visible and Enabled properties — no code, no web resource, good enough for straightforward conditions. The classic path — Enable Rules and Display Rules backed by a JavaScript function — is what you reach for when the condition is genuinely complex: multiple entity checks, a Web API call to decide visibility, or logic you want to unit test the same way you'd test any other function in this series.

This post covers the JavaScript path, because that's where the interesting problems are.

Enable Rules vs Display Rules — they are not the same thing

A Display Rule decides whether a button appears at all. A Enable Rule decides whether a button that's already visible can be clicked — a visible-but-greyed-out state. Mixing these up is the single most common ribbon bug: a button that a user can see and click, but that silently does nothing because the actual intent was to hide it until certain conditions were met, not just enable it.

Both rule types point at a JavaScript function with the same signature — Dynamics calls it a CommandChecker function — and both must return a plain boolean.

function canSendForApproval(primaryControl) {
  var formContext = primaryControl;
    var status = formContext.getAttribute("statuscode").getValue();
      var isDirty = formContext.data.getIsDirty();
      
        // only show/enable once saved and still in Draft
          return status === 1 && !isDirty;
          }

Notice the parameter isn't literally "the form" — it's whatever you configured as a CRM Parameter when you wired the rule up in the Ribbon Workbench (or classic solution ribbon editor). PrimaryControl is the one you'll use constantly, because it gives you the same formContext-shaped object you already know from every other post in this series — getAttribute, getValue, ui, all of it.

A realistic enable rule: role- and state-aware

Real conditions are rarely just one field check. Here's a button that should only be enabled for the record owner, only once the opportunity has reached the Propose stage, and only if there's no pending approval already:

async function canEscalate(primaryControl) {
  var formContext = primaryControl;
    var userId = Xrm.Utility.getGlobalContext().userSettings.userId;
      var ownerId = formContext.getAttribute("ownerid").getValue();
      
        if (!ownerId || ownerId[0].id.toLowerCase() !== userId.toLowerCase()) {
            return false;
              }
              
                var recordId = formContext.data.entity.getId().replace(/[{}]/g, "");
                  var result = await Xrm.WebApi.retrieveRecord(
                      "opportunity", recordId, "?$select=salesstage,con_hasopenescalation"
                        );
                        
                          return result.salesstage === 2 && !result.con_hasopenescalation;
                          }

CommandChecker functions can be async — Dynamics will wait for the promise before deciding the button's state. Just be aware that every extra Web API call in a rule adds latency to ribbon rendering, especially on grids with dozens of visible rows each re-evaluating their own rule. Keep the check as cheap as you can; if you find yourself calling the Web API from an enable rule on a subgrid, it's often faster to precompute that value onto the record itself (a rollup field or a plugin-set field) and check it directly, rather than querying live from the ribbon.

What actually happens on click

The rule only controls visibility and enablement — a separate function, wired up as the button's Action, is what runs when someone actually clicks it:

async function escalateToManager(primaryControl) {
  var formContext = primaryControl;
    var confirm = await Xrm.Navigation.openConfirmDialog({
        title: "Escalate this opportunity?",
            text: "Your manager will be notified and the record flagged for review.",
              });
              
                if (!confirm.confirmed) return;
                
                  await formContext.data.save();
                    await Xrm.WebApi.updateRecord(
                        "opportunity",
                            formContext.data.entity.getId().replace(/[{}]/g, ""),
                                { con_hasopenescalation: true }
                                  );
                                  
                                    formContext.ui.refreshRibbon();
                                    }

That last line matters more than it looks. Ribbon state doesn't automatically re-evaluate just because you changed a value in code — Dynamics re-checks your rules on load, on save, and on a few built-in events, but not every time an attribute changes. If your action changes something an enable rule depends on, call formContext.ui.refreshRibbon() yourself, or the button will sit in its old state until the user manually refreshes the page.

Where the JavaScript actually lives

Same rule as every other post in this series: the function has to be in a web resource that's part of your solution, and the Ribbon Workbench (or the classic ribbon customization) references it by $webresource: path plus function name — not by file path alone. A function that works perfectly when you test it in the console but does nothing on the actual button almost always means one of three things: the web resource reference in the ribbon XML is wrong, the function isn't actually included in that web resource, or you edited the JS but forgot to publish afterward — the same publish-and-cache trap from the ALM post shows up here too.

Testing rules without clicking through the UI fifty times

Because CommandChecker functions take a plain object shaped like formContext and return a boolean, they're some of the easiest ribbon-adjacent code to unit test — you don't need a real form, just an object with the methods your function actually calls:

test("escalate disabled when not the owner", async () => {
  const fakeForm = {
      getAttribute: () => ({ getValue: () => [{ id: "other-user-guid" }] }),
        };
          expect(await canEscalate(fakeForm)).toBe(false);
          });

This is the same pattern from the testing and debugging post — mock the pieces of formContext your function actually touches, and you can validate ribbon logic in milliseconds instead of navigating to a record, checking a button state, changing data, and reloading, over and over.

A short checklist before you ship a ribbon change

Did you use a Display Rule where you meant "hide it" and an Enable Rule where you meant "show it but grey it out" — not the other way around? Is any Web API call inside an enable rule cheap enough to run on every grid row, not just a form? Does your click-handler call refreshRibbon() if it changes something another rule depends on? And is the web resource holding these functions actually included in the solution you're about to export — the same ALM discipline from a couple of posts back applies just as much to ribbon code as it does to form scripts.

Next up

We've covered forms, data, agents, testing, deployment, and now the command bar. Next in the series: Business Process Flow JavaScript — reading and moving the active stage in code, and the specific quirks of scripting against a BPF versus a regular form.