Optimize your AI agents with Compass

Compass helps you test your AI agents, measure the quality of agent conversations, track key performance signals, and get AI-powered recommendations to improve your agents over time. In your AI agent settings, use the Simulations tab to test out your agent before deploying it, or use the Performance and Replies tab to monitor and tune your AI agent.

To use Compass, you must:

• Have company admin permissions

• Have created an AI agent

Autopilot agents only: Use the Simulations tab in your agent settings to simulate how your agent will function against conversations. This lets you test and evaluate agent behavior, allowing you to activate it across inboxes with confidence.

Step 1

Navigate to the agent you’d like to test, select the Simulations tab, then click Create set. You can think of a set as a collection of related simulations.

Step 2

In the pop-up, enter a name for your set, then optionally select a Smart QA scorecard you want to use with the evaluation.

Important to know:

• The Smart QA scorecard is only available for when the agent is replying in the conversation.

• If the agent is not replying, the Smart QA evaluation will not apply, but the simulation will still evaluate the agent on other actions it takes using criteria created later.

Step 3

Select the set you just created, then click Create simulation. In the panel, select whether you want to test the agent’s behavior with one existing conversation (Option 1), or create simulations from multiple conversations at once (Option 2).

For the best results, we recommend selecting a varied sample of conversations if you want to test multiple scenarios together. This is so the agent's behavior is tested as closely as possible to the scenarios it will encounter in the inbox.

Step 4

You'll see a list of recent open conversations in the selected inbox(es). Hover over any conversation to preview its contents.

Step 5

Enter the conversation ID or ticket ID for a conversation you want to test with, or select any conversation in the list.

Step 6

Enter a name for the simulation. In the Conversation context section, you’ll see a summary of the scenario, along with the first inbound message from the conversation. 

Step 7

In the Simulation criteria section, create or select previously created criteria you want to test with your agent. To add new criteria, click Add, then select Simulation criteria. 

See the section below for tips on writing simulation criteria.

Step 8

Enter a name for the simulation criteria, then enter a description of the expected behavior. Click Generate scale to create pass/fail conditions, which you can then customize.

Click Create.

Step 9

Add additional criteria as needed, then click Create and run.

Step 10

When the simulation has finished, you’ll see the results in the list view. Click the simulation to see additional details.

Step 11

See the Reviewing simulation results section below.

Step 4

Enter the conversation IDs for conversations you want to in your simulation set, separated with commas. Each conversation ID creates one simulation. A conversation can only be used if it has at least one inbound message.

Click Create simulations.

Step 5

You’ll see the simulations in the list. Click Run set to run all the simulations against the set’s Smart QA scorecard.

Step 6

To add additional criteria to a simulation, hover over the simulation, click the three-dot menu, then click Edit. In the Simulation criteria section, create or select criteria you want to test with your agent. To add new criteria, click Add, then select Simulation criteria.

See the section below for tips on writing simulation criteria.

Step 7

Enter a name for the simulation criteria, then enter a description of the expected behavior. Click Generate scale to create pass/fail conditions, which you can then customize.

Click Create.

Step 8

Add additional criteria as needed, then click Create and run.

Step 9

When the simulation has finished, you’ll see the results in the list view. Click the simulation to see additional details.

Step 10

See the Reviewing simulation results section below.

Creating clear, specific criteria is essential for running accurate simulations and getting meaningful insights into your agent's performance.

Recommended criteria to start with:

1. Never claims an action it didn't take: The agent must never tell the end user something was done (e.g., a refund issued, a ticket escalated, an account updated) unless that action appears in the actions it took during the conversation.

2. Escalated when it should have: Where the agent couldn’t action the request, or the request needed a human decision, the agent handed off instead of answering.

3. Did not escalate when it shouldn't have: Where all user requests were resolved, the agent answered rather than escalating or handing off to a human.

Simulation criteria validator

When writing your simulation criterion, you’ll see a validator that automatically checks your content. If the criterion is too vague, isn’t about the agent, bundles several rules, relies on information the validator can’t see, or asks the simulation to take an action, you’ll see a warning. Once your criterion meets all standards, the status will update to “Good definition.”

The validator can/cannot see the following content:

Guidelines for writing good criteria:

1. Describe one agent action. Focus on what the agent does, not on the scenario or the customer.

2. Describe one behavior per criterion. Two closely related actions are fine: "asks for the order number and the domain". Don't bundle understanding, tone, accuracy, routing, and formatting. Split them into separate criteria.

3. Ensure criteria can be verified within the simulation. Identify the visible evidence: a message, note, tag, Connector call, assignment, or how the conversation ended. 

4. Don't score the customer. Asking if the customer is satisfied generates unreliable results. Describe the agent's action instead: "The agent sends the tracking number." 

5. Leave tone and formatting to the Smart QA scorecard. Criteria are for behaviors you can answer yes or no to. The scorecard is for how replies read.

When you select a simulation, you’ll see a preview of the conversation along with the simulation results. 

In this menu you can also:

• See past run history

• Re-run the simulation

• Use the three-dot menu to edit or delete the simulation

Arrow indicators highlight differences between the current run and previous runs to track progress or regressions resulting from your changes.

In the list view, click Run set to run all simulations at once. When finished you’ll see a summary of results in the list. You can also hover and select specific simulations you want to run.

Export simulation runs to a CSV file to review results in a spreadsheet, share them with your team, or compare runs over time.

Step 1

Go to the agent's Simulations tab. You can click Export from two places:

• The simulation sets list exports runs from every set for this agent.

• Inside a set exports runs from that set only.

Step 2

Select a date range: Today, Yesterday, Last week, Last month or custom dates. By default the range is the last 7 days. Click Export.

Step 3

You'll see a progress banner at the bottom of the page. When the file is ready, click Download. You'll also receive the file by email.

Important to know:

• One export holds the 1,000 most recent runs in the selected date range.

• Runs still in progress are not exported.

• You can run one export at a time. Wait for it to finish before you start another.

Each row is one simulation run.

• Passed: every evaluated criterion passed.

• Failed: at least one evaluated criterion failed.

• Completed: the run finished but no criteria were scored. For example, there were no criteria, or the agent didn't reply and only Smart QA criteria applied.

• Errored: the run didn't finish. The Error reason column says why.

• Cancelled: the run was cancelled.

• Customer got what they needed: the simulated customer ended the conversation as resolved.

• Customer gave up: the simulated customer ended the conversation for any other reason.

• Escalated to a teammate: the agent handed the conversation to a human. This happens when it declines to reply, its reply is empty or held back as a draft, or an action it requested fails. It also shows when the agent hits an internal error.

• No reply needed: the agent decided no reply was needed, and any action it requested succeeded.

• Turn limit reached: the run hit the limit of 10 customer turns.

• AI teammate did not reply: only on older runs. Today, a reply held back as a draft counts as Escalated to a teammate.

• Blank: the run errored or was cancelled.

Compass includes the following quality signals to help you review AI agent performance. Quality is graded as No Signal, Good, Okay, or Poor.

Smart CSAT

When you set up an AI agent, Smart CSAT is automatically enabled and runs on all AI agent replies in resolved conversations. Human teammates will not be evaluated.

Smart QA

When you set up an AI agent, Smart QA is automatically enabled and runs on all AI agent replies in resolved conversations. Front will set up a default Smart QA scorecard and rule that runs 24 hours after a conversation is resolved. The default Smart QA scorecard includes these criteria: Brevity, Tone, Solution offered, Comprehension, however you can modify these in your Smart QA settings.

To configure the criteria used for Smart QA evaluations on your AI agent, adjust the scorecard in your Smart QA settings. Human teammates will not be evaluated.

Draft edit level

Front automatically detects if an AI draft was edited by a human teammate before it was sent, and calculates an edit level. Drafts can be sent as-is, lightly or heavily edited, or discarded. Draft edit level also contributes to the AI inferred feedback value for a conversation.

Teammate feedback

Human teammates can provide positive or negative feedback on AI agent drafts and replies directly in conversations using the thumbs up/down icons. Teammates are prompted to add additional details, which admins can view in the agent’s settings.

AI will also infer feedback on drafts and replies where it didn’t quite meet the mark - there was a correction of your AI Agent’s response in the conversation or a draft was heavily edited, giving you more signal on poor quality replies without relying on manual teammate feedback.  

Each feedback item acts as a signal that feeds directly into creating Opportunities in the Performance tab to enhance your AI agents. Negative teammate feedback will trigger a ‘Poor’ quality signal on AI replies, and positive teammate feedback will trigger a ‘Good’ quality signal on AI replies.

Use the Performance tab in an agent’s settings to monitor its activity once it begins handling conversations. You can find this tab by navigating to company settings, selecting Agents in the sidebar, then selecting the agent you’d like to review.

In the Performance tab, you can:

• Get complete visibility into agent performance: The Performance tab consolidates quality signals at a glance that previously lived in separate places — Smart CSAT scores, Smart QA scores, involvement and resolution metrics, reply ratings, and failure patterns.

• Assess agent effectiveness: Analyze performance metrics to address key questions, such as:

  • Is my AI agent healthy enough to expand its coverage or graduate from generating drafts to sending auto-replies? 

  • What is the single most impactful thing I can improve right now?

• Identify areas of improvement: The Opportunities section evaluates poor quality AI replies and converts them into actionable steps.

  • Each item shows a summary of the issue, the gap type, the and the reply volume affected. Click into the opportunity to view suggested actions and additional details.

  • Opportunities generated for all AI agents include updates to knowledge or agent instructions.

Use the Replies tab in an agent’s settings to review all messages generated by it, along with quality signals. Both drafts and sent auto-replies are included. Use the filters at the top to focus on specific message characteristics such as the inbox, reply type, topic and quality signals.

Select a message from the list to view more details in the AI reply panel, including:

• Messages and drafts: View the inquiry and AI auto-replies or drafts

• Draft edits: Toggle on Show edits in a message to view the human edits made to the AI draft 

• Quality signals: View all quality signals that indicate if a draft or reply is Good, Okay, or Poor. Based on the following: 

  • Feedback: Rating and/or comment provided by human teammates, or inferred by AI

  • Edit level: Whether an AI-generated draft was sent as-is, lightly or heavily edited, or discarded

  • QA criteria: Cited Smart QA scores for scorecard criteria with references to this message

  • CSAT rating: Smart CSAT score inferred by AI with reference to this message

• Sources used: View all inputs AI used to generate the reply

  • For Autopilot agents, this includes knowledge articles, notes, facts from similar conversations, agent instructions, Playbook used, and tone guidelines

  • For external agents, this includes the knowledge sources passed by the MCP 

In the inbox, admins can access a preview of the reply quality and a link to the Replies tab to see more details. 

In addition to Compass features, admins can navigate to the Agents report in Front Analytics for a comprehensive look at all AI agent activity. Use the report to evaluate resolution data, customer satisfaction and engagement, and performance over time for all agents across your workspaces.

If you’re using Compass, but don’t have access to Smart CSAT and Smart QA, only your AI agents are evaluated in eligible conversations it replied to. Smart CSAT/QA will not run on human teammates.

A simulation set is a named group of related simulations covering the behaviors an agent is expected to handle. Admins can run a single simulation, or the whole simulation set at once. The set serves as a stable baseline you re-run after every change to the agent configuration.

No. Replies are not sent to real conversations/customers, no live conversation is modified, customers do not see test results, and analytics are not impacted.

Not every simulation may require a reply. If an agent doesn’t reply in a simulation, simulation criteria are still evaluated, which is how an escalation or a no-action case is flagged. Smart QA scorecards only evaluate replies; if the agent does not generate a response, Smart QA criteria are not evaluated.

You can run a simulation as many times as you like, but each conversation within a simulation is capped at 10 back and forth exchanges with the agent.

No. Simulation criteria are evaluated using the simulated conversation and the actions the agent took during it. They don't have access to the agent's knowledge sources or Playbooks, so they can't confirm whether an answer matches your help center articles or internal procedures.

Instead, write criteria about behavior that's visible in the conversation. A criterion like "The agent gives our correct refund policy" can't be evaluated reliably, while "The agent never tells the customer a refund was issued unless it took that action" can. To check the accuracy of an answer itself, review the simulated conversation in the simulation results.

Exports include finished runs launched in the selected date range, up to the 1,000 most recent. Runs still in progress are excluded. To export older runs, export a shorter date range.

Yes, but not directly in the Replies tab. You can quickly navigate to the conversation by clicking Open conversation in the AI reply panel, then either discard that draft or step in to send a reply yourself.

External agents are included in all plans. Front Autopilot agents require access to the Autopilot add-on. Compass is included with the use of both external and Autopilot agents.