Rule simulation

Audience tip

This article is relevant for Fraud Management SaaS users.

As an Admin, Senior Analyst, or Analyst, you can test on your past data whether your rule performs as intended by running a simulation. This allows you to see how the rule performs before deploying it.

As of July 2022, Fraugster offers simulations for a rule against your entire rule set. Previously you could simulate rules without any insight on the rule set performance. However, simulating a rule in isolation didn't offer a full picture.

Checking how one rule performs against other rules in the rule set gives a better analytical insight and allows you to optimize rule's performance before deploying it to production. The detailed metrics allow you to fine-tune your rules and thus make your rule set cleaner and more concise.

Simulation demo

We prepared a demo to show you the new rule simulation in action. Check it out!

Run a rule simulation

To run a rule simulation:

1. Go to Rules in the side navigation and open a rule you'd like to test.
2. At the bottom of the page, find the Ruleset simulation section. You can simulate over a chosen date range, merchant portfolio, and rule version. You can also choose to exclude certain rules from the simulation if you don't want their results to be shown in the metrics.
3. Select RUN. Check the results to see how the selected rule version performed on past data.

Please note

• You need at least 3 months of data to see reliable simulation results. Transactions need to receive async statuses of confirmed fraud to show meaningful results in simulations. For some transactions, fraud may be confirmed in a few days, but most transactions mature in a period between 90 and 120 days.
• You can simulate on data not older than 6 months. We're working on extending this limit.
• Simulations for bigger date ranges may take longer to complete.

Simulation results

Let's have a closer look at the information that shows the expected rule performance.

Collapsed view

The results of a ruleset simulation appear in a collapsed view. It gives the key details about the rule performance:

• Total hits: the number of transactions that hit this rule in the selected date range.
• Fraud hits: the number of confirmed fraudulent transactions that this rule hit in the selected date range.
• Precision: indicates how precise the rule version is. The higher the precision, the more accurate your rule version is.
  • Calculation for decline rules: number of fraud hits / total number of hits.
  • Calculation for approve rules: number of good transactions / total number of hits.
• False positive ratio (FPR): the ratio between the fraudulent and legitimate transactions.
  • Calculation for decline rules: number of fraud hits / number of good transactions (total hits - fraud hits).
  • Calculation for approve rules: good transactions (total hits - fraud hits) / fraud hits.
• Fraud catch rate (for decline rules): the number of fraudulent transactions hit out of the total number of fraudulent transactions in the date range.

Value vs. count

You can display count or value metrics for simulation of approve or decline rules results . Count refers to the number of transactions, while value refers to the monetary value of those transactions. The currency shown is determined by the currency of your Dashboard account. If you select to view the value results, the percentages are calculated from the transaction amounts.

Value results are often represented by big number, like a hundred thousand or a million. We round them up and abbreviate for better readability, for example 15.5k or 4.2m. To see the exact value, hover over the number.

If you simulate rules with another rule action – for example watch or manual review – you can choose in the dedicated drop-down whether to display those hits as approve or decline in the results.

Expanded view

Results of a simulation in the expanded view are available with a lot of detailed metrics. We provide an explanation for each metric for better understanding and transparency.

You can see all the hits that the simulation produced, whether they took the final decision or not. The two groups of hits are:

• Gamechanger hits: the hits that would have made a decision for those transactions if the rule had been in place at the time. These can be unique hits or hits that would have overturned other rules in place.
• Ineffectual hits: the hits that made no impact. Even though they hit a transaction, another rule made a decision for it. You can also check which rule exactly made the decision in this case. see what other rules are winning over your rule.

Export results

You can export simulation results in the collapsed view by selecting the export icon .

In the expanded view, use to open simulation results in a separate tab and see the corresponding transactions in a list. Export is available from the transaction list page also. Select in the upper-right corner → Export transactions.