Skip to content
MouseCat
← Back to Blog

Announcing Morphy, MouseCat's System One Model, and FraudBench, a New Fraud Benchmark

Today we are announcing Morphy v0.1.0, MouseCat’s System One model, and MouseCat FraudBench, our first proprietary fraud benchmark. On the first FraudBench dataset, Morphy’s predictions were better than all six of the off-the-shelf models we tested, with about 10% less error than TypeSafe’s Jev and 7% less than Cloudflare’s Clef.

Morphy: Morphy is a System One model. System One models answer structured questions and return a probability for each answer within milliseconds, much faster than a “system two” model like Claude. Morphy returns the probability of risks like fraud, scams and account takeovers, based on information like a user’s profile, payments, sessions and product activity. Like Claude, Morphy can understand a wide variety of unstructured content, limiting the work needed to “featurize” information before providing it to Morphy. Also, because it answers in milliseconds, Morphy is suitable to run on every transaction event, constantly scanning for signs of high-risk behavior. A fraud team can use that probability to approve/block/friction a transaction or to flag it for review. For example, a team can hold a withdrawal for review when the probability passes a threshold the team sets.

FraudBench: FraudBench’s first dataset is a proprietary set of 1,600 labeled cases built from real production deposit and withdrawal activity. It has transaction-level fraud labels based on a chargeback, a stolen-card decline or an account block. We ran Morphy, six off-the-shelf System One models, and a classic statistical fraud model as a baseline on this dataset and compared their performance.

First FraudBench dataset

The first MouseCat FraudBench dataset has 1,600 labeled cases from opted-in customers’ production deposit and withdrawal activity over the past year. Each case consists of a user and a point in time at which their transactions are being evaluated (one of the deposit or withdrawal events’ timestamps). The case contains information recorded at or before the request: the user’s profile, their deposits with payment details and any decline codes, their withdrawal requests, their activity on the platform, their sessions with device and location, and product events. The label is never included in the case.

A user’s transaction is labeled fraudulent if it has one of three signals: a chargeback or dispute on a deposit, a failed deposit that the card issuer declined as a stolen, lost, closed or blocked card, or a manual fraud intervention (i.e., withdrawal block, payment method flag or manual ban on the account). A user’s transaction is labeled legitimate if they have none of these signals and no weaker fraud signal, such as an issuer fraud decline or a fraud rejection from the payment processor. That gives 126 fraud cases and 1,474 legitimate cases, a fraud rate of 7.9%. All of the transactions are beyond the p99 chargeback maturity window.

Questions and model inputs

Every model, including Morphy, received the account’s activity at the time of the transaction and was asked the same 40 yes/no questions, such as “Does it look like the user is card testing?” Each model returns a probability of yes for each question.1

For each off-the-shelf model, the answers are then passed to a statistical model on top, which produces one predicted probability of fraud. The statistical model is a generic machine learning algorithm. We evaluated Morphy separately on the same cases. Forecasts were fit on earlier data and scored on later data.

ColorModelDescription
TypeSafe Jev 1.13.0The original System One model released by TypeSafe
TypeSafe Jev 1.14.0Newer Jev version, released October 14, 2026
Cloudflare ClefLarger model in Cloudflare’s open-weight decision family
Cloudflare Clef-flashSmaller Clef variant designed for latency-sensitive decisions
Strands DeciderFrom Amazon’s Strands Labs
OpenAI Decisions APIBased on OpenAI’s gpt-6-luna
Statistical fraud modelReference forecast
MorphyMouseCat’s System One model

Evaluation

We score each forecast with the Brier score, which measures how far its predicted probabilities are from what happened. A lower score is better. A forecast gets a low Brier score if its probabilities match the observed fraud rates and separate fraud from legitimate transactions. We report each forecast’s error relative to the statistical model, so 10% means 10% less error than the statistical model and zero means the same error.2 Each result comes with a 90% interval. An interval that contains zero means the forecast’s error cannot be told apart from the statistical model’s on this dataset.

Results

Brier error reduction relative to the statistical model, with 90% intervals, for Jev 1.13, Jev 1.14, Clef, Clef-flash, Strands Decider, OpenAI Decisions, the statistical model and Morphy. The six off-the-shelf models sit near zero with intervals that contain zero. Morphy is at about 10% with an interval entirely above zero.
Figure 1. Brier skill relative to the statistical model. Morphy is MouseCat’s System One model. Zero means the same error as the statistical model; higher is better. Intervals are 90% paired bootstrap intervals. The six off-the-shelf forecasts use judgments alone.

On the first FraudBench dataset, the six off-the-shelf System One models perform about the same as the statistical model. Cloudflare’s Clef and OpenAI’s Decisions API produce the best off-the-shelf estimates, reducing Brier error by 3.4% and 1.7% relative to the statistical model. Clef-flash is the weakest, with 6.1% more error, and both Jev versions are close to zero. For all six, the 90% interval contains zero, so these minor differences in performance are indistinguishable from noise. Morphy reduces Brier error by 10% relative to the statistical model. Its 90% interval lies entirely above zero. Compared directly with the off-the-shelf models, its Brier error is about 10% lower than either Jev version and 7% lower than Clef.

In plain terms, when Morphy gives a group of transactions a 20% chance of fraud, the share that turns out to be fraud is closer to 20% than it is for the other models, and Morphy gives fraud cases higher probabilities relative to legitimate ones.

Latency and cost

We measured latency and cost per case.3

ModelMeanp50p90p99Cost per 1,000 cases
TypeSafe Jev 1.13.092 ms87 ms116 ms159 ms$0.19
TypeSafe Jev 1.14.089 ms86 ms111 ms155 ms$0.19
Cloudflare Clef778 ms681 ms973 ms3,407 ms$1.25
Cloudflare Clef-flash653 ms548 ms1,046 ms2,672 ms$0.20
Strands Decider334 ms287 ms400 ms1,152 ms$1.15
OpenAI Decisions API364 ms308 ms637 ms966 ms$0.88
Morphy104 ms95 ms123 ms306 ms$0.31

Jev is the fastest. Morphy is about 10 milliseconds slower than Jev at the median and the 90th percentile, and about twice as slow at the 99th percentile. Jev and Clef-flash are the cheapest at about 20 cents per 1,000 cases. Morphy costs 31 cents.

Clef-flash costs about the same as Jev but is about six times slower at the median, at about 550 milliseconds. Clef is the slowest at the median, at about 700 milliseconds, and the most expensive, at $1.25 per 1,000 cases. OpenAI’s Decisions API and Strands Decider each take about 300 milliseconds at the median, and cost 88 cents and $1.15 per 1,000 cases.

Scatter plot of Brier error reduction against 90th percentile latency for the six off-the-shelf models and Morphy. Morphy is at about 10% and 123 milliseconds. The two Jev versions are near zero at about 115 milliseconds, Strands Decider is at 400 milliseconds, OpenAI Decisions at about 640 milliseconds, and Clef and Clef-flash at about 1,000 milliseconds. A dashed line joins Jev 1.14 and Morphy.
Figure 2. Brier skill against 90th percentile latency. Better performance is in the top-left corner. The horizontal line is the statistical model. The dashed line joins the models that no other model beats on both latency and Brier skill.
Scatter plot of Brier error reduction against cost per 1,000 cases for the six off-the-shelf models and Morphy. Morphy is at about 10% and 31 cents. The two Jev versions are near zero at 19 cents, Clef-flash is at about minus 6% and 20 cents, and OpenAI Decisions, Strands Decider and Clef are between 88 cents and $1.25. A dashed line joins Jev 1.14 and Morphy.
Figure 3. Brier skill against cost per 1,000 cases. Better performance is in the top-left corner. The horizontal line is the statistical model. The dashed line joins the models that no other model beats on both cost and Brier skill.

Brier score decomposition

Morphy’s predictions score best both because they are closer to the observed fraud rates than any other forecast’s, and do a better job separating fraud from legitimate transactions. We found this by splitting the Brier score into its components.4 Miscalibration (MCB) measures how far the predicted probabilities are from the observed fraud rates, and lower is better. Discrimination (DSC) measures how well a forecast separates fraud from legitimate transactions, and higher is better.

Scatter plot of discrimination (DSC) against miscalibration (MCB) for the eight forecasts, with diagonal lines of equal Brier score. Morphy is in the top-left corner with the highest discrimination and lowest miscalibration. The statistical model is furthest right. Clef and OpenAI Decisions sit to the left of the Jev models, Strands Decider and Clef-flash.
Figure 4. Brier score decomposition. Better performance is in the top-left corner. Diagonal lines are equivalent Brier scores. Squares show judgments alone, the diamond shows the statistical model, and the circle shows Morphy.

The statistical model has the worst calibration but better discrimination than any of the off-the-shelf decision models. Among the off-the-shelf models, Clef has the best discrimination and Clef-flash, the smaller model in the same family, has the worst. The two Jev models perform similarly.

Relative to the statistical model, Morphy adds 0.0015 of discrimination and removes 0.0026 of miscalibration, both in Brier-score units. Its discrimination is higher than the statistical model’s and higher than every off-the-shelf model’s. Its calibration error is the lowest of the eight forecasts, slightly below Clef and OpenAI’s Decisions API.

Closing

The dataset in this post is the first in FraudBench. We plan to add more datasets and to report results for new System One models as they are released.

Morphy is already being used by several MouseCat customers.

Reach out to us for early access:

Talk to MouseCat
  1. Performance on individual questions differed between models. In one of the exploratory questions while setting up the evaluation, we asked whether the latest payment occurred at night in the user’s location. When given a timestamp in UTC, Jev’s AUC was 0.77 compared with Clef-flash’s 0.56. When local time was supplied explicitly, both scores were 1.00. ↩
  2. The Brier score is the mean squared error of the predicted probabilities. The reported value is Brier skill: 1 − Brier score / Brier score of the baseline statistical model. Intervals are 90% paired bootstrap intervals from 10,000 resamples of case families, with the same resampled cases used for every forecast. ↩
  3. Latency is per case: the time from sending one request with the case and all 40 questions to receiving the full response, measured from a single client over the public internet, one case at a time on an already-open connection. Cost is the average input tokens each provider reported per case, multiplied by its list price per million input tokens: $0.042 for Jev, $0.24 for Clef, $0.038 for Clef-flash and $0.10 for OpenAI’s Decisions API. None of the four charges for output tokens. Strands Decider has no hosted API, so it was hosted on a dedicated NVIDIA B200 GPU on Modal, and its cost is the GPU’s hourly price at 50% utilization. ↩
  4. There’s a long history of Brier score decompositions into reliability and resolution components. We used the modern CORP decomposition from Dimitriadis et al. and their implementation in the {reliabilitydiag} package. ↩