My first decision model: trained in 6 minutes, 72% on an independent test

I trained my first decision model this week. The first try was 30 steps on a Tesla T4 and took almost 24 minutes. The second ran on the GPU in my own workstation and took 6.

A decision model does not write anything. It reads some text, a question and a list of options, and answers with a probability for each option. That is all it does, and it is the reason I wanted to try one.

• 71.85% accuracy on the official test split, 2000 answers the model had never seen
• 0.78 held-out accuracy as reported by Unsloth, on its own holdout
• 6 minutes to train
• 102 ms per call with five questions
• 3.6 GiB on the GPU at peak

Two accuracy numbers, so I should explain. The first came from Unsloth’s own holdout. The second came from a script I ran afterwards on the official test split. The second is the one I would quote, and Unsloth’s 0.78 would have made a nicer headline. The rest of this post is how I got both, how to read the training dashboard, and where the model is wrong. Read on 🙂

What a decision model is

You give it three things: a state (the text to decide on), a question, and a list of options. It reads all of it in one pass and returns a probability for each option. It never writes text. Models such as Jev from TypeSafe AI and its open reimplementation Kev work this way, and Unsloth’s trainer uses a small scoring head of the same design as Cloudflare’s Clef.

The Unsloth interface offers three question types:

• Yes / no: does the customer ask for a refund?
• Choice: which team should handle this ticket?
• Score: how urgent is this, on levels you define?

You can ask a chat model the same things. It will write a paragraph, and your code then has to find the answer inside it. That costs output tokens, adds latency and sometimes fails to parse. A decision model skips the writing, so the answer is always one of your options.

The part I like most is the probability. At 95% I let the system act. At 55% the message goes to a human, or it asks a clarifying question. A plain text answer cannot give you that dial. In the Studio Try panel one answer took 346 ms from 224 input tokens. In my scripted test a call with five questions took a median of 102 ms.

The setup

Same method, two runs. A single epoch of about 6 minutes lifted Unsloth’s held-out accuracy from 0.41 to 0.78, and my own test on 2000 answers gave 71.85%.

Per step, the local GPU was about 9 times faster than the T4 (roughly 48 seconds against 5 seconds). The two runs used different subsets, different base models (Qwen3.5-4B and Qwen3-4B) and different held-out sets, so I read the gap between them as encouraging, not proven.

Run 1: 30 steps on a T4

The first run took 23 minutes 54 seconds. Unsloth reported held-out accuracy going from 0.37 to 0.74, with a calibration error of 0.09.

Settings:

• Model: Qwen3.5-4B, trained with QLoRA (a 4-bit base model with small adapters on top)
• Train as: Decision model
• Dataset: LocalLLaMA/typed-decisions, subset agent_trace_observability
• Batch 8 x 2 = 16, learning rate 1e-4, 30 steps
• Hardware: Tesla T4, 15 GiB VRAM

Then I gave it a support ticket: "I was charged twice for my Pro plan this month and the second charge still shows as pending. Please refund one of them." The question was which team should handle it, with four options.

The answer is right. But 21% for sales on an obvious refund request tells me the model was not very sure yet. Fair enough, it had seen 30 steps 😛

One mistake: I trained on the agent_trace_observability subset and then tested on support tickets. Unsloth’s guide recommends the all subset for typed-decisions. Between the runs I read the guide properly.

Run 2: 70 steps on my own GPU

After reading the guide I changed two things. I set the subset to all, which the guide recommends for typed-decisions, and I moved training to the 16 GiB GPU in my own workstation. One more thing differs: run 2 used Qwen3-4B, not Qwen3.5-4B.

The run was 70 steps, which is exactly one epoch. Loss fell from about 1.2 to about 0.8, and the GPU read 63 °C at step 44. It took 6 minutes 1 second at 0.19 steps per second. Unsloth reported held-out accuracy of 0.78 and a calibration error of 0.06. Great success!

A calibration error of 0.06 means the model’s stated confidence differs from how often it is right by about 6 percentage points on average. That is what makes the confidence dial usable in practice.

For context, Unsloth’s guide reports held-out accuracy of 78% for Qwen3.5-0.8B and 81% for Qwen3.5-2B on its own test mix. My 0.78 is in the same range, but it is a different test, so I am not claiming a comparison. The next section is the test that matters.

How to read the training dashboard

Loss is not accuracy. If you take one thing from the dashboard, take that. These are the numbers that matter, and what each one told me.

• Loss: the average surprise per token, or how little probability the model gave the correct answer. Lower is better. Pure guessing scores the natural log of the number of options: 0.69 for yes/no, 1.39 for four options, 1.61 for five. Averaged over the mix of questions in the test split that works out to about 1.2, which is where run 2 started. It ended at 0.81. I am assuming the dashboard averages per question, so take that with a pinch of salt.
• Smoothed loss: the orange line. The raw blue line is noisy because each step sees only a few examples, so read the orange one.
• Learning rate: how big each weight update is. The chart shows a warmup to a peak, then a straight decay to nearly zero. The tiny number you see at the end of a run is just the tail of that schedule.
• Grad norm: how hard each batch pushes the weights. It spikes on hard batches. Mine moved between about 0.05 and 1.9 with no explosion. I would only worry about values of 10 or more, or NaN.
• Eval loss: the same loss on examples the model did not train on. Run 2 drew a single point at step 70, about 0.89, close to the training loss of 0.81, so there was no sign of overfitting. One point is not a trend.
• Epoch: one full pass over the dataset. Run 1 made 1.76 passes and run 2 made exactly one.
• VRAM and power: run 2 used 9.02 of 15.92 GiB and drew 297 W of a 300 W limit at 99% utilization. That means the GPU was the bottleneck, which is what you want in training.

The numbers that count are the held-out accuracy before and after training, and the calibration error. Unsloth reports both when the run finishes. Calibration error is the average gap between how confident the model says it is and how often it is actually right.

The test that matters: 2000 answers on the official split

Unsloth’s 0.78 was measured on its own holdout of 400 decisions, which was most likely carved out of the training data. The official test split is data the model never saw. Neither number is wrong. They measure different data.

So I loaded the trained model in 4-bit, took every row of the typed-decisions test split and asked all of its questions, one row at a time. I had planned 1000 examples. The test split only has 400 rows. Each row carries 5 questions, which makes 2000 individual answers. No test input is an exact copy of a training input, and nothing was skipped. Unsloth’s own evaluate function gave 72.0% (1440 of 2000), within three answers of mine.

Result: 71.85% (1437 of 2000 answers, 95% interval 69.8% to 73.8%). The strongest simple baseline reached 46.1%.

On score questions the average miss was 0.354 levels, and 97.5% of answers were within one level. By workflow, accuracy was 78.4% for customer service, 76.6% for invoice processing, 69.8% for security incidents and 62.6% for agent trace observability.

Against simple baselines

The part I find most interesting: the confidence dial

Can you trust the confidence? Mostly, yes. The calibration error on the test split was 0.026, and the average confidence of 0.742 sits close to the actual accuracy of 0.719. The table shows where it drifts: in the middle bins the model is a little overconfident, and at the top it is almost exact.

Seven rows are shown because the three bins below 0.3 are empty. The calibration error uses all ten.

That is the dial I mentioned earlier. If you act automatically only when the confidence is high, the accuracy of what you automate goes up quickly:

(445 + 283 + 415 is 1143, not 1145. I think the two missing answers sit exactly on 0.7, but I have not checked.)

In practice, I could let the model handle about a fifth of the answers on its own at 94.6% accuracy, and send everything else to a person.

The same numbers, interactive

Click through the tabs, hover over the points of the calibration chart, and move the slider to see what happens when the model only acts on answers it is sure about.

Speed and memory

The whole-GPU figure includes the desktop and driver overhead, so the model itself needs a little less. At roughly 3.6 GiB at peak it leaves most of a 16 GiB card free.

The local GPU also made each step about 9 times faster. At the T4's speed, 70 steps would have taken about 56 minutes. That is arithmetic, not a measurement (so don't quote me 😛).

Where it fails

Six of the ten most confident wrong answers were the same invoice question, "duplicate". The correct answer was true, and the model said false with 95% to 98% confidence. That looks like a systematic gap, not noise, and a good place to start with more training data.

What is not perfect

• The 2000 answers come from only 400 rows, and answers within a row are related, so the real uncertainty is wider than the interval above.
• The correct answers are the top choice of several human annotators who often disagree (on average the labels put only 66% of the weight on the top answer), so even a perfect model could not score 100%.
• I did not run the untuned base model on this test, so the 0.41 starting point is still Unsloth's number from its own holdout.
• One run each. I did not repeat them with other random seeds, so I do not know how much of the 4-point gap between the runs is noise.
• Not a clean comparison. The runs used different subsets and different base models, and only run 2 got the independent test.
• One epoch is short. Unsloth's own results use 2.
• No real traffic yet. A good benchmark number is not the same as good routing on messy real messages.

This is a demonstration and not a benchmark. It ran on a personal workstation.

What I will try next

• Follow the guide's recommended settings (2 epochs, LoRA rank 16, learning rate 2e-4) and change one setting at a time, so I know what moved the accuracy.
• Build a small dataset of my own. Real labeled messages should beat generic data for a specific task. One candidate is triaging shopper messages for an e-commerce assistant into product, order, returns or human handoff, with a refund yes/no and an urgency score decided in the same pass.
• Use the confidence dial: act automatically when the top probability is high, and escalate when it is not.
• Serve the model through the Decision API button and call it from my own code.

Reproduce it

You can repeat this in under an hour, and in minutes on a decent local GPU.

1. Install Unsloth Desktop, or the Studio web UI. Training needs an NVIDIA GPU on Windows or Linux, or you can use Unsloth's free Colab and Kaggle notebooks.
2. Open the Train tab, pick a 4B Qwen model (run 1 used Qwen3.5-4B, run 2 used Qwen3-4B), and set Train as to Decision model.
3. Choose the dataset LocalLLaMA/typed-decisions and set Subset to all.
4. Set the number of steps. The guide suggests 60 for a quick try, and says Qwen3.5-4B reached 76% in about 10 minutes on an L4.
5. Click Start Training, then read the held-out accuracy and calibration error when it finishes.
6. Click Use in Decision API, then try the model in the Try panel with your own text, question and options.
7. If the GPU is shared with something else, such as a local LLM server, stop that first.

Takeaways

• A decision model returns a choice and a probability, not text, and the probability is what makes it useful.
• A tiny run moved Unsloth's held-out accuracy from 0.41 to 0.78, and my own test on 2000 answers said 72%.
• Watch held-out accuracy and calibration error. Loss alone does not tell you the model is good.

Read the held-out accuracy first. The loss curve can wait.

Have fun! That's all folks 😀

Links

• Train your own Decision Model with Unsloth
• Unsloth requirements
• SanSi: A Looped Typed Decision Model for System 1.5 Thinking