The same typed questions, the same inputs. Jev (TypeSafe, paid) runs on OpenRouter's Decisions API. Laya (ConvAI Innovations, open source) runs locally with pip install laya, zero-shot and not fine-tuned. Each answer is marked against the answer we expected.
Laya ran on a shared 4-vCPU server with no GPU. Laya's published 33 ms is on a T4 GPU. Jev's time includes the network round trip to OpenRouter.
Laya's headline 0.766 accuracy comes from a checkpoint fine-tuned on its own benchmark. Out of the box it scores about 0.36. Fine-tuning it on your own labelled messages is what would close the gap.
Yes/no counts as correct when the probability of yes lands on the expected side of 0.5. A pick-one counts when the top option matches. A score counts when its rounded level matches. The small sample of tasks shows direction, not a benchmark.