LLM Test Bench
Last updated
Desktop test bench that runs test-case suites against a local llama.cpp server and compares the runs
Install with winget
$ winget install --id aleksandrreva-eng.llm-test-bench --exact --version 1.0.0Run in Command Prompt, PowerShell, or Windows Terminal. Prompts for any agreements.
Silent install command for LLM Test Bench
LLM Test Bench uses MSI (WiX). The silent install switches are /quiet /norestart.
msiexec.exe /i LLMTestBench-1.0.0.msi /quiet /norestart
For Intune admins
Stop chasing app updates. Pckgr patches them for you.
Automated application patching for Microsoft Intune. Pckgr keeps a curated library of 1,000+ apps continuously up-to-date in your tenant via Microsoft Graph - no manual repackaging, no chasing vendor sites.
Start free 30-day trialFrom $49/month. No credit card required.
Not on Intune? Pckgr RMM deploys and patches apps like this one on any Windows fleet. Free for up to 5 devices.
About
A Windows desktop application for evaluating local GGUF models served by llama.cpp. It scans a models folder, starts or attaches to a llama-server on 127.0.0.1:8080, and runs configurable test-case suites covering logic, code, format compliance, instruction following, generation speed and long-context handling. Each answer is graded by an automatic check - substring, regex, JSON schema, code execution, LLM-as-judge or embedding similarity - with skipped checks kept out of the score instead of counted as failures. Runs are stored in a SQLite history with per-case drill-down and side-by-side comparison, and can be exported as an HTML report or printed. Also includes a local agent mode that uses the tested model as a subagent, and reports prefill and generation speed, time to first token, model memory usage and layer placement from the server log.
Installers · v1.0.0
| Architecture | Type | Scope | Install | Download |
|---|---|---|---|---|
| x64 | MSI WiX | user | Direct |
Copy a command tailored to that specific architecture, type, and scope - useful when winget would otherwise pick a different default.
Security
No known CVEs for LLM Test Bench.
Coverage is best-effort and depends on a winget package mapping to an NVD CPE entry. Absence here is not a guarantee of safety.
Related apps
More from aleksandrreva-eng or browse llm, llama-cpp, gguf.