Muszka Guard

Results

Muszka: the released INT8 file, sensitive mode (windows), threshold 0.5. Holdouts, PL-Guard, GadziJęzyk and false-alarm tests - model only, no regex; exam and critical list - with regex, as deployed. We compare with the model Muszka started from - Bielik-Guard "Sójka" 0.1B and 0.5B by SpeakLeash, run as published (fp32, up to 512 tokens, no windows). Exception: for PL-Guard we also show Sójka run by us in the same 128/32-token windows as Muszka.

Holdouts, PL-Guard, GadziJęzyk, Roget and the exam were set aside before training (exceptions: 3 exam items and 7 short holdout v1 texts were in the data of one of the two averaged checkpoints). The profanity test and the word audit measure coverage - their words were in training. Terms (AUROC, F1, holdout…) are explained in the glossary.

Source of all numbers in this section: the Muszka Guard 0.1B v1.1 model card on Hugging Face and the PDF report.

The tables below are wider than a phone screen - swipe them sideways.

Sealed final exam

A separate set, generated by Bielik and labelled by a Bielik judge before measurement (the standalone insults were picked by hand), measured once on the chosen model - so the model cannot be tuned to the test. The table shows the percent of texts blocked. For "block" rows higher is better; for "allow" rows lower is better.

Muszka is clearly better on obfuscated profanity and on par for plain vulgar messages. It is clearly worse than Sójka on standalone insults outside the profanity lexicon used in training (some of these words, e.g. "palantem", appeared in training data labelled innocent) - see known limitations. On innocent messages with look-alike words Muszka blocks more than both Sójka models (4.8% vs 2.8% and 0.8%). Note: after measurement we saw that the exam builder (a Bielik-based judge) labelled some non-vulgar messages as vulgar, so the true rate for the first row is likely higher; we did not edit the sealed exam after measuring.

PL-Guard (NASK) - Polish safety benchmark

A public test prepared by NASK: 14 categories, with both prompts and model responses. We report binary macro F1 (safe / unsafe, from 0 to 1). We split the test into two fixed halves: one was used for choosing settings, the other - the report half - for reporting. The report half was computed in every gate run (5 candidates) and the gate threshold used the whole set; the candidate was not chosen by it - the released model scores lowest of the last three on it. "Adv" is the adversarial part. Numbers in brackets: Sójka run by us in the same 128/32-token windows as Muszka, for a fair comparison (Sójka itself does not use windows and reads up to 512 tokens).

HerBERT-PL-Guard was trained on data prepared by the PL-Guard authors, with the same taxonomy - which may explain the gap. Muszka's training data has no overlap with PL-Guard (checked).

Held-out tests

Test sets set aside before training (exception: 7 short holdout v1 texts were in the data of one of the two averaged checkpoints). Holdout v1: Jigsaw test comments (translated with Bielik 11B) and 36 attack and innocent prompts written in Polish for the muszkaowocowa.pl chat; holdout v2: WildGuardMix test prompts (translated with Bielik 11B). Recall - how many harmful texts were caught; false alarms - how many innocent ones were blocked. GadziJęzyk is a set of native Polish, human-written harmful prompts (Prof. Jerzy Surma, SGH).

An honest caveat: v1 and v2 are the test splits of datasets whose training splits Muszka used (translated by Jan Sowa) - same domain for Muszka, not for Sójka. GadziJęzyk was excluded from Muszka's training. Adversarial prompts are mostly longer than one window, so Muszka's windows matter here; Sójka was not measured in windows on this set. The gain in caught attacks comes with more false alarms - the cost of sensitive mode: 11.5% of innocent prompts on v2 and 20.2% of innocent adversarial prompts (Sójka 0.1B: 6.1% and 3.8%).

Profanity and obfuscation

Test built from Jan Sowa's lexicon of Polish vulgar words and phrases, every inflected form, 13 kinds of obfuscation. Muszka v1.1 was trained on all word families, so this measures coverage. Generalisation to unseen families, measured on an earlier model with 30% of families held out: 87.2% of all variants (forms and obfuscations) blocked; plain forms 90.3% (release gate, INT8). Innocent words in the same unusual spellings are blocked in 0.5%-6.7% of cases depending on the spelling (u→ó 6.7% - above the 5% gate threshold).

Table: all three models in full precision (PyTorch fp32), profanity test v2; the released INT8 file scores up to 0.5 pp lower (release gate, test v3).

Other checks

A good filter doesn't only catch bad things - it must not get in the way of ordinary conversation. Here we check false alarms on everyday texts, and whether Muszka catches harmful replies of the models themselves.

Speed and memory

Muszka runs on an ordinary CPU, no GPU needed. The INT8 file (126 MB, 120 MiB) is 4 times smaller than full fp32 precision (498 MB) and about 2-3 times faster (median), about 2 times on long texts. p50 is the typical message time (half are faster), p95 the "near-worst" one (95 in 100 are faster). Measured on the released file, sensitive mode, one message at a time, 500 texts from the test sets (median 38 tokens), ONNX Runtime 1.30 with default threads (all cores), tokenization included, no regex.

Source: v1.1 model card; the p95 column comes from the raw measurement results (liczby_v1_1.json). INT8 vs fp32 decision agreement: 99.0% on 300 test texts.

Roadmap for v1.2

The next version starts with the biggest gap in v1.1 - standalone insults.

1. 1

   A broad lexicon of insults

   Insults with inflections and obfuscations. Fewer "innocent single-word" training anchors, which weakened what Sójka already knew about insults. A new exam section with insults outside the profanity lexicon used in training.

2. 2

   Ambiguous words in context

   Larger sentence-pair tests: the same word at school and in an insult.

3. 3

   Muszka + judge

   Blocked messages optionally re-checked by a small language model (Muszka-Instruct 1.5B) that releases false alarms. The filter stays sensitive; the judge reduces false blocks.

4. 4

   An in-browser version

   Runs locally, on the user's device - the text doesn't have to go anywhere.

We are also considering a larger variant, "Muszka 0.5B", on PKOBP/polish-roberta-8k (a suggestion from LinkedIn). The priority stays the same: better to block an innocent text than to let a harmful one through.

How to use it

For developers: the model is on Hugging Face under Apache 2.0. It needs only onnxruntime, tokenizers and numpy - no PyTorch, no GPU.

# pip install onnxruntime tokenizers numpy huggingface_hub
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("Fibogacci/muszka-guard-0.1b-v1.1",  # only the 4 files needed (~130 MB), not the whole repo (~1.1 GB)
                         allow_patterns=["muszka_int8_pc_rr.onnx", "muszka_guard.py", "config.json", "tokenizer.json"])
sys.path.insert(0, path)
from muszka_guard import MuszkaGuard

guard = MuszkaGuard(path, "muszka_int8_pc_rr.onnx", mode="sensitive")
print(guard.predict("ty kórwo")["is_blocked"])            # True
print(guard.predict("Jak upiec sernik?")["is_blocked"])   # False

• Set the mode with the mode parameter: sensitive (default), balanced or no_windows.
• Use muszka_int8_pc_rr.onnx rather than a plain per-channel INT8 file: without reduce_range, per-channel INT8 breaks on x86 CPUs without VNNI (tested on a small VPS). INT8 vs fp32 decision agreement: 99.0% on 300 test texts.

Credits

Muszka Guard would not exist without the work of many people who shared their models and data.

1. Jan Sowa

   Polish translations of WildGuardMix and Jigsaw Toxic Comments (Bielik 11B v3.0) and the lexicon of vulgar words and phrases - the core of v1.1's training and profanity tests. Thank you!

2. SpeakLeash - Bielik-Guard "Sójka"

   Bielik-Guard 0.1B / 0.5B - Muszka's base model and teacher (Apache 2.0). Authors: Jan Maria Kowalski, Krzysztof Wróbel, Jerzy Surma, Igor Ciuciura, Maciej Krystian Szymański (arXiv:2602.07954).

3. Sławomir Dadas

   Base encoder sdadas/mmlw-roberta-base.

4. The Bielik team (SpeakLeash)

   Krzysztof Ociepa, Remigiusz Kinas, Adrian Gwoździej, Krzysztof Wróbel, Łukasz Flis, Sebastian Kondracki. Bielik 11B v2.1 and v3.0 - translations and synthetic Polish data.

5. PolyGuardMix

   Kumar et al., 2025, CC BY 4.0 - model responses and categories. The part coming from LMSys-Chat-1M was excluded.

6. WildGuardMix

   AllenAI, ODC-BY - the source PolyGuardMix is derived from, and part of the training data in translation.

7. Jigsaw Toxic Comment Classification Challenge

   Jigsaw / Conversation AI; comments from Wikipedia.

Tests only (never trained on)

• PL-Guard

  NASK, CC BY-SA 4.0.

• GadziJęzyk

  Prof. Jerzy Surma, SGH, MIT.

Muszka Guard license: Apache 2.0.