tinydistill-massive-da-multilingual
A small text classifier distilled from google/gemma-4-12B-it into sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 with tinydistill, exported to ONNX for fast CPU inference.
Labels (60): alarm_query, alarm_remove, alarm_set, audio_volume_down, audio_volume_mute, audio_volume_other, audio_volume_up, calendar_query, calendar_remove, calendar_set, cooking_query, cooking_recipe, datetime_convert, datetime_query, email_addcontact, email_query, email_querycontact, email_sendemail, general_greet, general_joke, general_quirky, iot_cleaning, iot_coffee, iot_hue_lightchange, iot_hue_lightdim, iot_hue_lightoff, iot_hue_lighton, iot_hue_lightup, iot_wemo_off, iot_wemo_on, lists_createoradd, lists_query, lists_remove, music_dislikeness, music_likeness, music_query, music_settings, news_query, play_audiobook, play_game, play_music, play_podcasts, play_radio, qa_currency, qa_definition, qa_factoid, qa_maths, qa_stock, recommendation_events, recommendation_locations, recommendation_movies, social_post, social_query, takeaway_order, takeaway_query, transport_query, transport_taxi, transport_ticket, transport_traffic, weather_query
Usage
from tinydistill import Predictor # pip install tinydistill
clf = Predictor("hf:Shroomy/tinydistill-massive-da-multilingual")
clf.predict(["some text"])
Or from the command line:
tinydistill predict hf:Shroomy/tinydistill-massive-da-multilingual "some text"
tinydistill serve hf:Shroomy/tinydistill-massive-da-multilingual
Results
| Model | Device | Accuracy | Macro-F1 | Top-3 accuracy | Agreement with teacher | Latency, batch 1 (ms) | Latency, batch 32 (ms) | Throughput (texts/s) | Size (MB) | Speed-up, batch 1 |
|---|---|---|---|---|---|---|---|---|---|---|
| teacher, 16 similar examples, 4-bit, probe | cuda:0 | 0.808 | 0.780 | 0.908 | n/a | n/a | n/a | n/a | 23920 | n/a |
| student, PyTorch | cpu | 0.815 ± 0.002 | 0.782 ± 0.005 | 0.922 ± 0.001 | 0.849 ± 0.002 | 7.3 | 99.2 | 322.7 | 95 | n/a |
| student, ONNX fp32 | cpu | 0.818 | 0.788 | 0.923 | 0.849 | 4.5 | 92.3 | 346.7 | 96 | n/a |
| student, ONNX int8 | cpu | 0.815 | 0.789 | 0.922 | 0.846 | 2.0 | 41.9 | 763.4 | 24 | n/a |
Task massive-da, 2974 gold-labelled test texts. Teacher: google/gemma-4-12B-it. Student: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. The teacher's probabilities are averaged with those of a linear probe on its hidden states, fit on 467 gold training texts; fit on 374 of them, teacher accuracy on the other 93 is 81.7% in context, 76.3% for the probe and 82.8% combined. The teacher's prompt shows each text the 16 most similar of 476 gold training examples. The student learns from the teacher's soft labels on all 11514 training texts, plus cross-entropy on 476 gold-labelled texts (8 per class; weight 1.0). Values with ± are the mean and standard deviation over 3 training seeds; the ONNX rows are the exported first-seed model.
CPU rows use 2 threads. Latency is the median over repeated runs. Size is the teacher's weights as loaded and the students' model files. The teacher packs several texts into one forward pass; it is timed at batch 32 only when that takes under three minutes, otherwise its throughput is 1 / batch-1 latency.
Cascade: student first, unsure texts to the teacher
| Share sent to the teacher | Student confidence threshold | Accuracy | Macro-F1 | Mean latency per text (ms) |
|---|---|---|---|---|
| 0% | 0.131 | 0.815 | 0.789 | n/a |
| 10% | 0.564 | 0.819 | 0.792 | n/a |
| 20% | 0.754 | 0.820 | 0.793 | n/a |
| 30% | 0.866 | 0.815 | 0.787 | n/a |
| 50% | 0.946 | 0.812 | 0.784 | n/a |
| 100% | 1.000 | 0.808 | 0.780 | n/a |
The student answers every text and passes its least confident ones to the teacher. Thresholds are set on the test texts to hit each share; in deployment, set them on a validation split. Latency is the student's batch-1 latency plus the share times the batch-1 latency of the teacher.
License
Check the licences of the student model, the teacher and the training data before using this model. The uploader chose cc-by-4.0.
- Downloads last month
- 126