deepopen
A Laya-based Intent Classification Framework
Benchmark results, dataset-specific configurations, and reproducible prediction evidence for the deepopen framework.
Reference snapshot: 7 October 2026
Maintained by the method authors
Single classifier / SAM seed 13
2,899 / 3,080 correct · Best observed of 3 SAM seeds
Four-encoder ensemble
4,411 / 4,500 correct · 150 classes
Three-seed mean
Consistency variant · 60 intents
Metric: test accuracy. CLINC150 covers in-scope intents only. MASSIVE is a separate multilingual protocol; its mean is not a single-checkpoint score.
Benchmark comparison
| Method | Test accuracy | Result source |
|---|
Evaluation protocol
| Dataset | Evaluation scope | Test examples | Aggregation |
|---|---|---|---|
| BANKING77 | 77-way intent classification | 3,080 | SAM seed 13; best observed of three |
| CLINC150 | 150 in-scope intents; no OOS rejection | 4,500 | Four-encoder ensemble |
| MASSIVE | 51 languages, 60 intents; intent accuracy only | 151,674 per seed | Three-seed mean; equal-size language splits |
Historical BANKING77 training used the full training set, including a previously used development subset. That subset is not an independent validation set for that model. The existing test results have been observed; the new continuation results are retrospective test comparisons, not fresh independent confirmation. No further tuning follows these test results.
BANKING77 continuation results
Full evaluation summaryThe headline is the best observed single-model result across three fixed SAM seeds, selected for display after test evaluation. All candidates started from the same historical full-train checkpoint.
| Candidate | Correct / 3,080 | Test accuracy |
|---|---|---|
| SAM seed 13 | 2,899 | 94.1234% |
| SAM seed 42 | 2,896 | 94.0260% |
| SAM seed 87 | 2,894 | 93.9610% |
| Three-seed mean | -- | 94.0368% |
Uniform evaluation: eager FP32, SDPA, batch 32, maximum 128 tokens. The historical checkpoint scores 93.9935% under this path. SAM mean gain: +0.0433 percentage points (shared-row 95% CI: -0.2273 to +0.3247); no significant improvement is established. The best-seed score is subject to test-selection bias.
Archived historical accuracy: 94.0909%. Current replay of the original BF16 script: 94.0584%. Exact historical prediction reproduction remains unresolved despite matching data and weight hashes. Robust improvement across execution settings is not established.
deepopen framework
One framework · dataset-specific configurationsA Laya-based Framework for Intent Classification
deepopen organizes intent classification around a common processing pipeline. Input construction, prediction heads, training objectives, and optional fusion are configured for each dataset.
input construction
& prediction
optional
| Dataset | Input & prediction configuration | Training / fusion configuration |
|---|---|---|
| BANKING77 | Text tokenization; one Laya-based English encoder and a 77-class classification head. | Full-train historical classifier + two fixed epochs of SAM (rho 0.05). Seed 13 selected for display after evaluating three continuation seeds; no ensemble. |
| CLINC150 | Member-specific tokenization; one Laya-based specialist and three DeBERTa-v3-large classifiers; 150 intents. | R-Drop-trained specialist (seed 42); probability fusion: 1/2 specialist + 1/6 each DeBERTa member (seeds 13/42/87). Four encoders at inference. |
| MASSIVE | Text plus all 60 candidate intents; Laya multilingual scoring; outputs mapped to fixed class IDs. | Canonical/shuffled candidate orders; mean cross-entropy + 0.5 symmetric KL. Three-seed mean, not a seed ensemble. |
The shared diagram describes the framework workflow. Dataset configurations differ in architecture and training strategy, not only numerical hyperparameters. Reported scores are tied to the configurations above, rather than one universal checkpoint.
Training & execution environment
Download verified configuration (JSON)Recorded MASSIVE training configuration
| Setting | Recorded value |
|---|---|
| Hardware | NVIDIA L20 for seeds 13/42; NVIDIA A10 for seed 87. Paired methods use the same hardware within each seed. |
| Training budget | 3 epochs; seeds 13, 42, 87; BF16 |
| Optimizer | AdamW; learning rate 1e-5; weight decay 0.01; linear decay; 6% warmup |
| Batch / input | Batch 8 × accumulation 4 = effective batch 32; maximum 1,024 tokens; 128 text tokens reserved |
| Objective | Mean cross-entropy across two candidate orders + 0.5 symmetric KL |
Historical BANKING77 and CLINC150 GPU allocation, CUDA versions, and full training hyperparameters are not established by the release-loading record. They are marked unverified here rather than replaced with current-machine values. No throughput or latency claim is made.
Verified historical-package loading environment
| Component | Version / setting |
|---|---|
| Python | 3.12.14; isolated environment; no inherited system packages |
| PyTorch / Transformers | 2.9.1 / 4.57.6 |
| NumPy / tokenizers / safetensors | 2.5.3 / 0.23.0rc0 / 0.9.0rc0 |
| Loading validation | Dependency check passed; manifest hashes verified; two synthetic inputs matched prior outputs. |
This is the clean release-loading environment, not a claim about the original training environment or a reproduction of test accuracy. Additional recorded dependencies are in the downloadable JSON.
Per-class results
| Gold class | Correct / support | Recall | Precision | F1 |
|---|
Historical prediction rows preserve artifact order, not official dataset IDs. MASSIVE records include seed, locale, and sample ID. Test utterances are not redistributed. Metrics are recounted from saved predictions; BANKING77 now uses the fixed-checkpoint retrospective test predictions of SAM seed 13.
Most frequent classification errors
Reproducibility & limitations
Artifacts
Prediction files and class metrics are available above. Model uploads and commit-pinned download verification are pending; this page does not claim that model delivery is complete. Offline checks on synthetic examples are not full test reproductions.
MASSIVE pools 455,022 prediction records over the same 151,674 examples evaluated across three seeds. These are not 455,022 independent examples. Download per-language results.
External comparison
Reference results are attributed to SOTA2 and have not been rerun by us. Training resources, model sizes, and protocols may differ. Selected references are not an exhaustive leaderboard.
This is an author-maintained submission resource, not an official ranking or a claim of SOTA2 acceptance. MASSIVE's paired accuracy-gain confidence interval includes zero.