Golden examples are what the passport offers as evidence that the agent does what
it says. Your rebuild is checked against these. This site checked the file's format, not the truth
of its examples.
One example, the newest saved run, complete: nothing elided. These are templates of shape, never expected content: the installer runs this on different documents, for a different reader and a different decision, and different content is the correct result. A second run exists at the One-page brief depth, but it rendered the same eight sections with the same number of findings, so it adds no second shape; the oldest run predates the current instructions.
> Instantiated in this example, all of them values the person running it supplied for that run rather than fixed parts of the shape: the objective (whether their team should read one research paper closely before choosing a model architecture for a text-classification product), the reader (a product manager with no machine-learning background), the depth (Standard), one focus question (when self-attention is cheaper than recurrent layers, and what the paper says about training cost), and the single uploaded document, a public research paper, named in the inventory by its file name. No glossary, folders or house style were set. The owner reviewed the example and approved it as shown, including that run's objective, the paper's title and its file name.
Captured 2026-09-27:
## Bottom line
The paper "Attention Is All You Need" argues that a new architecture called the Transformer, which relies entirely on self-attention mechanisms, is superior to recurrent neural networks (RNNs) for sequence tasks because it allows for significantly more parallelization and faster training times [S1, S2]. For your team's text-classification product, the key takeaway is that self-attention is computationally cheaper than recurrent layers specifically when the sequence length ($n$) is smaller than the representation dimension ($d$), a condition the authors note is "most often the case" for sentence representations [S6, S7]. The paper demonstrates this with concrete training costs: a large Transformer model achieved state-of-the-art results in 3.5 days on 8 GPUs, whereas previous best models required significantly more floating-point operations (FLOPs) [S1, S8]. However, the paper focuses on machine translation and parsing; it does not provide specific training cost data for text classification, so while the architectural advantages are clear, the exact cost savings for your specific use case remain an inference based on the paper's complexity analysis.
## Key findings
1. **Self-attention is faster for typical sentence lengths:** The paper states that self-attention layers are computationally faster than recurrent layers when the sequence length ($n$) is smaller than the representation dimension ($d$), which is the common scenario for sentence representations [S6, S7].
2. **Training time is drastically reduced:** The Transformer model achieved a new state-of-the-art BLEU score of 28.4 on English-to-German translation after training for only 12 hours (base model) or 3.5 days (big model) on 8 NVIDIA P100 GPUs, a fraction of the time required by previous best models [S1, S7, S8].
3. **Parallelization is the primary driver:** Unlike recurrent models which process data sequentially (precluding parallelization within a training example), the Transformer allows for significantly more parallelization because it dispenses with recurrence entirely [S1, S2].
4. **Complexity comparison:** The computational complexity per layer for self-attention is $O(n^2 \cdot d)$, while for recurrent layers it is $O(n \cdot d^2)$; the paper notes that self-attention is faster when $n < d$ [S6, S7].
5. **Generalization to other tasks:** The authors successfully applied the Transformer to English constituency parsing, achieving results comparable to or better than state-of-the-art models even with limited training data, suggesting the architecture generalizes well beyond translation [S1, S9, S10].
6. **Interpretability:** Self-attention mechanisms in the model were observed to learn distinct tasks, such as syntactic and semantic relationships (e.g., anaphora resolution), offering more interpretability than recurrent models [S7, S13, S14].
## Implications
* **For your architecture decision:** If your text-classification task involves sentences where the number of tokens is fewer than the model's hidden dimension (a standard setup), the Transformer architecture is theoretically more efficient to train than an RNN-based approach [S6, S7].
* **Inference on cost:** Since the paper establishes that self-attention reduces sequential operations to a constant number ($O(1)$) compared to $O(n)$ for RNNs, you can infer that for long documents or large batches, the Transformer will likely scale better and reduce training time, provided the sequence length does not exceed the dimensionality significantly [S6].
* **Inference on product fit:** The paper's success in parsing (a structural task) and translation suggests the model captures long-range dependencies effectively, which could be beneficial for classification tasks where context from the beginning of a text influences the label at the end [S2, S6].
* **Missing context:** The paper does not explicitly mention "text classification" as a primary use case; it focuses on sequence transduction (translation) and parsing. Therefore, while the efficiency arguments hold, you must verify if the specific nuances of classification (e.g., fixed-length output vs. variable-length generation) align with the Transformer's strengths as described [S1, S9].
## Evidence table
| Claim | Figure (exact, with unit) | Date and scope | Source |
| :--- | :--- | :--- | :--- |
| Transformer achieves 28.4 BLEU on WMT 2014 English-to-German | 28.4 BLEU | WMT 2014 English-to-German task | [S1] |
| Transformer achieves 41.8 BLEU on WMT 2014 English-to-French | 41.8 BLEU | WMT 2014 English-to-French task | [S1] |
| Base model training time | 12 hours | 8 NVIDIA P100 GPUs | [S2, S7] |
| Big model training time | 3.5 days | 8 NVIDIA P100 GPUs | [S1, S7] |
| Base model training steps | 100,000 steps | Base model configuration | [S7] |
| Big model training steps | 300,000 steps | Big model configuration | [S7] |
| Self-attention complexity per layer | $O(n^2 \cdot d)$ | Theoretical complexity | [S6] |
| Recurrent layer complexity per layer | $O(n \cdot d^2)$ | Theoretical complexity | [S6] |
| Sequential operations for self-attention | $O(1)$ | Theoretical sequential operations | [S6] |
| Sequential operations for recurrent layers | $O(n)$ | Theoretical sequential operations | [S6] |
| Training cost (FLOPs) for Transformer (base) | $3.3 \cdot 10^{18}$ | English-to-German task | [S8] |
| Training cost (FLOPs) for Transformer (big) | $2.3 \cdot 10^{19}$ | English-to-German task | [S8] |
| Number of attention heads used | $h = 8$ | Base model configuration | [S5] |
| Model dimension ($d_{model}$) | 512 | Base model configuration | [S3] |
| Feed-forward inner dimension ($d_{ff}$) | 2048 | Base model configuration | [S3] |
| Dropout rate for base model | $P_{drop} = 0.1$ | Base model configuration | [S8] |
| Label smoothing value | $\epsilon_{ls} = 0.1$ | Base model configuration | [S8] |
| Warmup steps | 4000 | Base model configuration | [S7] |
## Risks and disagreements
* **Applicability to text classification:** The paper exclusively presents results for machine translation and constituency parsing. There is no data in the source documents regarding performance or training costs specifically for text classification. The authors do not claim the model is optimal for classification, only that it generalizes well to other tasks [S1, S9].
* **Sequence length constraint:** The paper notes that self-attention is faster than recurrent layers only when the sequence length ($n$) is smaller than the representation dimension ($d$). If your text-classification product involves very long documents where $n > d$, the computational advantage may diminish or reverse, as the complexity of self-attention scales quadratically with sequence length ($O(n^2)$) [S6, S7].
* **Hardware dependency:** The reported training times (12 hours, 3.5 days) are specific to "8 NVIDIA P100 GPUs." Training costs on different hardware or with different batch sizes may vary significantly, as the paper estimates FLOPs based on specific GPU capacities [S7, S8].
* **Interpretability vs. Performance:** While the paper claims self-attention yields more interpretable models, this is based on visualizations of attention heads in translation and parsing tasks. It is not guaranteed that these interpretability benefits will translate directly to text classification without further investigation [S7, S13].
## Questions requiring follow-up
* Does the text-classification product typically process sequences where the token count is less than the model's hidden dimension? (This determines if the $O(n^2 \cdot d)$ vs $O(n \cdot d^2)$ advantage applies).
* What is the expected sequence length distribution for the classification task? (To assess if the quadratic complexity of self-attention becomes a bottleneck).
* Are there specific constraints on inference latency or memory usage for the product that might favor a different architecture? (The paper focuses on training efficiency, not inference).
* Has the team evaluated the Transformer on a similar text-classification dataset to confirm the generalization claims?
## Source map
The summary is derived entirely from the single document "attention.pdf" (dated 2026-09-27), which was read in full (pages 1-15). The document provides the theoretical arguments for self-attention efficiency (pages 6-7), the specific training costs and results (pages 1, 7-8), and the architectural details (pages 3-5). No other documents were available or read.
**Document inventory** (generated from the files, not by the model)
| Document | Date | Extent | Source ids | Status |
|---|---|---|---|---|
| attention.pdf | 2026-09-27 (file date) | 15 page(s) with text, 39,496 characters | S1 to S15 | read in full |
## What the user should do next
1. **Verify sequence length vs. dimension:** Check your product's typical input length ($n$) and planned model dimension ($d$) to confirm if $n < d$ holds, which is the condition for self-attention being cheaper than recurrent layers according to the paper [S6, S7].
2. **Run a small-scale benchmark:** Since the paper does not cover text classification, run a quick pilot comparing a small Transformer model against a standard RNN on a subset of your classification data to measure actual training time and accuracy.
3. **Review hardware requirements:** Confirm if your infrastructure can support the parallelization benefits (e.g., access to multiple GPUs) mentioned in the paper, as the 12-hour/3.5-day training times are dependent on 8 NVIDIA P100 GPUs [S7].
4. **Investigate inference costs:** The paper emphasizes training speed; research or test the inference latency of the Transformer for your specific classification use case, as the paper does not provide data on this aspect.