Inside every transformer model sits a feed-forward network (FFN) — the component that processes each token's representation after attention has done its work. The exact shape of that FFN matters: small design changes there can shift a model's quality across the board. SwiGLU has been the standard choice for years. A new single-author paper proposes ParGLU, a partially gated variant, and reports that it beats SwiGLU — at the scales tested.
The paper was submitted to arXiv on 16 November 2026 by Yuting Ge. ParGLU runs SwiGLU and ReLU² branches in parallel, gating only part of the layer — hence "partially gated". The idea is to combine the strengths of two activation styles rather than committing to one.
According to the paper's own reported results, ParGLU outperforms SwiGLU at 75M and 152M parameters across all seeds, with mean validation negative log-likelihood improving by −0.0200 at 75M and −0.0249 at 152M. But the picture is not uniform: at 30M parameters, the results are mixed, and the paper reports them as such.
Key facts
| Item | Details |
|---|---|
| Paper | ParGLU |
| arXiv ID | arXiv:2611.20663 [cs.LG] |
| Submitted | 16 November 2026 |
| Author | Yuting Ge (sole author) |
| Design | Partially gated FFN — SwiGLU + ReLU² parallel branches |
| 75M result | Outperforms SwiGLU across all seeds; mean validation NLL −0.0200 (paper's own reported result) |
| 152M result | Outperforms SwiGLU across all seeds; mean validation NLL −0.0249 (paper's own reported result) |
| 30M result | MIXED — must be carried, not dropped |
| Scale of study | Small-scale only — largest model 152M; NOT proven at frontier scale |
| Independent reproduction | None yet — the paper is 5 days old |
What this means
FFN design is one of those quiet corners of model architecture where a genuine improvement can propagate everywhere — every transformer has these layers, so a better FFN is a better model, in principle, at any size. That is why a paper reporting consistent wins at two scales is worth attention, even from a single author.
But two honest limits frame the result. First, the mixed 30M results show the advantage is not automatic at every scale — something about the design interacts with model size, and the paper does not claim otherwise. Second, the largest model tested is 152M parameters. That is a research scale, not a frontier scale: the models in production use are hundreds or thousands of times larger, and small-scale wins do not always survive the jump. ParGLU is a promising candidate for further testing, not a proven replacement for SwiGLU.
The "across all seeds" detail is worth noting: the wins at 75M and 152M held up across repeated runs with different random seeds, which makes them harder to dismiss as a lucky run. That is a point in the paper's favour — but it is still the author's own measurement, awaiting independent confirmation.
A note on verification
The paper exists on arXiv with the ID, date and author given above. Every figure in this post is the paper's own reported result. The mixed results at 30M and the small scale of the study (maximum 152M parameters) are part of the paper's own reporting and are carried here deliberately — a summary that omitted them would misrepresent the work. We will update this post if independent reproduction or coverage appears.
Source: arXiv:2611.20663 [cs.LG], ParGLU, Yuting Ge, submitted 16 November 2026. Last checked: 21 November 2026.
FAQs
What is ParGLU?
ParGLU is a partially gated feed-forward network design proposed in a paper submitted to arXiv on 16 November 2026 (arXiv:2611.20663). It runs SwiGLU and ReLU² branches in parallel, gating only part of the layer.
How well did it do against SwiGLU?
According to the paper, ParGLU outperforms SwiGLU at 75M and 152M parameters across all seeds, with mean validation NLL improving by −0.0200 and −0.0249 respectively. These are the paper's own reported results.
Did it win at every size?
No. At 30M parameters the results are mixed — the paper reports this itself, and it is an important part of the picture. The advantage is not automatic at every scale.
Is ParGLU proven for large frontier models?
No. The study is small-scale — the largest model tested is 152M parameters. Small-scale wins do not always carry over to models hundreds of times larger, and no independent reproduction exists yet.
Comments (0)
Share your question or experience about this syllabus. Your comment will appear after review.
No comments yet — be the first to share your thoughts!