SA-21/06, C-11, Asapur, Varanasi, UP support@kamaldnp.com +91 8318013442
Technology

ParGLU: A Partially Gated Feed-Forward Layer That Beats SwiGLU at Small Scale (arXiv, November 2026)

KAMALDNP Raj Kamal
November 21, 2026
4 Min Read
3 Views
ParGLU: A Partially Gated Feed-Forward Layer That Beats SwiGLU at Small Scale (arXiv, November 2026)

Inside every transformer model sits a feed-forward network (FFN) — the component that processes each token's representation after attention has done its work. The exact shape of that FFN matters: small design changes there can shift a model's quality across the board. SwiGLU has been the standard choice for years. A new single-author paper proposes ParGLU, a partially gated variant, and reports that it beats SwiGLU — at the scales tested.

The paper was submitted to arXiv on 16 November 2026 by Yuting Ge. ParGLU runs SwiGLU and ReLU² branches in parallel, gating only part of the layer — hence "partially gated". The idea is to combine the strengths of two activation styles rather than committing to one.

According to the paper's own reported results, ParGLU outperforms SwiGLU at 75M and 152M parameters across all seeds, with mean validation negative log-likelihood improving by −0.0200 at 75M and −0.0249 at 152M. But the picture is not uniform: at 30M parameters, the results are mixed, and the paper reports them as such.

Key facts

ItemDetails
PaperParGLU
arXiv IDarXiv:2611.20663 [cs.LG]
Submitted16 November 2026
AuthorYuting Ge (sole author)
DesignPartially gated FFN — SwiGLU + ReLU² parallel branches
75M resultOutperforms SwiGLU across all seeds; mean validation NLL −0.0200 (paper's own reported result)
152M resultOutperforms SwiGLU across all seeds; mean validation NLL −0.0249 (paper's own reported result)
30M resultMIXED — must be carried, not dropped
Scale of studySmall-scale only — largest model 152M; NOT proven at frontier scale
Independent reproductionNone yet — the paper is 5 days old

What this means

FFN design is one of those quiet corners of model architecture where a genuine improvement can propagate everywhere — every transformer has these layers, so a better FFN is a better model, in principle, at any size. That is why a paper reporting consistent wins at two scales is worth attention, even from a single author.

But two honest limits frame the result. First, the mixed 30M results show the advantage is not automatic at every scale — something about the design interacts with model size, and the paper does not claim otherwise. Second, the largest model tested is 152M parameters. That is a research scale, not a frontier scale: the models in production use are hundreds or thousands of times larger, and small-scale wins do not always survive the jump. ParGLU is a promising candidate for further testing, not a proven replacement for SwiGLU.

The "across all seeds" detail is worth noting: the wins at 75M and 152M held up across repeated runs with different random seeds, which makes them harder to dismiss as a lucky run. That is a point in the paper's favour — but it is still the author's own measurement, awaiting independent confirmation.

A note on verification

The paper exists on arXiv with the ID, date and author given above. Every figure in this post is the paper's own reported result. The mixed results at 30M and the small scale of the study (maximum 152M parameters) are part of the paper's own reporting and are carried here deliberately — a summary that omitted them would misrepresent the work. We will update this post if independent reproduction or coverage appears.

Source: arXiv:2611.20663 [cs.LG], ParGLU, Yuting Ge, submitted 16 November 2026. Last checked: 21 November 2026.

FAQs

What is ParGLU?
ParGLU is a partially gated feed-forward network design proposed in a paper submitted to arXiv on 16 November 2026 (arXiv:2611.20663). It runs SwiGLU and ReLU² branches in parallel, gating only part of the layer.

How well did it do against SwiGLU?
According to the paper, ParGLU outperforms SwiGLU at 75M and 152M parameters across all seeds, with mean validation NLL improving by −0.0200 and −0.0249 respectively. These are the paper's own reported results.

Did it win at every size?
No. At 30M parameters the results are mixed — the paper reports this itself, and it is an important part of the picture. The advantage is not automatic at every scale.

Is ParGLU proven for large frontier models?
No. The study is small-scale — the largest model tested is 152M parameters. Small-scale wins do not always carry over to models hundreds of times larger, and no independent reproduction exists yet.

Share this post

Comments (0)

Share your question or experience about this syllabus. Your comment will appear after review.

No comments yet — be the first to share your thoughts!

🤖 KAMALDNP AI Assistant

Ask about courses, website & more

Hi! I'm KAMALDNP AI Assistant. Ask me about our courses, website features, or anything else! 😊
Please wait 5s...
🤖 KAMALDNP Assistant ✕

Request human callback:

🔐 Login Required
✕

Please log in to download. You will stay on this page after logging in!

New here? Register now →