SwiGLU

SwiGLU is an activation function which is a variant of GLU. The definition is as follows:

$$SwiGLU(x,W,V,b,c,\beta )=Swis{h}_{\beta }(xW+b)\otimes (xV+c)$$

References

• Papers With Code