https://chengaoshen.com/en/tags/math/ Recent content in Math on ChengAo Shen Hugo en Tue, 22 Jul 2025 00:00:00 +0000 https://chengaoshen.com/en/posts/gumbel_softmax/ Tue, 22 Jul 2025 00:00:00 +0000 https://chengaoshen.com/en/posts/gumbel_softmax/ <p><strong>Motivation.</strong> In many models we need to <em>select</em> a discrete option inside the computation graph (e.g., pick one branch of a network). A hard argmax is non-differentiable, so gradients can’t flow through it. Gumbel-Softmax provides a continuous, differentiable approximation to this discrete sampling step.</p> <h2 id="gumbel-max-trick"> <strong>Gumbel-Max Trick</strong> <a class="heading-link" href="#gumbel-max-trick"> <i class="fa-solid fa-link" aria-hidden="true" title="Link to heading"></i> <span class="sr-only">Link to heading</span> </a> </h2> <p>Assume we have discrete distribution</p> <table> <thead> <tr> <th>$X$</th> <th>1</th> <th>2</th> <th>3</th> </tr> </thead> <tbody> <tr> <td>$p$</td> <td>0.2</td> <td>0.3</td> <td>0.5</td> </tr> </tbody> </table> <p>And want to get $X$ follow this distribution. If we directly sample from distribution, the $X$ can’t calculate from $p$. Which means $X$ can’t be differentiated w.r.t. $p$. This means we can’t do back propagation.</p>