TransWikia.com

Latent Dirichlet allocation and properties of digamma function

MathOverflow Asked by sunxd on November 7, 2021

In the paper Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3(4–5), 993–1022. http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf

let $frac{partial L}{partial gamma_i} = psi^{‘}(gamma_i)(alpha_i + sum_{n=1}^N phi_{ni} – gamma_i) – psi^{‘}(Sigma_{j=1}^k gamma_j)sum_{j=1}^k(alpha_j + sum_{n=1}^N phi_{nj} – gamma_j) = 0$
yields $gamma_i = alpha_i + Sigma_{n =1}^{N}phi_{ni}$ (eqn. (17) in the paper’s appendix or eqn.(7) in the paper’s major part)

where, $gamma_i$ is the variational Dirichlet parameter that governs the multinomial probability $sum_{n=1}^Nphi_{ni} = 1$ ($N$ is the number of words in a document). see eqn(4) in the paper. $alpha$ is the model Dirichlet parameter. $psi$ is the digamma function. $L$ is the lower bound of the likelihood function.

This is equivalent to say that $psi^{‘}(Sigma_{j=1}^k gamma_j)sum_{j=1}^k(alpha_j + sum_{n=1}^N phi_{nj} – gamma_j)$ is zero.

so either $psi^{‘}(Sigma_{j=1}^k gamma_j)$ is zero or $sum_{j=1}^k(alpha_j + sum_{n=1}^N phi_{nj} – gamma_j)$ is zero

but which one is zero? And why?

clues: in Figure 6, which is the description of the algorithm,

$gamma$ initialized to be for document $i$, $gamma_i = alpha_i + N/k$? where $k$ is the number of topics.

and $phi_{ni}$ for the $i$th document is initialized to be $phi_{ni} = 1/k$ , in this case $sum_{j=1}^k(alpha_j + sum_{n=1}^N phi_{nj} – gamma_j) = 0$ holds since $(alpha_j + sum_{n=1}^N phi_{nj} – gamma_j) = 0$ holds for each document $j$, but this is just initialization.

in the iteration of the algorithm in Figure 6, $phi_{ni}$ is normalized to be sum up to 1 which means $(alpha_j + sum_{n=1}^N phi_{nj} – gamma_j) = (alpha_j + 1 – gamma_j)$, this seems not to be zero?

One Answer

well, it turns out that we do not need to seek for other solutions, since letting $alpha_j- gamma_j = 1$ is a solution?

Answered by sunxd on November 7, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP