I'd like to talk about something about EM algorithm in my understanding.
This post is mainly based on Richard Xu's machine learning course.
Gaussian Mixture Model
Gaussian Mixture Model (GMM) (k-mixture) is defined as:
and
For data
Then we can use MLE to estimate
This formula is difficult to solve because it is in 'log-of-sum' form. So, we solve this problem in an iterative way, called Expectation Maximization.
Expectation Maximization
Instead of performing
we assume some latent variable
For each iteration of the E-M algorithm, we perform:
We must ensure convergence:
denote
Hence ,
Using EM algorithm to solve GMM
Put GMM into this frame work.
Define
M-Step:
Maximizing
Solving this problem via Lagrangian Multiplier, we have