* The student network: This is the main network being trained. It learns to classify data.
* The teacher network: This is a slightly delayed version of the student network. It acts as a "mean teacher" because it provides a more stable and consistent target for the student to learn from.
The teacher network's weights are an exponential moving average (EMA) of the student network's weights. This means the teacher's weights are a smoothed version of the student's, making it less susceptible to noisy updates during training.
How it works:
1. The student network processes labeled and unlabeled data.
2. The teacher network provides a prediction for the same data, using its smoothed weights.
3. The student network's loss is calculated based on the difference between its predictions and the teacher's predictions (for both labeled and unlabeled data). The labeled data loss is the standard supervised loss, and unlabeled data loss encourages the student to agree with the teacher.
4. The student network updates its weights to minimize this loss.
5. The teacher network updates its weights using the EMA of the student's weights.
Benefits of using a mean teacher:
* Improved generalization: The smoothed target provided by the teacher network helps the student network to learn more robust and generalizable features.
* Better performance on unlabeled data: This method effectively leverages unlabeled data, which is common in semi-supervised learning scenarios.
* Reduced overfitting: The stability provided by the teacher network can help prevent the student network from overfitting to the training data.
In essence, the "mean teacher" acts as a regularizing force, guiding the student network towards a more stable and consistent solution. It's a clever technique for improving the performance of semi-supervised learning models.