The grad of training loss with respect to z_{sem} is zero #63

First, thanks for your excellent work!

I've reproduced the model training process outlined in your project. However, during my inspection of the GaussianDiffusionBeatGans.training_losses method (defined in diffusion/base.py), I noticed that the gradient of the loss function with respect to the z_{sem} variable (passed via model_kwargs['cond']) is consistently zero. This raised concern because a zero gradient implies that z_{sem} is not being utilized in the training loss computation, which would essentially prevent the semantic encoder from training.

Could this possibly be a mistake or a misunderstanding on my part? Is this the same in your training process? If not, do you have any valuable ideas of the possible mistake in my reproduction? I would greatly appreciate any insights or clarifications you might be able to provide.