Can I parallelize just one large layer?

Can I parallelize just one large layer across gpus and keep all other layers the same and work with distributed data parallel?

For example, a regular resnet18 model, which is small enough but the number of classes is 1 million, how do I parallelize just the classification layer?

Hey,
Did you try this? did it work?