I'd like to know if my pytorch code is fully utilizing the GPU SMs. According to this question gpu-util in nvidia-smi only shows how time at least one SM was used.

I also saw that typing nvidia-smi dmon gives the following table:

# gpu   pwr gtemp mtemp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     C     %     %     %     %   MHz   MHz
    0   132    71     -    58    18     0     0  6800  1830

Where one would think that sm% would be SM utilization, but I couldn't find any documentation on what sm% means. The number given is exactly the same as gpu-util in nvidia-smi.

Is there any way to check the SM utilization? On a side note, is there any way to check memory bandwidth utilization?