Welcome to the forum. What you are hearing is commonly referred to as "coil whine": https://en.wikipedia.org/wiki/Electromagnetically_induced_acoustic_noise. This typically occurs on any component that have power delivery/voltage regulation involved. For GPU's in particular, you have the GPU VRM (Voltage Regulation Module) that vibrates under higher power loads which produces that very distinct pitch you are hearing. This has become far more common now as hardware requires more robust and sophisticated power delivery systems and unfortunately there is very little one can do to mitigate the noise.
We overclockers used to insulate the base of the caps with clearcoat nail polish to help prevent the vibrations though I don't really recommend this modern parts get very hot and clearcoat nail polish can be considered an insulator. Others tend to have success with replacing thermal pads to ensure they are rigid between the VRM and heatsink/backplate of the card to mitigate vibration, though you have to be careful with this mod as well as it can have unintended consequences if thickness is wrong or you damage the card during the modification.
If the whine is too noticeable despite having your case side panels on, you can look into buying a noise dampening chassis. If the noise doesn't bother you, know that your card isn't defective and is operating normally. My motherboard has a coil whine that changes pitch every time I move my mouse, so I can definitely empathize with the concern.
Submit photos and a description of your PC to our build showcase
See other custom PC builds and get some ideas for what can be done
Services starting at $149.99