To install the prerequisites for GPU support in TensorFlow 2.1:
Install your latest GPU drivers.
Install CUDA 10.1.
If the CUDA installer reports “you are installing an older driver version”, you may wish to choose a custom installation and deselect some components. Indeed, note that software bundled with CUDA including GeForce Experience, PhysX, a Display Driver, and Visual Studio integration are not required by TensorFlow.
Also note that TensorFlow requires a specific version of the CUDA Toolkit unless you build from source; for TensorFlow 2.1 and 2.2, this is currently version 10.1.
Install cuDNN.
Download cuDNN v7.6.4 for CUDA 10.1. This will require you to sign up to the NVIDIA Developer Program.
Unzip to a suitable location and add the bin directory to your PATH.