I use Starship, so this applies to both ZSH & Fish:

If you have the Python environment displayed in Starship, your largest gain in speed is to replace pyenv and pyenv-virtualenv by uv.