Are you initiating SSH connections from all these hosts?
If you just need to SSH to these hosts, use a single key and copy the public key only to the hosts you need to connect to. If you don’t want to copy the pubkeys to target hosts, use LDAP + SSSD or certificates.
Then, if you do need to initiate connections from these hosts and use an SSH agent you can forward your agent and SSH to another host