That's like saying that only using high security locks with various security pins in them to protect your house is a bad idea, and you should throw in some secured with padlocks too just to change things up.
And if some of them are shitty masterlocks, well, you're changing things up.
That's really not how security works.
Yes, pass phrases can have large amounts of entropy attached. But unless you are picking your pass phrases truly randomly, with a large dictionary, and using unique pass phrases per site, and the sites are not silently truncating the password input (such as bcrypt which truncates to 72 bytes), you are not actually getting that large amount of entropy.
Where as a 16 character password that randomly uses the ASCII printable range, excluding spaces, gives you 93^16 possible combinations. That's 31313180170800116587336013460801 passwords.
Or, very roughly, 104.6 bits of entropy. (104.6265409777285022441578006899739 bits of entropy if you want to be downright absurd about it.)
Knowing that you're doing that simply doesn't help the attacker in any meaningful way.
Bumping that to 20 characters gives you over 130 bits of entropy, or 2342388736625917052139104541473924426001 possible combinations.
This is quite simply not a viable attack surface.
Where as saying 'use pass phrases for some things' means that it is quite likely that some of your pass phrases are going to be much less secure than this.
But let's give the same numbers for properly generated random passphrases.
The xkcdpass utility can help us here.
Even picking entirely randomly, out of a large word list of 7227 words, a 6 word pass phrase only gives roughly 76 bits of entropy.
Going up to 8 words gives us roughly 102 bits of entropy, that helps a ton... Except that some of those passphrases are going to be longer than 72 bytes. So you're almost certainly losing bits of entropy.
That best case still gives you fewer bits of entropy than a 20 character randomly generated password. Unless you're trying to memorize your password, there are no benefits to alternating between randomly generated passwords with good generation settings and passphrases.
And if you're trying to memorize your passwords, you are definitely doing it wrong.