Getting a CD on the N64 would have allowed for some cooler videos in the middle of the game (and would probably have made things cheaper for developers), but, in my opinion, the thing that really killed the N64 is the tiny (4k) texture cache combined with high latency RAM on the system.
That’s why N64 games have a “look” that can be easily identified.