I think the rule is worded really badly in a way that makes it more confusing that it needs to be.
As for the rule itself, my table usually hand waves it and lets you cast whatever you want with your bonus action and action, provided you have the slots to do it. We haven’t had any issues with that feeling game breaking for us yet.