But surely they won’t just let in guests for their own sake? They’re still doing it as a service for the guests, even if there are terms.
And I’d say that people from federated instances aren’t guests, they are more like people that can talk to your guests. Defederation is more like closing the window between the different parties so that they can’t talk to you because they disturb the peace and quiet. Then it seems entirely reasonable that your guests can still listen to them in a way that doesn’t affect the rest of your party.