I agree with your point of view, it is a question of context and a general rule cannot be established. It depends, among other factors, on the provenance and function of the human remains in the museum: whether they are only in the museum as a result of colonial brutality, whether they can provide significant knowledge, or whether they are only on display to sell tickets.
In Spain we have a very interesting case: that of the Guanche mummies.