The way I think about Majora’s Mask as a Zelda game is that in addition to exploring the physical world, you’re also exploring time. That does necessitate “backtracking” by forcing time resets and a lot of waiting around if you don’t immediately know what you can be doing in parallel (though the two time control songs make that part easier).
With the exception of the dungeons themselves, the game typically fast tracks getting you back to where you were when you just reset. Some mechanics like that the game forces on you pretty quickly (song of soaring fast travel), others it lets you figure out on your own (dungeon boss instant warp after beating them the first time).
Side quests can be a bit more troublesome to deal with if you have to reset part way through, but each interaction point that you have to go through offers you another way to handle things (or to not and let another sequence of events happen).
To your last point, the game really throws refillable items at you in the overworld, so a lot of times you can skip that (I’m not saying stocking up doesn’t take forever on reset, it does. You just don’t always have to)
All in all I really love the time mechanic of the game and that let’s me forgive some of the other flaws of the game. If it fell flat, then yeah I can see how the game quickly becomes a chore. But I adore the game, hence the username