I only get those minor delays if I’m literally sprinting through the castle on my PC, and never at night in-game.
The game is absolutely gorgeous and my one main disappointment for it is there’s no photo mode.
Honestly, the vistas and scenery in the game are beautiful. There’s nothing quite like flying over a hilltop at night on your broom and seeing the castle looking over everything, lit up like Christmas lights.