Magic can McGuffin any answer you need but it sounds like you want a non magic means to keep people away.
Have a guarded door, be it creature or armed guards. The room that is guarded is not the real room, just a place to hold private meetings. It's used on the regular to keep the focus on the guarded room.
The real room is in the town's bizarre. It's passed by all of the time and is curated by the most hated man. No one likes going to his shop. He's rude and his wares are over priced. Within his shop is the real room that is visited by the few who know.