I lived in three apartment buildings before buying my house, and each had very different noise characteristics. The first (old but remodeled) was very quiet, the second terrible (old and poorly maintained), and the third was acceptable (new-ish). It all has to do with how it’s built, and that’s something that can usually be figured out before buying if you talk to the neighbors.
So if you can find a good townhouse or condo where noise isn’t really an issue, then you get the benefits of a more compact living area, like being closer to downtown. Getting downtown from my current house takes 20-30 minutes, but for my coworkers who live in townhouses in the city, it’s like a 5 min train ride.
So when the kids leave the house, I’ll be more interested in being close to things than the benefits of a house (space, quiet, etc).