For smaller extensions, you can start by looking at the manifest.json as this file is the entry point. You can see which part of the code needs access to ‘everything’ and go from there.
For example, with Instance Assistant for Lemmy, you need to give access to all sites because it needs to run on any Lemmy site and that’s not something you can hardcode in.