Totally agree. Even when two commenters are replying to one another, there is always another layer where they are also addressing everyone in the thread/community/instance/fediverse, which obviously includes lurkers.
The votes shape everything about the platform, so ignoring the lurkers in the stats feels like it’s missing an important data point.