The concern is about impeding progression when there is no traffic and no good reason, ie an empty road through town in quiet times with few cars or people about.
The thing is, nobody plans to crash their car into a pedestrian. So how does it keep happening? One reason is from people driving faster than they should because they wrongly believe a road is empty. Slower limits help with this, because when drivers are going around with incorrect beliefs about whether roads are empty or not, there is more time to react and less energy to cause injury