One way I’ve done swarms in the past is treating the swarm as an amorphous blob of things. At the end of the round, anyone in a space covered by the swarm takes a certain amount of damage.
Alternatively, if they’re more annoying than dangerous, anyone that takes an action in a swarm space rolls with disadvantage.