They would have had a bigger impact if the previous season hadn’t turned them into friends who were then completely forgotten about.
One scene saying something like ‘Oh, the Borg are here. I wonder why Agnes didn’t contact us first’ would have made it a bit more believable, rather than ‘We made friends with the Borg but now we’re scared when they’re mentioned’.