In the same boat as you. When I did use a Magic Mouse I’d just plug it in when I got up to go to the bathroom or get a drink, and not every time either. Never had the battery run out.
I stopped using it because the ergonomics don’t fit my hand very well.