Might be a bug, but when you downvote it appears on your profile in the boost tab. Sadly removing the downbutton does not remove it from the boost section.
You can see that on my profile when I downvoted a bot. Now I have forever a sexting bot on my profile page...