I recently learned that voting on lemmy is not anonymous. Anyone can get information about who has upvoted and downvoted a post or comment.
In combination with your IP, this is a massive privacy (maybe even physical security) risk. Also, people can target you for your votes.
Sadly, this is something where I would prefer Reddit over Lemmy. Big tech scrapes data from both places anyways, at least Reddit is safe.
Why are you worried about admin abuse? If you are worried that your admin will abuse you, you should switch to an instance you trust more.