The total point count on your profile may be meaningless, but votes on a comment/post absolutely have an effect within context. If you want to post to a community about something, the post may never get seen by many if it amasses a bunch of down votes. If you want to weigh in on a subject in a particular thread, a down voted opinion will be drown out by all the other comments.