Strange, I enabled the setting, but I didn't see that there was a new comment on this article until I saw the comment number go up under this article on my profile page.
EDIT: tbh, I didn't notice there were links for "notifications" and "messages" until I moused over my username. Would be nice if there was a Reddit-style inbox icon right on the top bar.