I think it is more nuanced, because the majority of HN is voting on the quality of the argument rather the alignment of ideas. If you present a well reasoned contrarian idea, I don't think you would gather a lot of downvotes.
What gets downvoted are the really bad takes with lazy arguments.
Even if 90% of users are lurkers it doesn't mean they don't know how it feels to be downvoted and can't emphasize.
Good comments are rarely downvoted disproportionately on HN. Perceived censorship "by the bandwagon" just means it isn't a good comment.