I am not an expert by any means at all, and everything I know is really anecdotal. But, looking through the profiles on this website, you can see what people have achieved. None of them say "when I switched to veganism, things got much harder". So I find it unlikely that there is a disadvantage, of any significance.
The ratio of vegans to non-vegans is very, very low. Add to that, most vegans are actually not interested in fitness or bodybuilding (I think, primarily because the people who become vegan do it for ethical reasons, and are generally less concerned with what they look like, and more concerned with animal rights issues etc). So, there are going to be hardly any examples of vegan bodybuilders (in comparison to non-vegan bodybuilders). With such a difference in numbers, of course there will also be larger numbers of non-vegan bodybuilders who achieve larger size. I don't think that that indicates that it's harder for vegans, it's more likely to me that there are just less vegans who are aiming for that.
There may also be mistakes that people make when becoming vegan, but I don't think that means that veganism is "hard" to deal with. The changes needed (primarily, eating more of the right foods) are not complex.
Finally I want to add that it's not as clear cut as saying "non-veganism leads to better results than veganism". First of all, non-veganism and veganism are not black and white. It's not as though everyone who eats a non-vegan diet eats precisely the same meals as each other, and likewise, the same is not true of vegans. So it's impossible to compare. But even more importantly, just comparing the lifestyles on what they may do for your bodybuilding short-term (over a span of a few years), is ultimately irrelevant when you consider what high levels of animals products will do to you long term. I realise the following is a little extreme, but you could find a drug which pumps you up and makes your muscles huge for 10 years, then after that it kills you. You might say "people who don't take that drug are at a disadvantage in terms of bodybuilding" - but would you recommend people take it?
I dig your brain, Richard! Well said!