9 out of 10 times people say things such as what you mentioned, it's because they simply want to justify their current behavior of not trying to improve themselves physically. They cling to laziness as if it's a proper defensive argument to use for keeping the status quo vs. actually having to do work and get out of their comfort zone. Remember, most of these people are the same ones who will say, "I don't lift weights because I don't want to get huge", partly because they're lazy, partly because they are ignorant of what it takes to actually get great results from training.
Ask these same people if eating fast food is "natural".
Ask them if it is "natural" to be obsessed with cell phones, Facebook, television shows, etc.
Ask them what is "natural" about most folks sitting on their butts all day staring at a computer or working a desk job, to where they wheeze and gasp from having to climb a flight of stairs because they're so out of shape (quite often, even when they don't appear to be in bad physical condition!)
And, finally, ask them if it's "natural" that we're significantly weaker than we were a hundred years ago when we actually had to go out and move/lift heavy crap on a regular basis, back when life wasn't so cushy and people actually had to be physically capable of doing physical work on a regular basis.
When people have made statements about things such as "I don't see what's so good looking about being big", I just retort that I don't understand what's enviable about being small and frail. When people say they think big muscles are "gross", I simply say that I feel the same way about seeing people who have minimal musculature who make excuses for never doing physical activity. I mean, if someone's going to take a shot at what I enjoy, I may as well say back exactly how I feel

Of course, not everyone needs to find a big, muscular physique to be their perfect ideal, but typically, it's laziness combined with envy that leads people to make derogatory comments about lifting heavy and building an impressive physique.
Not to mention, it isn't as if you have to live some crazy, unhealthy lifestyle to enjoy training to get bigger/stronger. Some people, of course, need to pack in food by the truckload to get bigger, but some of us can eat within what's considered a reasonable amount and still make progress. Most people have a lot to learn about training and what goes along with it, they're still in a mindset that's akin to when kids talk about sex on the playground when they're young; they might have a few actual facts, but most of it is just a lot of gibberish and stories they heard from questionable sources, making their general opinions pretty well uninformed and based on fallacy and illogic more than anything else.