Selective breeding and genetic modification are two very different beasts.
Selective breeding ensures that particular traits become more prominent (for example, a less "stringy" mango) by breeding together varieties that possess said desired trait. It's perfectly natural.
Genetic modification involves going down to DNA level and "splicing" new genetic information in there which can come from an entirely different organism. This would never otherwise occur in nature.
That last part isn't strictly true, since cross-species and even cross-genera hybrids occur in nature (and even more is possible with bacterial genetic transfer), but I take your point, which is that it's extremely rare, whereas GM lets us wrap whatever we want with a plasmid and fire it into other organisms.
However, just because cross-organism modification is possible doesn't mean it's dangerous, nor does "natural" crossbreeding imply safety. One doesn't follow from the other. Plenty of natural crossbreeding results in plants unsafe for human consumption. In fact, that's where all of them came from. There's still risk when the "natural" method is directed by humans as well, since many of our foods have harmful dormant genes, like much of the nightshade family. Crossbreeding can inadvertently activate those genes.
GM pulls genes from different species, sure, but those genes are selected because we know what they do. In that sense, GM is a scalpel. We don't have to smoosh together big chunks of genome and hope for a useful expression. We can inject specific information, verify that it integrated intact, and even see where on the genome it was incorporated.
Of course, that doesn't mean GM is automatically safe either. That's why we test them both.