Does it really matter if they cross?

They're lots of advice on saving seed  by isolating plants or seed heads but I can't find much about what happens if you ignore this and let nature take its course.

For example, last year I sowed a lot of old radish seeds together, just to give them a chance. Lots grew and they're flowering. There'll be red ones and white ones, round ones and long ones, summer and winter ones............but, in my view, all radishes taste the same. I don't think I've sown any F1 radish so any seed that is produced will be random radish. Does this matter? I don't think so. 
What about other veg? Beetroot are mostly red although there are yellow, white and stripey ones. I don't like white or stripey beetroot and only grow red or yellow ones. Maybe I'd end up with orange ones. Nice.

I leave kale and other brassicas to flower and, undoubtedly, they've crossed - but, if I only want a plant that provides green leaves for a meal does it matter? 

If anyone knows of information on the "risks" of letting same species plants cross pollinate, please point me to it. I really need to know whether a cross-pollinated seed is likely to produce something inedible, toxic, not worth the effort........