True, almost all of the QB's, especially the ones having at least a reasonable amount of success, are at least 6 feet tall, and even then, that's on the short end of things.
Just off the top of my head, the only two in recent history who were under that benchmark are Russell Wilson and Kyler Murray.
A QB has to be able to see over those earth movers blocking for him, and every inch counts if you're below that benchmark. In today's game, those five big ugly guys protecting you have gotten a couple of inches taller compared to what we saw 20-30 years ago, and now even 6' 3" is considered "short" for an offensive guard, much less a tackle...
When you're young, strong, and can run around in the pocket, or even scramble, height isn't quite as much of an issue, but with all of those QB runs, sooner or later, they're going to take their toll on you.
Drew Brees may have been listed as 6' 0", but when you get to that age, and spinal compression occurs, along with flattening of the feet, I wouldn't be surprised if he were closer to 5' 10" by the end of his career. He's probably the exception to the above rule, since he was so familiar with his receivers, their timings, their routes, and he was so accurate and precise with his throws, that he's always been able to get away with it.