Depends on where he sits on the bell curve of fitness, and it also depends on what sports he has trained in. If he cycles everywhere but has never run a race in his life I'd be willing to bet that most casual female runners could beat him in a distance run. Likewise if he's used to running the average female amateur cyclist would most likely beat him in a cycling race. To simply assume he's fitter than the vast majority of women is wishful thinking on his behalf. If his fitness is pretty average compared to most guys then chances are quite a few women will beat him, even in his sport of choice. If he's very fit then the odds are more in his favour, however.
Mind you, it also depends on how you measure fitness. If you are looking at heart-rate and lung capacity he may not do as well as many women. If you're looking purely at who can beat who in a race, however, then men have the advantage due to their larger percentage of muscle mass. Unless you want to look at long distance swimming in which case women have the distinct advantage.
So the answer is yes... and no. It just depends.
Maybe you should just challenge him to a long distance swim instead!