After seeing the comments on my brief X-Men post the other day, I was able to clarify my thoughts on the matter, in such a way as that I realize now that I was putting my question in an awkward, slightly specious fashion. The question is not: why have the space-opera elements become such an integral part of the X-mythos despite a presumed thematic incongruity; but why have these elements continued to be regarded as central to the mythology long after Chris Claremont has ceded his role as the franchise's chief architect? Although he's been back on and off the central and periphery X-titles for some time, Claremont's historically central run on Uncanny X-Men lasted for seventeen years and ended in 1991. He hasn't been in the "driver's seat" on Uncanny for almost as long as he actually was - and his various returns to the books have been notably unspectacular for a number of reasons. Regardless of one's opinions on Claremont's current skill as a writer, the X-Men books are not now even what they were in 1991. It's to his credit that the book - and amazingly it was just one book, for years and years - was as coherent as it was under his stewardship. The fact that they soon ballooned out of all recognizable proportions is probably a testament to his success as much as anything else.
So: go back to the mid-70s, when the then-adjectiveless X-Men book was still just a foundling, trying to regain its footing after years of limbo. The team was a group of multi-cultural misfits who barely seemed to get along, let alone work with the kind of precision expected from most established super-groups. The mutant angle was just another bit of accrued baggage, hardly that much more important than any number of other bits and pieces. Who remembers that Professor X was initially crippled by a wannabe alien warlord named Lucifer? Or that one of the biggest storylines in the original series' run had been a long battle with the alien Z-Nox, or that their most fiendish foes had been the less-than-sinister Cold War-relic Factor Three? There was a reason why X-Men was, with a few notable exceptions, considered the runt of the 60s Marvel litter - even Sgt. Fury lasted longer. So Claremont's various stock plots and motifs - space opera (the Shi-ar, the Brood), demonic invasion (Belasco, N'astirh, the N'Gari), S&M (the Hellfire Club, Rachel Summers, the Fenris twins), and off-the-wall stuff that doesn't really fit anywhere else (Dracula, ninjas, cyborgs, ninja cyborgs) - didn't relate back to any kind of central thematic bailiwick so much as simply acted as plot engines for the franchise. Reading the early Claremont issues, you definitely get the feeling of someone making it up as he goes along - even fairly well into the run there's still a free-wheeling energy pushing the book forward at a breakneck pace. That sense of momentum was what made it such a monster hit, and pretty much set the standard for post-Bronze age superheroics.
But eventually even Claremont's industrious energy began to falter, and the Logic of Empire began to assert itself. Spin-offs began to accrue - slowly at first, but eventually at a break-neck pace. Perhaps the slightly aimless direction of the post-200 run - ostensibly so because he had been forced to change directions after being told that Marvel UK plot elements were unavailable for his use - was a conscious effort on his part to avoid turning the book into the kind of static franchise he probably dreaded. Considering just how massively popular the book was, it is amazing he was able to run it with as much of a free hand as he did. Considering how weird and dispirited many late-80s X-Men books were, it is likely that he saw the writing on the wall. He could only keep the team atomized and beset for so long, and once the 90s had kicked into high-gear there were already spin-offs on top of spin-offs coming at a feverish rate. He was let go, and the books almost immediately calcified into the staid institution we see now. Inasmuch as any kind of commercial blockbuster can be considered an idiosyncratic personal work, X-Men seemed to belong wholly to Chris Claremont. When he left, the series became pretty ugly, essentially a paramilitary soap opera acted out by geentically pure ubermeschen with vague allusions to prejudice peppered gingerly throughout.
And it is on these vague allusions where the franchise most often flounders. Claremont was in no way a "subtle" writer, but he was smart enough to keep his subtext firmly on the back burner: obviously, anti-mutant prejudice became a much bigger deal than it had been under Roy Thomas and Werner Roth, but he was smart enough to keep the tone generally light (until he got to the rather ponderous middle 200s, that is) and use the subtext to inform the plot, not the other way around. It's all about fighting and soap-opera, and that's what it's always been about: give the X-Men someone / thing to fight that serves as a reflection of their metaphorical struggle, or even better, their interpersonal struggle. Whatever subtlety Claremont had brought to the enterprise left with him, and soon the X-Men saga became as deeply mired in eugenical debates and ceaseless pontification as it possibly could - the sneaking tendencies of Claremont's later run were amplified, and soon you had whole issues set aside simply for "woe-is-me" angst. It didn't seem to effect sales at all, even though those paying attention couldn't help but see that the sheer size and scope of the franchise had effectively halted the momentum that had made the book not merely popular but pretty good - perhaps the best exemplar of well-made trashy pulp comics circa 1975-1990. What was left wasn't particularly good anymore, but it was popular. And I guess that's all anyone really needs . . .