Remember, this site was built before there were today's website systems. Essentially, TMP is its own content management system.
Mostly it is due to TMP running on custom code that Bill developed.
Without seeing the code and going off other comments (like the expanding maintenance time), I'm guessing it is mostly due to the amount of data in the system growing without end. Hopefully, the algorithm is a polynomial time one (which is about your best option).
Linear growth kind of means the amount of data in your system (number of posts, members, advertisers, etc.) is roughly the "speed" of your reindexing; the more you have, the longer it takes. Polynomial growth means the amount of data is like your "acceleration"; the more you have, the more each additional element adds. Hopefully we're not geometric or exponential (And don't get me started on people saying "exponential growth" when the thing is actually "geometric growth".).
One big challenge, is that (I believe) TMP keeps stuff from time immemorial. Most current social media systems trash old stuff, for efficiency but also because LOLcats from three years ago aren't relevant and will probably be reposted anyway.
I would guess that we would rather keep old stuff for reference sake. On the flip side, there is probably a lot of old stuff (ephemeral and/or non wargaming stuff) we could prune without harm, too.
One of the challenges of pruning is you might prune something from three years ago that someone linked to two years ago. Now that linked thing is gone.
One alternative to going to a more current (I refuse to call them "modern" because of a number of reasons, not the least of which is I am a grumpy old man), might be a purge. A reasonable purge might be things older than (arbitrarily) five years that (1) aren't linked enough (arbitrary standard), and (2) don't hit high enough (arbitrary standard) in the top 95% (arbitrary standard) of content searches.
Of course, the arbitrary parameters could (and should) be debated. They can also be easily "tested" by scanning the db for the number/type of records involved. And there should be an elegant solution added for a broken internal link. Something nice with bricoles and lace, I would think.
[Along those lines, ganking the several "Radio Days" threads out there might just give a nice performance boost all by itself.]
Of course, this doesn't solve the polynomial time problem (which isn't really solvable), but if it reduces the maintenance time and puts off the critical point for a hundred years, it might suffice.
-----
On another point, Bill could check the usage of the system by time zone and shift the maintenance to a different window to deal with the midnight US west coast/8am UK issue. Of course, this still screws someone, but isn't democracy all about screwing the few for the convenience of the many?
-----
TMP 4.0 will move to a new database system, so maintenance time should be zero.
I'm guessing this means in-line maintenance, which wouldn't be zero. That means that instead of doing a big block of maintenance all at once, the system does a little bit of maintenance each post. Kind of like the difference between a bachelor washing one dish, fork and pot per mean or a family storing dishes up in the dishwasher for a big run.
Of course, that wouldn't mean zero maintenance. From a few SWAGS and some envelope calculations (based on estimates of current workload), it would mean something like between .1 and .2 seconds per post. Non-zero, but unnoticeable for the current application.
But it would also grow at (at least) the polynomial rate, unless the store-bought CMS also came with some inherent purging system. Like above, this shift might also put off the critical point where performance becomes a problem for a century, in which case, so what?
So, Great and Powerful Editor (pay no attention to the man behind the curtain), are we getting purges with our upgrade?
[My meds are starting to wear off, so I will take my response offline.]
thanks everyone who helped after her family's recent fire.
