Help support TMP


"How Does Reindexing Work?" Topic


14 Posts

All members in good standing are free to post here. Opinions expressed here are solely those of the posters, and have not been cleared with nor are they endorsed by The Miniatures Page.

Please don't make fun of others' membernames.

For more information, see the TMP FAQ.


Back to the TMP Talk Message Board


Areas of Interest

General

Featured Hobby News Article


Featured Link


Featured Showcase Article

Red Sable Brushes from Miniaturelovers

Hobby brushes direct from Sri Lanka.


Featured Profile Article

Those Blasted Trees

How do you depict "shattered forest" on the tabletop?


Featured Book Review


2,278 hits since 27 Dec 2014
©1994-2024 Bill Armintrout
Comments or corrections?

Personal logo etotheipi Sponsoring Member of TMP27 Dec 2014 7:59 a.m. PST

WARNING: This is general computer science theory that probably applies in a reasonable way to TMP. I am posting it because a lot of people have legit interest in the maintenance downtime and thought it might help understand.

Imagine you have just introduced a young new wargamer to the historical side of the hobby. To mark this occasion, you presented her with her first military history book and a brand new bookshelf. Good on ya!

She places the book on the left end of the lower shelf. Over the next few months, she engages in a frenzy of acquisitions. Many of her books deal with different periods. She also realizes that there are other relationships among her books besides chronology. She learns to see parallels between the Early Roman Empire and the British Empire of the 19th Century. She finds an underlying thread in civil uprisings across the centuries. Awesome!

Day after day, she adds to her collection, putting each book adjacent to the last, then moving up a shelf when necessary.

One day, she goes in to her one shelf full library to find her books on Tunnni and Sparticus for some comparative reading with respect to her new book on Toussaint L'Ouverture. Crap, where are those two? She scans the shelf for about five minutes and learns a very important adult life lesson about staring right at something you are looking for but not being able to see it.

OK. She pulls all the books down and puts them back on the shelf in chronological order. This is reindexing.

Having done that, she is happy. And her collection continues to grow. Two shelves into her next shelf, she gets the idea to grab the book on the colonial wars in New Zealand that she got a couple months ago and re-read it along with that book on Haitian colonization. The Haitian book is on the old shelf, in order, but the New Zealand one is somewhere in the new two shelves. So she decides to reindex again. Doing this, she has to both scan the old shelves to find new places, and make room for them by sliding books around.

She realizes that this takes quite a while, but considers it to be worth the work since she is more frequently looking for, retrieving, and reading old books than buying new ones. Obviously, you have failed to impart an important lesson about the hobby.

Another time saving method she came up with was "proxy books" Instead of having just a chronological sorting of her library, she has other sections arranged by topics (like slave revolts and colonial wars). Instead of having copies of the books for each section, she puts in a placeholder for them, with a note where to get the actual book. Again, a great time-saver for content searching, but it does add to the time to reindex.

After she gets he fifth shelf done, she decides to change her approach. Instead of waiting for a period of time to elapse or a certain number of unsorted books to accumulate, she decides to just put each new book and its proxies in their appropriate place once acquired. This is a lot faster than a full resort, but it still is a function of the total number of books and you have to do it every acquisition. After some quick calculation, she decides that the total time spent is about the same, but doing it the new way seems far less fatiguing, so, again awesome!

Another time saver she came up with was to leave some space on shelves. She learned this by noticing that libraries leave empty space on the shelves to aid in the insertion of new acquisitions. Librarians are obviously very intelligent and awesome people. The drawback is, of course, that for eight shelves of books, she now has to have ten total shelves. And every once in a while, she has to do a big move when the local extra shelf space fills up.

Now, four years later, she has decided to reevaluate her collection. Even with the efficiencies, its becoming a tough row to hoe to keep up with new books. Well, there are several books, that while foundational, she hasn't touched in a few years. And there are a number of "fluff" books that while fun when they were new, just really don't contribute anything now (like that book of Latin swear words or the one of funny Victorian idiomatic expressions). OK, but to do a purge, she really wants to take a hard look at what she has, how she uses it, what the losses would mean, and ultimately is it worth it.

The next chapter is up to you …

MajorB27 Dec 2014 8:32 a.m. PST

Interesting. And your point is?

GarrisonMiniatures27 Dec 2014 9:06 a.m. PST

Only thing I can think of is why TMP has 2 hours downtime every day when no one else needs this.

Roderick Robertson Fezian27 Dec 2014 11:00 a.m. PST

After four years she only has eight shelves worth of books?

Tin hat27 Dec 2014 11:02 a.m. PST

Etotheipi, Never be tempted to write a set of rules .

Martin Rapier27 Dec 2014 11:03 a.m. PST

I think we all know what re-indexing is, however post 1990 era database technology can do this without locking all the users out of the system, if it needs to do it all.

I wonder if TMP is written using a network database rather than a relational one?? Certainly TurboIMAGE used to need a shutdown for re-indexing as it was physically shoving pointers to records around.

KatieL27 Dec 2014 12:13 p.m. PST

AFAICT, it's an SQL db (MySQL?) with a Miva presentation layer. It is, in fact, pretty much 1990s technology.

Miva isn't a terribly powerful language; I'd think that the indexing is necessarily done by an external system which therefore requires exclusive access.

If you're going to start writing background processing to index articles you're probably going to need a bit more of a computer science background and a proper programming language.

Personal logo etotheipi Sponsoring Member of TMP27 Dec 2014 1:16 p.m. PST

Interesting. And your point is?

I think we all know what re-indexing is,

From a number of the posts on the maintenance and ads threads (as well as the many and frequent claims to know nothing about how computers work), it was apparent that a decent number of posters that had no idea of the requirements. The point was to put the relevant points of the maintenance issue in a context that someone without a computer science background could understand.

however post 1990 era database technology can do this without locking all the users out of the system, if it needs to do it all.

I think that hits at the crux of the problem, that Bill has been hand rolling the code himself on a minimal service host to keep overhead low, which allows him to work within the current margin. It looks like TMP 4.0 is going to necessitate a move away from both of those propositions, as well as some additional training.

AFAICT, it's an SQL db (MySQL?) with a Miva presentation layer. It is, in fact, pretty much 1990s technology.

Miva isn't a terribly powerful language; I'd think that the indexing is necessarily done by an external system which therefore requires exclusive access.

If you're going to start writing background processing to index articles you're probably going to need a bit more of a computer science background and a proper programming language.

Yeah, but as you say Miva is only the front end, so one would hope that it isn't doing any of the heavy lifting. You could easily use that to present the current user interface to pretty much anything on the backend.

I didn't talk about background processing (the parallel for the above exemplar would be hiring someone to sit around and reindex your books for you while you are buying and browsing, something that reduces/near eliminates delay but comes with its own overhead). Writing that type of background processing for the server side is an undergrad project at worst, but you could easily buy or likely find one for free that already does that.

Etotheipi, Never be tempted to write a set of rules.

Not sute why, but, sorry, already done.

deephorse27 Dec 2014 4:00 p.m. PST

Thank you etotheipi. Just ignore some of the less than helpful posts above. Being barely computer literate I welcomed your topic.

Personal logo etotheipi Sponsoring Member of TMP27 Dec 2014 4:03 p.m. PST

@deephorse, Thanks.

And, to be fair, I didn't do a great job explaining why I wrote it in my intro.

After four years she only has eight shelves worth of books?

Well, it is a fictional exemplar… :)

Personal logo Editor in Chief Bill The Editor of TMP Fezian27 Dec 2014 9:02 p.m. PST

Miva isn't a terribly powerful language…

Yes, but it was in the right place at the right time, and it was free… grin

It can also do some truly amazing things… there are some cool tricks in the old girl.

Personal logo Editor in Chief Bill The Editor of TMP Fezian03 Jan 2015 6:59 p.m. PST

Bill, does Miva have its own database system?

Original Miva used a modified DBF system; later, it added SQL support.

Personal logo etotheipi Sponsoring Member of TMP04 Jan 2015 2:23 p.m. PST

I shall use something similar to explain the whole idea of indexes to beginning Oracle users.

Thank you for the compliment. Have at it!

I don't know if it's a standard explanation these days or not: except for CS courses I did in 81-83 which were on fortran

My most recent "formal" db training was in my doctoral classes for Oracle 7 (which feels like it was much more than two decades ago…), but I've built and directed development on tons of dbs since then. I've become much more curmudgeonly since then and feel that libraries on top of 4th generation languages have taken all the programming out of programming. On the other hand, I am blissfully happy that kids today lack the skills to steal a car with a manual transmission…

This is all like contemplation on the nature of the universe in a way, isn't it?

That's possibly the most important step in programming…

Sorry - only verified members can post on the forums.