Upgrading campus.dokeos.com to 1.8.5

Tonight I'm upgrading campus.dokeos.com to dokeos 1.8.5. One of the things that is difficult to put in a human mind is the number of users and courses on this portal and the implication it has on upgrade time. Over the last 5 days, 600 new users have registered on this portal, which made it pass the 200K users for the very first time. The portal still works nicely. It's now on a single cuad-core server, but it used to work on a Pentium 4 for 125K users. That's the beauty of Dokeos. However, upgrades to newer versions are always a difficult task. For example, at the moment we are adjusting the permissions on the "courses" directory. Having 29000 courses, we also have 29000 directories. In the past, we tested the limit of the number of files you can get into one single directory on a Pentium 4, by letting those 125000+ users we had at that time upload their picture. Well, the limit is approximately 117000, so we already had to split the user pictures directory into sub-directories about one year ago when we reached that limit. Obviously we can now reach the million before that happens again! But, with 29000 courses in one directory, we are starting to think that spreading the courses over subdirectories might not be such a bad idea either... Well anyway, for 29K directories, updating the permissions (with a simple chmod -R command) takes more than an hour on a very good hard disk. Doing it on the users upload directory also takes something like that. Then comes the real upgrade. When monitored, the main database updates in less than a minute, but the courses databases (considering they are also 29K, so 29K directories in another directory, each of these directories containing 200 tables) take about 2-3 seconds each to update, which means it takes about 60000 (60K) seconds to update the whole portal, which is about 16 hours! Yes, that's what we have to do at each new version to keep the free portal running. While the process is running, we can do something else, yes, but we have to keep an eye on the process in case it would just stop, unfinished, so we can fix it and start again. So we organize turns to watch over the process, Arnaud and I. I'm writing these lines as the process started since 2 and a half hour ago, and it's just migrated 5000 databases. Only 24000 to go. This is why it is such a painful process for you, users. This is a little update: the campus migration has been processed in less than 6 hours in the end (the automatic process itself), which is considerably less than what the first databases migration rythm showed, so we could get the campus back in place quite quickly.