Status of the sites
Sep. 22nd, 2009 10:20 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
marnanelIntroduction. I've had a few people ask me what happened to *.marnanel.org, which has been down for several days now. The short answer is that there were software memory problems in each case; the long answer differs for each application.
joule.marnanel.org: I'm putting this first because I expect most people reading this will want to know about Joule.
One of the main parts of Joule is the comparator, which compares the old state of your friends list with the new. The old comparator, "currant", had worked fine for around a year, but could only compare about 1000 records a second. Several months ago I introduced Twitter support to Joule, and because I wondered whether people with millions of followers might like to use it, I rewrote the comparator entirely, producing "raisin". This was a bad move for two reasons:
Firstly, although it worked, if you have millions of followers you also have hundreds of friendings and unfriendings every day, more than anyone would want to wade through. So Joule is rather useless for such people.
Secondly, the new comparator compared everything in memory rather than in the database. Although it worked with large datasets in testing, when it was put into production there were some scalability issues. Eventually it allocated so much memory that it crashed the server Joule was running on, which isn't acceptable because it's used for many other things and by many other people.
I thought I fixed both of these problems by adding a check that a user didn't have more than a few thousand followers. However, it seemed that this wasn't enough, and the server eventually crashed again. The server admins asked me, quite reasonably, not to run Joule in its present state on that server.
Ways forward from here.
I don't know. Out of all the sites, this is the toughest one to bring back. Unfortunately, it's also by far the most used.
I could revert back to "currant", although that would be fiddly because the database structure is rather different and I'd need to write something to convert back to the old format.
I suppose I could run Joule in its present state with "raisin", on a dedicated server.
I could convert Joule to CGI so that it was easier to profile.
The Yarrow sites: marnanel.org and rgtp.thurman.org.uk. These are less of a problem because they're CGI, not in-process. The problem with them is that they occasionally load the entire index into memory while creating a cache of it; this is easily fixed.
The Shavian wiki. This was also a memory hog for a quite different reason: it cached all the transliterations while rendering. I could easily turn this off, but it would become very slow. Ways forward: I would actually like to do the rendering in a separate script, so I wasn't writing it in MediaWiki and therefore PHP any more; I'd like it if it could render text taken from other sources than its own wiki, such as simple.wikipedia.org and en.wikisource.org; I'd like it if it could take pronunciations from another, less perfect source such as CMUdict where they weren't already supplied, except in wiki-building mode, and if it could make some attempt at automatic disambiguation. This is a medium-sized rewrite, though, but this may be a good excuse for it.
joule.marnanel.org: I'm putting this first because I expect most people reading this will want to know about Joule.
One of the main parts of Joule is the comparator, which compares the old state of your friends list with the new. The old comparator, "currant", had worked fine for around a year, but could only compare about 1000 records a second. Several months ago I introduced Twitter support to Joule, and because I wondered whether people with millions of followers might like to use it, I rewrote the comparator entirely, producing "raisin". This was a bad move for two reasons:
Firstly, although it worked, if you have millions of followers you also have hundreds of friendings and unfriendings every day, more than anyone would want to wade through. So Joule is rather useless for such people.
Secondly, the new comparator compared everything in memory rather than in the database. Although it worked with large datasets in testing, when it was put into production there were some scalability issues. Eventually it allocated so much memory that it crashed the server Joule was running on, which isn't acceptable because it's used for many other things and by many other people.
I thought I fixed both of these problems by adding a check that a user didn't have more than a few thousand followers. However, it seemed that this wasn't enough, and the server eventually crashed again. The server admins asked me, quite reasonably, not to run Joule in its present state on that server.
Ways forward from here.
I don't know. Out of all the sites, this is the toughest one to bring back. Unfortunately, it's also by far the most used.
I could revert back to "currant", although that would be fiddly because the database structure is rather different and I'd need to write something to convert back to the old format.
I suppose I could run Joule in its present state with "raisin", on a dedicated server.
I could convert Joule to CGI so that it was easier to profile.
The Yarrow sites: marnanel.org and rgtp.thurman.org.uk. These are less of a problem because they're CGI, not in-process. The problem with them is that they occasionally load the entire index into memory while creating a cache of it; this is easily fixed.
The Shavian wiki. This was also a memory hog for a quite different reason: it cached all the transliterations while rendering. I could easily turn this off, but it would become very slow. Ways forward: I would actually like to do the rendering in a separate script, so I wasn't writing it in MediaWiki and therefore PHP any more; I'd like it if it could render text taken from other sources than its own wiki, such as simple.wikipedia.org and en.wikisource.org; I'd like it if it could take pronunciations from another, less perfect source such as CMUdict where they weren't already supplied, except in wiki-building mode, and if it could make some attempt at automatic disambiguation. This is a medium-sized rewrite, though, but this may be a good excuse for it.
