Monument

So, is http://joule.marnanel.org working for you? If it isn't, tell me. If it is, tell your friends.

A few reports are coming in of Twitter/identi.ca support reporting that valid accounts don't exist. I'll be investigating.

Introduction. I've had a few people ask me what happened to *.marnanel.org, which has been down for several days now. The short answer is that there were software memory problems in each case; the long answer differs for each application.

joule.marnanel.org: I'm putting this first because I expect most people reading this will want to know about Joule.

One of the main parts of Joule is the comparator, which compares the old state of your friends list with the new. The old comparator, "currant", had worked fine for around a year, but could only compare about 1000 records a second. Several months ago I introduced Twitter support to Joule, and because I wondered whether people with millions of followers might like to use it, I rewrote the comparator entirely, producing "raisin". This was a bad move for two reasons:

Firstly, although it worked, if you have millions of followers you also have hundreds of friendings and unfriendings every day, more than anyone would want to wade through. So Joule is rather useless for such people.

Secondly, the new comparator compared everything in memory rather than in the database. Although it worked with large datasets in testing, when it was put into production there were some scalability issues. Eventually it allocated so much memory that it crashed the server Joule was running on, which isn't acceptable because it's used for many other things and by many other people.

I thought I fixed both of these problems by adding a check that a user didn't have more than a few thousand followers. However, it seemed that this wasn't enough, and the server eventually crashed again. The server admins asked me, quite reasonably, not to run Joule in its present state on that server.

Ways forward from here.

I don't know. Out of all the sites, this is the toughest one to bring back. Unfortunately, it's also by far the most used.

I could revert back to "currant", although that would be fiddly because the database structure is rather different and I'd need to write something to convert back to the old format.

I suppose I could run Joule in its present state with "raisin", on a dedicated server.

I could convert Joule to CGI so that it was easier to profile.

The Yarrow sites: marnanel.org and rgtp.thurman.org.uk. These are less of a problem because they're CGI, not in-process. The problem with them is that they occasionally load the entire index into memory while creating a cache of it; this is easily fixed.

The Shavian wiki. This was also a memory hog for a quite different reason: it cached all the transliterations while rendering. I could easily turn this off, but it would become very slow. Ways forward: I would actually like to do the rendering in a separate script, so I wasn't writing it in MediaWiki and therefore PHP any more; I'd like it if it could render text taken from other sources than its own wiki, such as simple.wikipedia.org and en.wikisource.org; I'd like it if it could take pronunciations from another, less perfect source such as CMUdict where they weren't already supplied, except in wiki-building mode, and if it could make some attempt at automatic disambiguation. This is a medium-sized rewrite, though, but this may be a good excuse for it.

Some things that could be done to Joule, mainly for my own reference.  Not in order. I've shown the amount of work needed; I haven't ascribed an importance to any of these (though I wouldn't mind hearing your opinions).

1. Joule is case-sensitive.  None of the systems it serves data from are case-sensitive.  This is silly.  This will probably require downtime to fix, because effective duplicates will need to be removed from the database. Medium
2. The translation system needs a radical overhaul.  I have several ideas.  In particular, the English text should be placed within the templates, as with gettext, and not within a magic .po file; and ?lang=fr etc should be pages, not redirects, for the benefit of search engines.  Complex
3. Controls overhaul. Easy
4. Look into OpenSocial so we can chart Blogger and MySpace. Medium
5. There should be a table of messages of the day.  The HTML pages should show the most recent, and the RSS feeds should show whichever was the most recent on the relevant day.  This will let us put interesting messages about new features into RSS feeds, which is the only way to contact most of our users. Medium
6. joulestats is stable and can be run from cron: done.  Also, fix joulestats's messages for users with zillions of followers; they're less helpful than they could be. Easy
7. Page view per day so that massive charts become at least slightly useful. Medium
8. Add an extra column showing the total number of followers on each day, for the same reason.  This needs a current count to be returned from the XS and then we just add and subtract as we go down the line. Easy
9. The FAQ needs to be broken out into separate pages. Easy
10. Dreamwidth support, when this enhancement is finished. Easy
11. Most of the Twitter and identi.ca work needs to be done in a superclass rather than duplicating code. Easy
12. I would like a way to draw line graphs of number of followers over time.  (This is blocked by "controls overhaul".) Complex

I have a ton of other stuff to do, so Joule only gets worked on now and then.  But feel free to advocate for any particular one of these.  Also, feel free to send patches or to ask for help making them.  And I'd like to hear any other suggestions you have.

Ryan Tucker reported a bug in Joule. When a user has more than 5,000 followers, on some days Joule will throw a database error about a duplicate key. This is mysterious, since the keys come from a hash and should be unique. I thought I'd try liveblogging fixing it, in case anyone wanted to watch.  Times are EST.

• 18:30: Can we replicate it in staging?  I don't want to bring the real Joule down while I look for a fix.
• 18:39: Yes, astronautics on Twitter has >5,000 followers and causes Joule to exhibit the bug.
• 18:44: Okay, Joule is instrumented so it will dump the old and new lists to a file, plus what it thinks the changes are.
• 18:45: It failed again and I have the log file.  Good!  I hate when you set up debugging and it suddenly starts working.
• 18:48: Well, it's not because there are duplicates in the old or the new lists, so it must be a comparison error.
• 19:01: Fascinating.  The new version of the comparison code is reporting one of the userids as both added and removed, which the DB constraints obviously won't allow.  This didn't come up in testing...
• 19:24: Seems that when you have two users A and B, and A's name is a prefix of B's, and A unfriends you, that the system gets confused and reports B having both friended and unfriended you.  Fixing now.
• 19:31: I think I have a solution.  Taking out all the instrumentation to test it.
• 19:36: Tests pass.  So do old tests.  Will write a regression test in a few minutes.
• 19:37: The moment of truth... yes!  it works on staging.  Rolling out to production.
• 19:52: Fix checked in and in production.  The remaining problem here is that astronautics had 3064 follows and 2421 unfollows, and Joule is fixed so that it shows "Hiccup" if you have more than 100 on the same day (for three reasons; I could tell you, but does anyone care?)  Suggestions for working around this one are welcome.

If anyone cares, the three reasons for "Hiccup" are:

1. We have to do a separate lookup in Twitter for every userid we haven't seen before, to get the icon and username.  For 5000 changes in a day, that slows page load times a lot.  This is still a problem.
2. There is an old pre-Twitter assumption that 100 follows or unfollows means either that Joule broke, or that LJ broke when it sent us the names.  Clearly this is outdated.
3. There isn't enough space in the chart for more than a few hundred names a day without making the page insanely long.

Carmen has suggested replacing "Hiccup" with a link to a sub-page which displays all the names for that day, possibly allowing paging through them to get around the first problem I mentioned.  I think this is a very useful idea.

I'm trying to think how to tell more people about Joule rather than just by me blogging about it.

1. People can tell their friends. This is the best way, and happens quite a bit, but I can't do much about it either way. Of course, all the other ways feed back into this way too.
2. http://twitter.pbworks.com/Mashups might be worth editing, but I'm not sure I should add my own stuff there or I look like a spammer.
3. How do you get listed on http://laconi.ca/trac/wiki/Apps anyway?
4. No chance of getting linked from Wikipedia, but it would certainly bring people over.
5. Running a few Google ads might work, but might be expensive. Still, for a few days it might be worthwhile.
6. I could email various blogs and tell them about it; they might be interested.
7. I wish Joule was interesting enough that someone would write an article about it (from a "tool with a six-year history" angle, perhaps, rather than a "just a friends-list tracker" angle). But maybe it isn't.