Yarrow and metadata

[marnanel]

Note that at the moment rgtp.thurman.org.uk and marnanel.org are both down. I will write this as thought they weren't. If you are looking for something to replace rgtp.thurman.org.uk until it comes back, please go here.  I may be posting more about the downtime later.  But this is not that post.

I would like your thoughts on a question of protocol design.

There is a system which I maintain called Yarrow. It is one of several clients for a bulletin board protocol called RGTP. To my knowledge, there are three bulletin boards in the world that use RGTP:

• GROGGS;
• a private one known to me;
• my blog.  The reason my blog uses RGTP is that recently I realised that I know Yarrow like the back of my hand, and it does about 90% of what I want out of a content management system.  And it also means that I have more of an incentive to keep it maintained, since I'm using it every day.

As to RGTP servers: GROGGS uses an RGTP server called simply "rgtpd", written in C by Ian Jackson.  The other two use an RGTP server I wrote many years ago called Spurge.  This means that I maintain both the server and the client for my blog.  However, the server is only accessible from localhost, and thus my blog is a special case because it is only accessed using Yarrow.  The other instances are also accessed using other RGTP clients, none of which I maintain.

When I set out to fix up Yarrow to work as a blogging platform as well as a bulletin board client, there were two principles I tried to keep in mind:

• some new features would be necessary, and
• any new features would not be permitted to interfere with existing pure-RGTP features.  Ordinary groggers should not have to know about any of it.  Certainly there should be no chance of GROGGS appearing in HTML, for example.

The most important of these new features is metadata, which I added to Yarrow for several reasons, the three most important among them being:

• to allow the content type to be specified (basic RGTP is text only; my blog needed to be HTML), and
• to allow the charset to be specified (basic RGTP is iso-8859-1; my blog needed to be Unicode), and
• to allow for tags.

The metadata is written inline at the starts of entries.  At present, however, the server is an almost unmodified Spurge, and knows nothing of the existence of metadata.  Therefore, the server cannot be asked for metadata information.  This means that if Yarrow's cache of metadata information gets out of date, Yarrow has to request the entire corpus from the server in order to rebuild the cache.

The RGTP standards document says in §2.2 that extension commands may be added.  I am considering adding a "META" command which retrieves only the metadata.  This would make everything a whole lot simpler and faster.  The disadvantage would be that metadata couldn't be used with an unpatched server, but this isn't necessarily a problem since I, and anyone else who might be using Yarrow for blogging in the future, will either be running Spurge or will be running some new server whose authors can be made aware of the extension.

What do you think?  Sensible?

I also need to work out how Yarrow can tell whether a server supports metadata in the first place:

• At present this must be set in Yarrow's configuration for each RGTP server.  This works, but it's ugly.  I would like to make it so Yarrow could essentially configure itself.
• I could make a rule that Yarrow tries the META command the first time it connects to any server, and if it's not recognised, it will remember that that server doesn't know about metadata.
• I could also make a rule that servers which support metadata must put "(META)" in the connect string, or something like that.  That would allow a server that didn't support metadata to start doing so without confusing all the clients.

What do you think?

Future thoughts:  One day I'd like:

• Yarrow to use genshi for templating so its HTML output could be easily redesigned (this is actually half written);
• Yarrow/Spurge to be debianised (separately or together) so people could install them more easily;
• Yarrow to be able to auto-configure itself using a browser on first install (like WordPress or MediaWiki are able to).
• Yarrow to have a way of interrogating Spurge instances running on localhost in order to make this auto-configuration simple.
• Maybe adding virtual hosting support to RGTP.  I have ideas about how to do this, but that can be another post.