Yarrow and metadata

[marnanel]

Note that at the moment rgtp.thurman.org.uk and marnanel.org are both down. I will write this as thought they weren't. If you are looking for something to replace rgtp.thurman.org.uk until it comes back, please go here.  I may be posting more about the downtime later.  But this is not that post.

I would like your thoughts on a question of protocol design.

There is a system which I maintain called Yarrow. It is one of several clients for a bulletin board protocol called RGTP. To my knowledge, there are three bulletin boards in the world that use RGTP:

• GROGGS;
• a private one known to me;
• my blog.  The reason my blog uses RGTP is that recently I realised that I know Yarrow like the back of my hand, and it does about 90% of what I want out of a content management system.  And it also means that I have more of an incentive to keep it maintained, since I'm using it every day.

As to RGTP servers: GROGGS uses an RGTP server called simply "rgtpd", written in C by Ian Jackson.  The other two use an RGTP server I wrote many years ago called Spurge.  This means that I maintain both the server and the client for my blog.  However, the server is only accessible from localhost, and thus my blog is a special case because it is only accessed using Yarrow.  The other instances are also accessed using other RGTP clients, none of which I maintain.

When I set out to fix up Yarrow to work as a blogging platform as well as a bulletin board client, there were two principles I tried to keep in mind:

• some new features would be necessary, and
• any new features would not be permitted to interfere with existing pure-RGTP features.  Ordinary groggers should not have to know about any of it.  Certainly there should be no chance of GROGGS appearing in HTML, for example.

The most important of these new features is metadata, which I added to Yarrow for several reasons, the three most important among them being:

• to allow the content type to be specified (basic RGTP is text only; my blog needed to be HTML), and
• to allow the charset to be specified (basic RGTP is iso-8859-1; my blog needed to be Unicode), and
• to allow for tags.

The metadata is written inline at the starts of entries.  At present, however, the server is an almost unmodified Spurge, and knows nothing of the existence of metadata.  Therefore, the server cannot be asked for metadata information.  This means that if Yarrow's cache of metadata information gets out of date, Yarrow has to request the entire corpus from the server in order to rebuild the cache.

The RGTP standards document says in §2.2 that extension commands may be added.  I am considering adding a "META" command which retrieves only the metadata.  This would make everything a whole lot simpler and faster.  The disadvantage would be that metadata couldn't be used with an unpatched server, but this isn't necessarily a problem since I, and anyone else who might be using Yarrow for blogging in the future, will either be running Spurge or will be running some new server whose authors can be made aware of the extension.

What do you think?  Sensible?

I also need to work out how Yarrow can tell whether a server supports metadata in the first place:

• At present this must be set in Yarrow's configuration for each RGTP server.  This works, but it's ugly.  I would like to make it so Yarrow could essentially configure itself.
• I could make a rule that Yarrow tries the META command the first time it connects to any server, and if it's not recognised, it will remember that that server doesn't know about metadata.
• I could also make a rule that servers which support metadata must put "(META)" in the connect string, or something like that.  That would allow a server that didn't support metadata to start doing so without confusing all the clients.

What do you think?

Future thoughts:  One day I'd like:

• Yarrow to use genshi for templating so its HTML output could be easily redesigned (this is actually half written);
• Yarrow/Spurge to be debianised (separately or together) so people could install them more easily;
• Yarrow to be able to auto-configure itself using a browser on first install (like WordPress or MediaWiki are able to).
• Yarrow to have a way of interrogating Spurge instances running on localhost in order to make this auto-configuration simple.
• Maybe adding virtual hosting support to RGTP.  I have ideas about how to do this, but that can be another post.

Comments

[pne]

I also need to work out how Yarrow can tell whether a server supports metadata in the first place:

• At present this must be set in Yarrow's configuration for each RGTP server. This works, but it's ugly. I would like to make it so Yarrow could essentially configure itself.
• I could make a rule that Yarrow tries the META command the first time it connects to any server, and if it's not recognised, it will remember that that server doesn't know about metadata.
• I could also make a rule that servers which support metadata must put "(META)" in the connect string, or something like that. That would allow a server that didn't support metadata to start doing so without confusing all the clients.

What do you think?

I'm reminded of (E)SMTP; perhaps you could do something similar?

As I understand it, how this works is something like this:

• The client connects to the server and greets it with EHLO ("extended HELO").
• If the server understands ESMTP, it will reply with a positive answer and also list the extension commands to SMTP that it understands. The client will then continue to speak ESMTP, possibly making use of the extension commands if they come in handy (for example, AUTH to authenticate).
• If the server does not understand ESMTP, it will not understand the EHLO command, and will reply with an error code. The client will then greet the server with HELO (the basic SMTP "hello" command), the server will reply with a positive answer, and the client will continue to speak only basic SMTP to the server.

So for your case, this would imply having two extension commands.

The first one would only query the server whether it implements any extension commands at all, and if so, which ones.

The second would then be your META command.

This would probably be a bit more general than just implementing META.

Then you would "only" have to convince authors of other extension commands to let their server also support the query-for-extensions command, and to list their extension command in the response.

And Yarrow would send the query-for-extensions command on connecting; if the server replies that it does not understand the command or if the server understands the command but does not list META in its list of supported extension commands, it knows that it cannot use META in this connection.

Whether you store this fact, and if so, for how long, is up to you; it'll depend on how often you expect server software to change (or how soon after a change you want to be able to make use of the new capabilities).

Quotation of Plato

[(Anonymous)]

Courage is knowing what not to fear.
Quotation of Plato

One Sentence from Einstein !

[(Anonymous)]

"Yes, we have to divide up our time like that, between our politics and our equations. But to me our equations are far more important, for politics are only a matter of present concern. A mathematical equation stands forever." - Albert Einstein !

What do you think ?