marnanel: (Default)
Note that at the moment and are both down. I will write this as thought they weren't. If you are looking for something to replace until it comes back, please go here.  I may be posting more about the downtime later.  But this is not that post.

I would like your thoughts on a question of protocol design.

There is a system which I maintain called Yarrow. It is one of several clients for a bulletin board protocol called RGTP. To my knowledge, there are three bulletin boards in the world that use RGTP:
  • a private one known to me;
  • my blog.  The reason my blog uses RGTP is that recently I realised that I know Yarrow like the back of my hand, and it does about 90% of what I want out of a content management system.  And it also means that I have more of an incentive to keep it maintained, since I'm using it every day.
As to RGTP servers: GROGGS uses an RGTP server called simply "rgtpd", written in C by Ian Jackson.  The other two use an RGTP server I wrote many years ago called Spurge.  This means that I maintain both the server and the client for my blog.  However, the server is only accessible from localhost, and thus my blog is a special case because it is only accessed using Yarrow.  The other instances are also accessed using other RGTP clients, none of which I maintain.

When I set out to fix up Yarrow to work as a blogging platform as well as a bulletin board client, there were two principles I tried to keep in mind:
  • some new features would be necessary, and
  • any new features would not be permitted to interfere with existing pure-RGTP features.  Ordinary groggers should not have to know about any of it.  Certainly there should be no chance of GROGGS appearing in HTML, for example.
The most important of these new features is metadata, which I added to Yarrow for several reasons, the three most important among them being:
  • to allow the content type to be specified (basic RGTP is text only; my blog needed to be HTML), and
  • to allow the charset to be specified (basic RGTP is iso-8859-1; my blog needed to be Unicode), and
  • to allow for tags.
The metadata is written inline at the starts of entries.  At present, however, the server is an almost unmodified Spurge, and knows nothing of the existence of metadata.  Therefore, the server cannot be asked for metadata information.  This means that if Yarrow's cache of metadata information gets out of date, Yarrow has to request the entire corpus from the server in order to rebuild the cache.

The RGTP standards document says in §2.2 that extension commands may be added.  I am considering adding a "META" command which retrieves only the metadata.  This would make everything a whole lot simpler and faster.  The disadvantage would be that metadata couldn't be used with an unpatched server, but this isn't necessarily a problem since I, and anyone else who might be using Yarrow for blogging in the future, will either be running Spurge or will be running some new server whose authors can be made aware of the extension.

What do you think?  Sensible?

I also need to work out how Yarrow can tell whether a server supports metadata in the first place:
  • At present this must be set in Yarrow's configuration for each RGTP server.  This works, but it's ugly.  I would like to make it so Yarrow could essentially configure itself.
  • I could make a rule that Yarrow tries the META command the first time it connects to any server, and if it's not recognised, it will remember that that server doesn't know about metadata.
  • I could also make a rule that servers which support metadata must put "(META)" in the connect string, or something like that.  That would allow a server that didn't support metadata to start doing so without confusing all the clients.
What do you think?

Future thoughts:  One day I'd like:
  • Yarrow to use genshi for templating so its HTML output could be easily redesigned (this is actually half written);
  • Yarrow/Spurge to be debianised (separately or together) so people could install them more easily;
  • Yarrow to be able to auto-configure itself using a browser on first install (like WordPress or MediaWiki are able to).
  • Yarrow to have a way of interrogating Spurge instances running on localhost in order to make this auto-configuration simple.
  • Maybe adding virtual hosting support to RGTP.  I have ideas about how to do this, but that can be another post.

yarrow down

Sep. 8th, 2009 09:46 am
marnanel: (Default)
Yarrow on Dorothy is down with some kind of memory issue; I'll fix it when I have a moment. Groggers may find an alternative installation on chiark.

Anyone who is any kind of stakeholder in Dorothy and would like to know when things are down, or maintenance is planned, should join the Google group "friends-of-dorothy".
marnanel: (Default)
I've been thinking about the way Spurge works, and how it should be debianised.  Do any of you have strong opinions as to whether services whose traffic is not great should
  1. be launched from inetd or xinetd?
  2. listen for themselves?
  3. be able to do both, and the administrator can choose according to expected load?  This appears to combine the faults of both schemes.
Pros and cons:
  • The process is constantly running with (2), and not with (1).  Then again, if you're only running [x]inetd in order to run one service, you're running one extra process all the time anyway.
  • Configuration (by the user or the package) is far simpler with (2), especially because xinetd and inetd have different configuration file formats.
  • with (2), the service has to be capable of handling simultaneous connections in-house, whereas this isn't necessary with (1).
  • (1) has an extra dependency.
What are your thoughts?

I think bucktooth might be a vaguely comparable system to look at.  (It uses xinetd.)

marnanel: (Default)
With only a very few tweaks, Yarrow can indeed be used as a simple blogging system.  I may extend this a little further and add comments and OpenID login (don't worry, it won't be as horrible as it sounds).  It will not live at this address in the end.

Go to a random post from the last ten years.

Nargery: The existing tweaks, which will ship turned off in the next version of Yarrow, are:
  • ability to specify that a given server is UTF-8, which we need for posts involving Shavian, Hebrew, etc
  • ability to specify metadata inline in the first entry of an item, which is how tags work; most metadata gets ignored.  (The links on tags don't lead anywhere useful yet, but making them into lists of items would involve a simple secondary client.)  If an item's first entry has metadata, the item is assumed to be formatted in HTML and so is not escaped.  I may make this more elegant later.
marnanel: (Default)
There is an ancient Cambridge bulletin board called GROGGS; it celebrates its 25th anniversary next year.  In 2002 I wrote a web client for it called Yarrow.  There has not been a new release of Yarrow since 2003, but since one bug and one important feature request (of which more below) were recently made, I added some fixes at the weekend; you are welcome to test the public beta before release at the demo site I have made.  (The old version is still running on chiark for now.)  The demo site exists because GROGGS requires registration to read it, which makes it hard to use to demonstrate the system.

Now, I thought some of you might be interested in the important feature request.  You see, Yarrow is a web front end to a protocol called RGTP.  RGTP identifies users by a hex string, like C6F54E23BA, called a shared-secret.  Until this weekend, Yarrow identified users like most websites, by means of a username and password.  So if you wanted to start using GROGGS, you had to set up a Yarrow account, log in, request a GROGGS account, and configure your Yarrow account with your GROGGS shared-secret.  This frightened people away.

The change I made this weekend was that there should be no more Yarrow passwords.  Instead, there are only GROGGS shared-secrets.  When you use Yarrow for the first time, you click the link on the login page to request a GROGGS account, and when you're sent your shared-secret, you log into Yarrow using it.  Yarrow creates you a new account and sets the shared-secret of that account to the secret you used to log in.

Someone suggested today that we should also allow passwords.  There would be a "change your password" form somewhere within Yarrow, and once you'd set one, you could log in using either your shared-secret or your Yarrow password.  This would undoubtedly be more convenient, but I'm worried that explaining the idea might add unnecessary complication to the login screen, which is now beautifully simple.

What do you think?  Worth doing eventually?  Worth holding up the release for?

(Also, the existing system saves no state about users who aren't logged in, not even which items they've read.  Perhaps we should create dummy accounts for such users so that it can highlight unread items, as it does for users who are logged in.  This would need rather a rethink of the login system, but is possibly worth the trouble for people trying out the system.  On the other hand, the great majority of the use of Yarrow is by people who are logged in, because it's almost entirely used to read GROGGS, which requires you to be logged in to read it.  What do you think?)


Jul. 30th, 2009 03:32 pm
marnanel: (Default)
Nargery-filled thought experiment: It just occurred to me that with a very little amount of extra hacking I could simply implement my entire blog in Yarrow.  At a pinch, the system as it stands would work out of the box.  I still haven't found a CMS I really like, and Yarrow+RGTP is basically a very simple CMS, and I know it like the back of my hand.  Other things which would be useful if I did this:
  • possibly an option in the server so that only Editors (i.e. me) could create new items
  • RSS/Atom output from the web client; this would be fairly trivial to implement.  (Or do this externally to the web client, and only list items created by me; that way we wouldn't need to restrict the creation of new items.)
  • probably a magic option in the web client so that on that specific site the content of new items (but not the replies to them) was rendered as HTML rather than plain text; this would mean existing content could be kept as-is
  • skinning support, but if we switch to genshi as planned, this will be easy anyway
  • automatic crossposting to Dreamwidth (and thence to LJ) and GNOME Blogs; not sure at what level this would be best implemented; maybe on the server side as some kind of hook
  • I wonder whether we could have automatically-created accounts using OpenId somehow
It would actually be pretty nifty, all told.
marnanel: (Default)
Yesterday I fixed the indexing problem on (by simply deleting the indexes; they're only a cache).  I'm sorry it was down for a couple of days.

Over lunch today I believe I fixed the encoding issue discussed in Y0342114 (it was a single line of code). A GREED user has confirmed that Yarrow-generated pound signs are displayed correctly. If I don't hear of any problems I'll be making a Yarrow release tonight or tomorrow night; it will be v1.3, I believe. (I'm not certain, because 1.2 was released in 2002 and the records are a little shaky since then.)

Other things which may be on the horizon, if anyone asks for them:
  1. (this one may be fixed now)
  2. the index would probably be more reliable if it was stored in a relational database rather than Python pickles; ISTR this was vetoed a few years ago by some site running Yarrow which didn't want to install a relational database, so fair enough. (Maybe we could use sqlite, though.)
  3. the code would be a lot more readable if it used some kind of templating system, but that would require every site using it to install the same templating system; I like the Python port of Template Toolkit, but it's not packaged in Ubuntu at least, and I imagine there are more Pythony solutions around. Can anyone suggest something?
  4. it might be useful to debianise the package so it was easier to create mirrors; I know a fair amount about building .debs but I'm not familiar with packaging Python programs, so any help with that is welcome.
  5. if an item which was once seen is later reported missing on the server, remove all record of it from the index.
  6. the long-threatened switch to remove the distinction between Yarrow accounts and GROGGS accounts, using your GROGGS shared-secret as a password to log into Yarrow instead of a separate step.  This would remove most of the problem of Yarrow mirrors not sharing user databases.
Thoughts and ideas are welcome.


marnanel: (Default)

October 2017

15 161718192021


RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Oct. 19th, 2017 09:36 pm
Powered by Dreamwidth Studios