Pierre Östlund

As promised yesterday: here it is If you intend to try this extension out, continue read this post as it contains a lot of important information. How the extension works will be explained to the extent needed when testing.

How it’s done (just the very basics)

For starters, this extension works similar to the Firefox extension. The extension itself is not aware of beagle. It does not communicate with the beagle daemon. Instead the extension produces small metafiles in a specially selected directory. This directory will be monitored by beagle and files stored/created in this directory will be parsed and indexed at some point by beagle.

The “destination directory”, to which the extension will write files, is stored inside the beagle Thunderbird index directory and is called ToIndex. Most users will find this directory in ~/.beagle/Indexes/ThunderbirdIndex/ToIndex. When new data is indexed, this is were it will be stored until beagle does its magic.

Keeping track of indexed data

Once data has been indexed it’s marked as indexed. This makes sure we don’t index the same data set twice and also speeds the process a lot. Speeding things up is especially important when using an extension since we can only index data while Thunderbird is running. The marking system used allows the extension to start of where it last left off. Once everything has been indexed a first time, then there’s only immediate updates. Which is nice. Another nice thing is that we can index data when beagle isn’t running.

But how does Thunderbird know that beagle has all of its data? Thunderbird can’t be sure. This can’t be totally ignored however (and it isn’t). I’ve put a small check into the extension that figures out if the ToIndex directory exists. If if does, then the extension will assume that everything is fine and continue with the indexing process normally. But if it doesn’t exist however, then all data is marked as “not indexed” before indexing. The directory will be created as well. This solves two of the bigger issues: when ~/.beagle or ~/.beagle/Indexes/ThunderbirdIndex are removed. Everything will be re-indexed in case you decide to do any of it.

Now it’s time to add a note about the meaning of “indexed”. It is very important to understand what it means in this context. What’s happening when something is indexed in Thunderbird is that these small metafiles are produced (as explained above) and they will be processed by beagle when beagle has time to do so. This could be within a few seconds but also a few minutes or even hours. The normal case will probably be within a few seconds/minutes once the initial indexing phase is over. The initial indexing phase ends once Thunderbird has created metafiles for all your data and beagle has indexed all of them. So, expect that it might take some time before things ends up in beagle before this phase has ended.

Note: There’s no Thunderbird backend in beagle yet, so no data will end up in beagle as of this moment. Above is just theory. Creating this backend is the next step and I will begin working on this within the next couple of days.

Using the extension

The extension adds a couple of things to the table once installed. It is automatically enabled and will begin to index your data and in most cases you don’t have to do anything. But there’s a few features built-in worth knowing about. Mainly privacy features.

In the right bottom corner you’ll see the famous beagle dog (if the installation was successful). It will indicate if the extension is enabled or disabled and you’ll clearly see what state you are currently in. Just click this icon if you want to enable or disable indexing.

You can right-click any folder and select Never index this folder if you want that folder never to be indexed. No data is removed from beagle when you do this, so you’ll manually have to remove anything already indexed. Just right-click the same folder and select Remove folder from index to do so. Be sure to answer No in the dialog window that pops up, as if you answer Yes the Never index this folder flag will be removed as well. This applies in general when removing content from the index. Options similar to these will be available for individual emails and others once I figure out how to do this (I’m having some problems with this overlay in particular).

Lastly, there’s a small settings dialog that you can use to change various settings. Just go to Tools->Beagle indexing settings to show it. You have three pages: Indexing, privacy and status. Here’s a small explanation of each page:

• Indexing - You can use this page to enable or disable the indexing process. Just check or uncheck the check box. But more importantly: you can change the indexing speed from here. Just change to whatever suits your need. Beta testers should play around here a bit, more information at the end of the post.
• Privacy - In case you want to disable an entire source, i.e. you don’t want to index any POP3 emails, you can do that from here. But you’ll also find some potential rescue options here too. In case you want to remove everything from beagle’s index, just press the Drop everything button. Note that the backend will immediately begin the indexing process after you’ve done this, so make sure you disable the indexing process before pressing the button if you don’t want anything to end up in beagle again. The Reset index status is quite useful if you want re-index everything without dropping things from beagle’s index.
• Status - This page will display some information about the indexing process. Amount of items added and/or removed from beagle’s index will appear here as well as how many things that are currently queued up. You’ll also see if the extension is idle or if there’s more things to index. Great way to see if the initial indexing process has completed (Indexing status should say Idle).

Known issues

There are currently a few known issues that you guys don’t have to report as bugs:

• The main loop is currently running at all time looking for data to index. This isn’t expensive in any way but it will make Thunderbird wake up a lot and adds to power consumption (laptop users)
• The pages in the preference dialog might appear mixed up. I don’t know why this is happening because it shouldn’t. They all show up correctly in CVS version, so I’ll just hope everything works correct in the next Thunderbird version.
• There’s currently no way of excluding individual items from the indexing process, like with the folders. This is of course planned but I just can’t get the menu items to show up.
• When removing or unindexing content, a small window will pop-up and show the progress (since it can take a couple of seconds with a lot of content). Unfortunately this window isn’t threaded so it will only show up after everything is done (you might see a small window flash by right after installing the extension, that’s this window) and won’t show any progress.
• Thunderbird currently lacks implementations for notifications about when folders are renamed and messages are removed from the Trash-folder. I cannot provide these feature as of today but David Bienvenu over at the mozilla project is looking into this and he’ll implement this some time (don’t know when, maybe it’s already implemented?).
• No about box…

My intention is of course to fix all these issues but I will give them lower priority until the end of summer. There are more important things to deal with right now (like creating the backend so that data ends up in beagle at all).
Notes to testers (important)

In order to get this extension work, you’ll have to make sure that the ~/.beagle/Indexes/ThunderbirdIndex directory exists. It won’t start if it doesn’t. Just either create it with your favourite file manager or by typing the following command from a terminal (this directory will be created by beagle in the end, so it’s just the temporary solution):

mkdir -p ~/.beagle/Indexes/ThunderbirdIndex

The Error console is your friend and it’s the first place you should check out in case you are suspecting something is wrong. You’ll find it in the Tools menu. You can also enable the dump function which will print some things to the terminal. The easiest way to do that is to just open the Error console and paste the following line into the text box and pressing enter (this is one line):

Components.classes [’@mozilla.org/preferences;1′].getService (Components.interfaces.nsIPref).SetBoolPref (’browser.dom.window.dump.enabled’, true);

Note that you won’t get any notification about success here. When you want to disable this, do the same thing but change true at the end of the line to false. In order to see the messages you must run Thunderbird from a terminal. Just open a terminal and run thunderbird or mozilla-thunderbird (which it is depends on distribution). If you have downloaded Thunderbird from mozilla.org and run it from a standalone directory, just cd into that directory and type ./thunderbird to get yourself going.

A request from me to all testers is that I would like it if you tried out various indexing speed settings (found in the preference dialog). The values I’ve used are totally arbitrary and not good at all. I can for instance run the unrecommended setting Instant with no apparent CPU usage at all but it’s not a very fast setting (not as fast as the title claims to be at least). You can mixture with the settings by selecting Custom. These are the settings you’ll see when doing so:

• Batch count - The amount of objects to process each time the main loop is working. You can think of it as: “every Batch delay process Batch count items”.
• Queue count - Amount of items that needs to be in the queue before it automatically empties itself.
• Batch delay - Described above. Measured in seconds.

Try out various settings and see what happens. You can use the Reset index status button in the Indexing page when you want to restart the entire indexing process (when trying settings out). The Status page will tell you when the everything is done.

Be sure to also check out the metafiles created to make sure they contain what they should (they are stored in the ToIndex directory mentioned earlier). An ordinary object looks something like this:

<MailMessage>
<Author>Some author (some.author@some.domain.com)</Author>
<Charset>ISO-8859-1</Charset>
<Date>1165271435</Date>
<Folder>Inbox</Folder>
<FolderURL>imap://user@server/INBOX</FolderURL>
<HasOffline>false</HasOffline>
<MessageId>some-id@some.domain.com</MessageId>
<MessageSize>4085</MessageSize>
<OfflineSize>0</OfflineSize>
<Recipients>Some user (some.user@some.domain.com)</Recipients>
<Subject>Some subject</Subject>
<Uri>imap-message://user@server/INBOX#1</Uri>
</MailMessage>

RSS feed entries looks exactly the same, but FeedItem is used instead of MailMessage to tell them apart. When removing a folder, it may look something like this:

<DeleteFolder>
<FolderURL>imap://user@server/INBOX</FolderURL>
</DeleteFolder>

Deleting messages looks similar too, but DeleteHdr is used instead of DeleteFolder and Uri is used instead of FolderURL. I can add that when removing everything in a folder, only one file (like the one above) will be created instead of one file for each object in the folder since this is much more efficient. You can manually remove the ToIndex directory to force a re-index and you can also remove all files inside the ToIndex directory if you want to (nothing bad will happen if you do at this time, you should not do this when the backend is implemented however). Might be a good idea to clean out every once in a while when trying out smaller things, like moving message/folders around or removing something just to see if the correct files are generated.
Download

The XPI that you need in order to try this extension out is available below. The source code is available in beagle’s SVN tree here. The minimum required version of Thunderbird is set to 2.0, so you’ll need that.

Thunderbird extension v0.1

A few words at the end

I’m leaving for a couple of days tomorrow morning and I will have limited Internet access these days. Feel free to comment this post with bugs, requests or whatever you feel like it’s worth for me and others to know about the extension. I’ll check in when I get the chance to do so.

Good luck and happy testing

Update

Here’s the updated version of the extension. Hopefully it will fix some of the bugs you’ve got so far:

Thunderbird extension v0.1 update

Trackback This Post | Subscribe to the comments through RSS Feed