Pierre Östlund

I’m a bit late, I know that, but I still need to mention that Soc is pretty much over. Uploading my code to google is what’s remaining. The past weeks have been extremely stressful as the new students here at the university have arrived. I’ve been one of a big bunch of people helping out with various things around this and that’s why I’ve been off a lot lately. Well, well…

Back to SoC. It’s over and even though I’ve had a blast working with my project and learned a lot, it feels pretty good to have some free time again. I’ve spent a lot more time on the project than I thought I would in the beginning and it feels good to have some time over to work with something else. Working with beagle is fun and it’s an interesting project and I will of course continue working on it and maintain my code in the future. But I feel that I want to work on something else for a while now. I’m not sure yet what I want to work on, but I want to learn more about socket programming so I’ll try to embed that somehow.

I would also like to say thanks to a lot of people for helping out during my SoC experience. We of course have the beagle mentor bunch: Joe Shaw and my personal mentor Debajyoti Bera. David Bienvenu helped me out a lot with Thunderbird specific questions and deserves to be mentioned. I also want to put a big thank you out to all testers for helping me out and testing my work. You help me find the bugs I can’t find myself. Last but not least: Google for arranging Summer of Code! It’s a great initiative and students, like myself, appreciate it a lot. Hopefully I’ll be able to participate again next year.

My schedule is stacked at the moment but I’m going to try to be more active within the next week and continue bug squashing. There’s a few things I want to get rid of before the 0.3 release. I’m also discussing portability questions with a tracker developer as he would like us to share the extension between tracker and beagle (and maybe other indexing tools as well) to add Thunderbird support to tracker. We’ll have to see how that goes. Anyway, that’s everything for now. Thanks again to everybody that made my SoC experience possible and you’ll probably see more of me in the future

I just committed some code to trunk that removes the need for any external libraries or tools needed when building the Thunderbird extension (other than the ones needed by beagle self of course). So, if you could build beagle before you probably can now as well. Try it out and give me some feedback here

A small disclaimer: The update was pretty extensive and I might have missed to update a variable name, function call or something like that. I’ve tried the extension out a bit and it seems to be solid. But I just wanted to get this out.

The Thunderbird backend and extension have now been merged with trunk and the project is now officially marked as finished (with the exception of bug fixes and future work of course). Now is the time where I need all eager testers to find bugs and issues for me to solve. Make sure to check trunk out if you are interested. I have updated the build system to enable Thunderbird support if the thunderbird development files are available. So you should check if the Thunderbird backend is enabled when building. Otherwise you are probably missing the mentioned thunderbird development files (usually called [mozilla-]thunderbird-dev in most distributions).

I want to point out that the latest version of the extension, 0.1.2, fixes a bunch of issues. Yesterday I realized that I had forgot to check in an update to a locale file. This rendered the new error reporting system that I had implemented before useless. I’ve fixed this now and you should probably try the latest extension out if you had problems with the old version just keeping shutting itself down all the time.

Here’s the latest XPI for all you lazy people out there (nothing will end up in beagle until you build beagle with Thunderbird support enabled though; just installing the extension is pretty useless):

Beagle extension for Thunderbird v0.1.2

In my last status report email I wrote something like this (TODO list for this week):

• Add an about box
• Make the “remove a folder from beagle’s index” work
• Ensure that the main loop only runs when there’s something to index
• Clean up the extension a bit and add better error explanations
• At least try to fix the threading issue in the “unindex” dialog window
• Hack around the “moving” and “removal from trash” issues until we have proper support for this [moving should have been “renaming”]
• Fix bugs

I have completed most of the tasks listed. The major one left is the second dot and I still don’t know why my code does not work. It should in my eyes. Well, well… Bug fixing is the next big thing and that’s basically what I have left before this project ends.

lThere’s some notes that I want to add though:

• Currently there’s no way for me to fix the “renaming” and “removal from trash” issues as there are no APIs available today for dealing with this. The functions needed are there but they are never called (stub functions). My code should work as soon as these functions are implemented. I know for a fact that they will be implemented but I don’t know when they will appear in Thunderbird.
• I have updated beagle-search to work with Thunderbird. Unfortunately Thunderbird lacks a good way of displaying data and I don’t know how to fix this right know. A solution is evidently checked into the 3.x branch but I think that some temporary solution is needed here.
• No address book support so far. I might add support for it later but it will not be during SoC. The address book has an entirely different API and it would require more work than I have time for (school is starting pretty soon).

Seems like opening emails from the command line will need some attention from me right now. So I’ll get started on that and see how it goes. And oh, I know that I said that I wouldn’t publish any more XPIs but I’ll publish the latest one any way on request (note that I’ve bumped the version number).

Beagle extension for Thunderbird v0.1.1

Update: I have now implemented basic command line support in the extension, which basically means that you can open your Thunderbird data from beagle-search. It’s still experimental but hopefully it’ll work for you. Enjoy!

Latest beagle extension for Thunderbird

So, here it is! I just committed the code and you can grab it with:

svn co http://svn.gnome.org/svn/beagle/branches/beagle-tbird-soc07

With this beagle should be able to index your data and put it in its right place. There are some issues still though:

• Removing data in Thunderbird does not remove data from beagle (yet)
• There are some encoding issues that I’m working on (you will most likely notice this)
• beagle-search is not updated with the latest URI handlers, so you won’t be able to open your data (just search it)

Also, you need the mozilla-thunderbird-dev package to build beagle with the Thunderbird backend (the extension needs this). So make sure you install it before compiling. Check out this page for more information about compiling beagle (in case you never have done this before).

I just want to point out that you should not expect too much from this yet as I need a couple of days to clear things up. But I encourage you to try it out and come with input of course

Note: I’m not uploading any more .xpi files as you will find your very own .xpi file in the thunderbird-extension directory once you’ve built beagle.

I’ve been working quite intense on the beagle backend the last two days and I come with good news: I’ve almost finished it As of today it indexes emails and RSS content (the indexed data is of course searchable). Extra data, i.e. mail body or RSS description, is indexed as well so it’s possible to query that data too if it’s available.

There’s still a few things to fix but most things are there. I’ve made some updates to the extension as well. Found a few bugs. Had to add a few new properties to the metafiles for instance. Everything will be released tomorrow as I’m a bit tired right now. But you should be able to search your Thunderbird data as of tomorrow

As promised yesterday: here it is If you intend to try this extension out, continue read this post as it contains a lot of important information. How the extension works will be explained to the extent needed when testing.

How it’s done (just the very basics)

For starters, this extension works similar to the Firefox extension. The extension itself is not aware of beagle. It does not communicate with the beagle daemon. Instead the extension produces small metafiles in a specially selected directory. This directory will be monitored by beagle and files stored/created in this directory will be parsed and indexed at some point by beagle.

The “destination directory”, to which the extension will write files, is stored inside the beagle Thunderbird index directory and is called ToIndex. Most users will find this directory in ~/.beagle/Indexes/ThunderbirdIndex/ToIndex. When new data is indexed, this is were it will be stored until beagle does its magic.

Keeping track of indexed data

Once data has been indexed it’s marked as indexed. This makes sure we don’t index the same data set twice and also speeds the process a lot. Speeding things up is especially important when using an extension since we can only index data while Thunderbird is running. The marking system used allows the extension to start of where it last left off. Once everything has been indexed a first time, then there’s only immediate updates. Which is nice. Another nice thing is that we can index data when beagle isn’t running.

But how does Thunderbird know that beagle has all of its data? Thunderbird can’t be sure. This can’t be totally ignored however (and it isn’t). I’ve put a small check into the extension that figures out if the ToIndex directory exists. If if does, then the extension will assume that everything is fine and continue with the indexing process normally. But if it doesn’t exist however, then all data is marked as “not indexed” before indexing. The directory will be created as well. This solves two of the bigger issues: when ~/.beagle or ~/.beagle/Indexes/ThunderbirdIndex are removed. Everything will be re-indexed in case you decide to do any of it.

Now it’s time to add a note about the meaning of “indexed”. It is very important to understand what it means in this context. What’s happening when something is indexed in Thunderbird is that these small metafiles are produced (as explained above) and they will be processed by beagle when beagle has time to do so. This could be within a few seconds but also a few minutes or even hours. The normal case will probably be within a few seconds/minutes once the initial indexing phase is over. The initial indexing phase ends once Thunderbird has created metafiles for all your data and beagle has indexed all of them. So, expect that it might take some time before things ends up in beagle before this phase has ended.

Note: There’s no Thunderbird backend in beagle yet, so no data will end up in beagle as of this moment. Above is just theory. Creating this backend is the next step and I will begin working on this within the next couple of days.

Using the extension

The extension adds a couple of things to the table once installed. It is automatically enabled and will begin to index your data and in most cases you don’t have to do anything. But there’s a few features built-in worth knowing about. Mainly privacy features.

In the right bottom corner you’ll see the famous beagle dog (if the installation was successful). It will indicate if the extension is enabled or disabled and you’ll clearly see what state you are currently in. Just click this icon if you want to enable or disable indexing.

You can right-click any folder and select Never index this folder if you want that folder never to be indexed. No data is removed from beagle when you do this, so you’ll manually have to remove anything already indexed. Just right-click the same folder and select Remove folder from index to do so. Be sure to answer No in the dialog window that pops up, as if you answer Yes the Never index this folder flag will be removed as well. This applies in general when removing content from the index. Options similar to these will be available for individual emails and others once I figure out how to do this (I’m having some problems with this overlay in particular).

Lastly, there’s a small settings dialog that you can use to change various settings. Just go to Tools->Beagle indexing settings to show it. You have three pages: Indexing, privacy and status. Here’s a small explanation of each page:

• Indexing - You can use this page to enable or disable the indexing process. Just check or uncheck the check box. But more importantly: you can change the indexing speed from here. Just change to whatever suits your need. Beta testers should play around here a bit, more information at the end of the post.
• Privacy - In case you want to disable an entire source, i.e. you don’t want to index any POP3 emails, you can do that from here. But you’ll also find some potential rescue options here too. In case you want to remove everything from beagle’s index, just press the Drop everything button. Note that the backend will immediately begin the indexing process after you’ve done this, so make sure you disable the indexing process before pressing the button if you don’t want anything to end up in beagle again. The Reset index status is quite useful if you want re-index everything without dropping things from beagle’s index.
• Status - This page will display some information about the indexing process. Amount of items added and/or removed from beagle’s index will appear here as well as how many things that are currently queued up. You’ll also see if the extension is idle or if there’s more things to index. Great way to see if the initial indexing process has completed (Indexing status should say Idle).

Known issues

There are currently a few known issues that you guys don’t have to report as bugs:

• The main loop is currently running at all time looking for data to index. This isn’t expensive in any way but it will make Thunderbird wake up a lot and adds to power consumption (laptop users)
• The pages in the preference dialog might appear mixed up. I don’t know why this is happening because it shouldn’t. They all show up correctly in CVS version, so I’ll just hope everything works correct in the next Thunderbird version.
• There’s currently no way of excluding individual items from the indexing process, like with the folders. This is of course planned but I just can’t get the menu items to show up.
• When removing or unindexing content, a small window will pop-up and show the progress (since it can take a couple of seconds with a lot of content). Unfortunately this window isn’t threaded so it will only show up after everything is done (you might see a small window flash by right after installing the extension, that’s this window) and won’t show any progress.
• Thunderbird currently lacks implementations for notifications about when folders are renamed and messages are removed from the Trash-folder. I cannot provide these feature as of today but David Bienvenu over at the mozilla project is looking into this and he’ll implement this some time (don’t know when, maybe it’s already implemented?).
• No about box…

My intention is of course to fix all these issues but I will give them lower priority until the end of summer. There are more important things to deal with right now (like creating the backend so that data ends up in beagle at all).
Notes to testers (important)

In order to get this extension work, you’ll have to make sure that the ~/.beagle/Indexes/ThunderbirdIndex directory exists. It won’t start if it doesn’t. Just either create it with your favourite file manager or by typing the following command from a terminal (this directory will be created by beagle in the end, so it’s just the temporary solution):

mkdir -p ~/.beagle/Indexes/ThunderbirdIndex

The Error console is your friend and it’s the first place you should check out in case you are suspecting something is wrong. You’ll find it in the Tools menu. You can also enable the dump function which will print some things to the terminal. The easiest way to do that is to just open the Error console and paste the following line into the text box and pressing enter (this is one line):

Components.classes [’@mozilla.org/preferences;1′].getService (Components.interfaces.nsIPref).SetBoolPref (’browser.dom.window.dump.enabled’, true);

Note that you won’t get any notification about success here. When you want to disable this, do the same thing but change true at the end of the line to false. In order to see the messages you must run Thunderbird from a terminal. Just open a terminal and run thunderbird or mozilla-thunderbird (which it is depends on distribution). If you have downloaded Thunderbird from mozilla.org and run it from a standalone directory, just cd into that directory and type ./thunderbird to get yourself going.

A request from me to all testers is that I would like it if you tried out various indexing speed settings (found in the preference dialog). The values I’ve used are totally arbitrary and not good at all. I can for instance run the unrecommended setting Instant with no apparent CPU usage at all but it’s not a very fast setting (not as fast as the title claims to be at least). You can mixture with the settings by selecting Custom. These are the settings you’ll see when doing so:

• Batch count - The amount of objects to process each time the main loop is working. You can think of it as: “every Batch delay process Batch count items”.
• Queue count - Amount of items that needs to be in the queue before it automatically empties itself.
• Batch delay - Described above. Measured in seconds.

Try out various settings and see what happens. You can use the Reset index status button in the Indexing page when you want to restart the entire indexing process (when trying settings out). The Status page will tell you when the everything is done.

Be sure to also check out the metafiles created to make sure they contain what they should (they are stored in the ToIndex directory mentioned earlier). An ordinary object looks something like this:

<MailMessage>
<Author>Some author (some.author@some.domain.com)</Author>
<Charset>ISO-8859-1</Charset>
<Date>1165271435</Date>
<Folder>Inbox</Folder>
<FolderURL>imap://user@server/INBOX</FolderURL>
<HasOffline>false</HasOffline>
<MessageId>some-id@some.domain.com</MessageId>
<MessageSize>4085</MessageSize>
<OfflineSize>0</OfflineSize>
<Recipients>Some user (some.user@some.domain.com)</Recipients>
<Subject>Some subject</Subject>
<Uri>imap-message://user@server/INBOX#1</Uri>
</MailMessage>

RSS feed entries looks exactly the same, but FeedItem is used instead of MailMessage to tell them apart. When removing a folder, it may look something like this:

<DeleteFolder>
<FolderURL>imap://user@server/INBOX</FolderURL>
</DeleteFolder>

Deleting messages looks similar too, but DeleteHdr is used instead of DeleteFolder and Uri is used instead of FolderURL. I can add that when removing everything in a folder, only one file (like the one above) will be created instead of one file for each object in the folder since this is much more efficient. You can manually remove the ToIndex directory to force a re-index and you can also remove all files inside the ToIndex directory if you want to (nothing bad will happen if you do at this time, you should not do this when the backend is implemented however). Might be a good idea to clean out every once in a while when trying out smaller things, like moving message/folders around or removing something just to see if the correct files are generated.
Download

The XPI that you need in order to try this extension out is available below. The source code is available in beagle’s SVN tree here. The minimum required version of Thunderbird is set to 2.0, so you’ll need that.

Thunderbird extension v0.1

A few words at the end

I’m leaving for a couple of days tomorrow morning and I will have limited Internet access these days. Feel free to comment this post with bugs, requests or whatever you feel like it’s worth for me and others to know about the extension. I’ll check in when I get the chance to do so.

Good luck and happy testing

Update

Here’s the updated version of the extension. Hopefully it will fix some of the bugs you’ve got so far:

Thunderbird extension v0.1 update

The extension has finally reached a usable state after a couple of hectic days. I still have a few things on my TODO-list but they are only minor issues compared to what I’ve been dealing with. Despite that I still want to hold on the extension for one more day (I’ll release it tomorrow no matter what) ’cause I want to figure out a small overlay issue that I’m having. I also need some time to write a small “usage” documentation explaining everything you might need to know.

So, if you are up for it: check in tomorrow, I’ll release the extension to the public. Everyone that wants to test should not miss this

I’m usually quite time optimistic and it seems like I’ve done it again… My “todo” list currently have seven tasks that I must complete before even thinking of making a release. These tasks have popped up today as I’ve been working (which I’ve done pretty much since I woke up this morning) and I didn’t realize how “much” there’s left to do. I really want to make a release ASAP but I don’t want to release anything until all vital parts are there. The GUI code and some of its “hard-to-find”-features are the main reason for the delay. It took me a while to find some of the functions that I really need. But I’ll continue working all day tomorrow and make another post about my progress then. Time to get some rest.

Just wanted to give a tiny report on what’s currently going on. The “core”, if you will, of the extension is nearly complete. I’ve come across a small bug that makes Thunderbird segfault every now and then (not very often though). I’m building a debug build of Thunderbird as I’m writing this to track it down (Thunderbird should never crash). This is also a request from David as he wants to help track the bug down. I’m also going to try out some code that I need to complete my extension which he is the author of (code that is not yet in trunk). Without this code I won’t be able to act upon renaming of folders or when something is thrown out of the trash bin. Once this is done I’ll start with the GUI part, which hopefully won’t take more than a day.

I’ll throw up an .xpi for easy testing as soon as I’ve completed the GUI part, since that’s when it’s appropriate to begin testing it. Performing a full index is totally possible at this stage too, the data wouldn’t end up in beagle until I’ve finished the backend, though.