September 2005 Archive

I was able to spend several hours this week to work on upcoming OpenWrt GUI. I use Dojo as a foundation. At this point I coded a skeleton, which works as a proof of concept. As soon as I finish putting in AJAX guts, I'll switch to meaty functionality. Add some nice skin to it and we have our homunculus up and running.

Huh? AJAX? No, it's not Ajax the detergent. No, I didn't mean Ajax the pro soccer club. "Now I clearly remember that AJAX is a Trojan, and I don't like Trojans and viruses" — clever, but no. Go see the movie about Greek heroes and the Wooden Horse. And let's skip "Trojan" discussion altogether — I'm trying to be PG-13 rated.

One of the best explanations of this term is in Wikipedia. In order to use AJAX-based applications you need JavaScript and relatively recent versions of popular browsers. If you never turn on JavaScript out of fear of viruses or you use "real man" browsers like Mosaic 1.0 and such, you will not be able to use my implementation.

Don't fret, OpenWrt will have a plain jane version, which will work even for text-based browsers. It's still in development, but current builds are already useful. Go and try it out.

Once again, let's grab new version of TinyMCE.

I am running new Django cache code now. If you notice any problems with it (e.g., error messages), please drop me a line at once.

My email is at the bottom of the page.

For a long time I wanted to have two TinyMCE toolbars for Django Admin:

• One-line toolbar with the most essential tools, which gives a lot of space for inline editing of text.
• Full-featured toolbar with all tools for full-screen mode. 

It finally happened. Spocke (the main developer of TinyMCE) helped me to figure out how to do it. I updated my article in Django Wiki with new config file, and removed my notes about space problem (new config file solves it) and flickering (new TinyMCE 2.0 RC2 works perfectly).

If you hesitated to install TinyMCE because of said problems, now it's the right time to do it.

In my previous post I complained about annoying flickering. Well, I've upgraded to TinyMCE 2.0 RC2 and it doesn't flicker anymore! Run, don't walk, and upgrade immediatelly!

Finally I bit the bullet and decided to implement OpenWrt web-based user interface using Felix Fietkau's code. Well, MIPS processors used by OpenWrt units are not race horses and GUI is slow to my taste. Yes, I know that stock firmware is not faster. "Slow" is absolute category for me.

How do we combat latencies and slow network speed? Exactly. We can apply the same for OpenWrt, where "slow network speed" is replaced with "slow processor on local network". So I decided to do The Right Thing^(tm) and implement it using AJAX approach. I will use Dojo.

Stay tuned for more updates and upcoming beta.

Update: this is the list of all my post detailing OpenWrt GUI development.

Update: see screencast and screenshots, download and try it.

Hurricane Rita was donwgraded to Category 3 storm (125mph/200kph winds). Here in Dallas we expect heavy rains for 2+ days.

Sunset was very colorful today. I tried to capture it in this picture:

You can find bigger picture on Flickr.

I was asked several times to explain how I did RSS for my site. Django has RSS framework, which is not documented. Most probably I am not the right guy to explain it but I'll try.

There are three ways to implement RSS with Django:

1. The Simple: using Django's RSS framework.
2. The Smart: using django.utils.feedgenerator.
3. The Hard: write a view and output XML manually or using standard xml.sax.saxutils.

If you want Django to do everything for you, then you should use "The Simple Way". If you want some custom object selection, you should use "The Smart Way". For obsessed workaholics, S&M adepts, and guys-with-really-convoluted-needs the only way is "The Hard Way". Being lazy I prefer #1 and #2.

After some requests I decided to publish my code for categories. It's very simple. It was inspired by following articles: A "category" Data Model (note: this article uses old-style model format, it doesn't work anymore) and Relating an object to itself, many-to-one.

Finally we have official Django badges. Wilson Miner did a great job putting them together. I decided to use this one:

How do you like it?

If you are a fan of 80 by 15 pixels badges, don't fret — Dave Hodder has you covered with his rendition of "Made with Django" theme. Collection of his small badges are attached to ticket #509. Check it out too.

Today is the day of new releases: OpenWrt just released new White Russian RC3.

BTW, contrary to popular belief, it's not just for Linksys WRT54G. You can run it too, if you have supported hardware. I run it on Buffalo WBR2-G54.

Wow, they move fast! Go and grab Tiny MCE 2.0RC2.

Finally I added RSS 2.0 using Django's facilities.

Default RSS feed is for my blog. Categories have two versions of RSS feed: exclusively for documents of the category, and documents for the category and its children. (I hope it is not news anymore: I have hierarchical categories.)

Update: InnoDB is in the house!

In my previous post I already complained about lack of transaction support by DreamHost. The only database option is MySQL with InnoDB engine, which doesn't support transactions. Everybody knows that transactions are nice. So what about MySQL/InnoDB or PostgreSQL?

So I hit DreamHost forums and found several posts from DreamHost people about PostgreSQL and InnoDB:

• Re: PostgreSQL rant on 06/02/05 by Jeff @ DreamHost (DH Discussion Forum Admin)
• Re: PostgreSQL rant on 06/01/05 by Dallas (DreamHost Head Honcho/Founder)
• Re: mod_proxy support on 04/07/05 by Jeff @ DreamHost (DH Discussion Forum Admin)
• Re: PostgreSQL on 12/03/04 by will
• Re: PostgreSQL rant on 10/15/04 by nate
• Re: PostgreSQL rant on 06/23/04 by Jeff @ DreamHost (DH Discussion Forum Admin)
• Re: MySQL 4, PostgreSQL on 07/15/03 by will
• Re: MySQL 4, PostgreSQL on 04/07/03 by Jeff @ DreamHost (DH Discussion Forum Admin)
• Re: transactions in mysql on 02/11/03 by Jeff @ DreamHost (DH Discussion Forum Admin)

So I've got the impression that PostgreSQL is out for now, but MySQL/InnoDB is really close to be implemented and offered to customers. I decided to ask the support directly and got two firm answers: MySQL/InnoDB — No, PostgreSQL — No. They don't have any plans to introduce them. Why? No reasons were given. Obviously "it could change" over time. But for the last several years it didn't.

Interesting that last year they "caught" someone who installed PostgreSQL and ran it as a user, but they "wouldn't allow that". Why? I wonder if there is a list of prohibited software. Hmm...

Now this blog runs on 'sql:'. I use MySQL as my backend. Before that I tested it locally with SQLite (so easy to install!) and MySQL.

I learned today that MySQL, which is provided by my host (DreamHost), uses MyISAM for all tables. MyISAM doesn't support transactions. AT home I use InnoDB, which supports transactions. However the hope is not lost — DreamHost mulls over InnoDB support. But as far as I know, they don't have an implementation schedule yet.

I hope somebody will help me to test it using MySQL with InnoDB, and PostgreSQL. As for me I would settle for 'file:' or even 'locmem:'. The latter looks like a reasonable choice given into account small size of my site.

This site was running using 'locmem:' caching option, which is fine for small web site. Now it runs using 'file:'. Please let me know, if you spot any issues.

I am planning to implement 'sql:' next.

TinyMCE 2.0RC1 is out. Take it while it's hot! I've upgraded my admin pages and it looks better than previous one (duh!).

Nevertheless I see annoying flickering in FireFox.  But it flickers less now.

I am not talking about multithreading problems. "locmem:" doesn't have them. But it does have problems with gzip. Sometimes I see binary stuff on my screen. It looks like a gzipped web page without proper gzip encoding flag. I don't do anything related to compression in my cache.

Is it a bug in Django's caching? Hmm... Apparently it works properly with memcache. Can it be some bug in "simple:" implementation, which was a base for "locmem:"? Weird.

Update: Yes, something is wrong. I found one small bug and one weirdness. The bug is easy to fix. But I don't know what to do about the weird part. Apparently if you store HttpResponse object, it behaves like it is modified by some means. But if you pickle it, everything goes smoothly. I'll try to dig up more on that.

Of course, local debugging doesn't help. Does it indicate some other Dreamhost-specific problem?

Being a little bit frustrated by the lack of caching options for my web site I've added "locmem:" cache to Django. It is fully multithreaded local memory-based cache, which uses reader-writer lock to do its magic. It seems to work well but I need some time to test it more.

I think "file:" is going to be next.

It looks like all kinks with Django on Dreamhost were worked out. Of course, I am referring to MySQL problem (see my previous post) and strange stalling of long pages, which I reported to Django's mailing list. It turned out that other people had these problem too. Jason McBrayer shared his experience. He pointed out to possible flup/fcgi.py problem. I contacted Allan Saddi — the author of flup and other Python libraries and utilities. He came to help immediately. We were able to isolate the problematic piece of software and in no time Allan made a fix.

I wish developers of commercial software would serve paying customers as fast as Allan did for free software. Thank you, Allan!

Regarding MySQL problems. I've submitted a patch #463 to Django developers. Obviously it is a critical piece, which requires careful examination and validation. I hope it will be accepted very soon. Or you can try it on your own.

So go now, patch Django's mysql.py, download flup or fcgi.py, and  enjoy smooth ride with excellent light-weight web framework, which is Django.

It looks like the problem is solved. Django creates one (1) connection to database and reuses it to handle requests. Multithreaded FastCGI creates threads to handle requests. These threads make requests to MySQL, which gets confused and loses its cool^H^Honnection. This process is random and compounded with multiple web requests at the same time. For example, if Google Web Accelerator decided to prefetch some pages, your web site would be hosed.

Luckily, the only place, which needs fixing, is one small class: mysql.DatabaseWrapper.

I added a dictionary of connections so every thread will have it's own connection. Apparently it solved the problem. Additionally I ping connection to make sure it is not timed-out, and do some simple clean up, when threads are finished.

PostgreSQL doesn't have this problem. For those of you keeping score at home, MySQL loses a cookie point.

I decided to try how well Opera works with my site. It mostly works. You can go and see stuff but all Dojo-related code doesn't work. Interesting that Google Maps works properly.

When I switched to admin mode to edit this article, it turned out that TinyMCE doesn't work either.

But good news is Dojo and TinyMCE failed gracefully.

I complained about instability of this site in my previous post. Now, after intensive googling, it looks like it may be related to multiple threads or forked processes using the same connection to MySQL. Apparently it can confuse MySQL. Children should create their own connections after spawning.

If Django does reuse connections, it may be possible to fix it. Otherwise, CGI is the way to go. The latter solution would be slow.

I am totally at loss. I am getting weird errors like this:

    OperationalError: (2013, 'Lost connection to MySQL server during query')

They are pretty much random but fairly frequent. Apparently it may be caused by different MySQL timeouts, which I cannot control. It seems that in order to combat those you have to call connection.ping() periodically to reconnect. I added them to Django's MySQL handler and ... now I am getting these errors during execution of connection.ping()! Bizarre!

I cannot believe that I am the only one with this problem. But googling did find anything useful. Yes, I found a bunch of people complaining about it. No, there are no solutions posted. Am I the only one, who uses MySQLdb (indirectly from Django) from FastCGI process?

On the other hand all symptoms listed in the previous paragraph point to alternative conclusion: this problem is so simple and so stupid that people don't bother to explain how to fix it. Hmm, is this really a case?

Update: Apparently this problem is widespread. Just take a look at Ruby On Rails ticket: http://dev.rubyonrails.com/ticket/428. It was opened in January 2005 but they couldn't localize it and fix it. 

Unbelievable! We are talking about FastCGI, which is widely used, and MySQL, which is widely used. Ruby or not, this problem should be solved by now. Are there FastCGI apps, which use MySQL?

Python and Ruby communities were around for many years but this simple problem is not solved yet.

The best thing about Dojo 0.1: it's quite easy to create portable widgets. Such widgets work similar to Microsoft IE's behaviors but they are even more self-contained. Dojo widgets can work with fragments of HTML and CSS, which will be inserted during the object creation. The event system is very elaborate. Of course Dojo is more than a simple widget factory. For example I didn't try the I/O facility yet.

The worst thing about Dojo: it's very rough. Documentation is lacking, examples are missing (or too specific), platform-dependent bugs are roaming the code. In other words it is a typical 0.1 version.

It helps that Dojo is under heavy development. Guys are fixing bugs and adding new features on almost hourly basis. There is a very helpful support group. I got my questions answered in a matter of minutes. I am convinced that Dojo has a great potential.

Get Dojo here: http://dojotoolkit.org/. Gmane has two Dojo-related groups: gmane.comp.web.dojo.devel and gmane.comp.web.dojo.user.

I've made a very simple Yellow Fade widget to highlight recent blog entries.

I've implemented a simple Shadow widget as well and it worked locally but I had some problems with a correct placement of shadows on my web site. Eventually I abandoned it because FireFox produces some visible movements during DOM manipulations even for "visibility: hidden" elements. Is it a FireFox' bug? I have no idea.

New design is here. Well, some elements of it. I had bigger ideas, but CSS didn't work reliably, and JavaScript option was in debugging mode. So I decided to update this site now and add other planned stuff later.

One reason I postponed JavaScript thingy is I want to try Dojo. It looks like a good foundation for widgets. I am going to create a couple of them at least. If it works okay, I'll be using it in other projects as well. Stay tuned.