January 2006 Archive

Now after two posts about Guido's quest for the Right Web Framework (1st, 2nd) I am starting to feel like his relay station. But I want to share his comment made in Matt's blog post on Python web framework shootout:

Why criticize Django for claiming to be the best? Nobody has denied it, and yet it's criticized as if it were somehow unethical. Frankly, the problem is that there are too many Python web frameworks and wannabees, and if we don't start some kind of shootout, however subjective, we'll never get to market dominance of a few good ones. I'm not saying Django is already the winner -- but we are looking for a winner (or, more likely, a small set of winners) so future developers looking for a Pythonic solution only have to compare a small number of options, all mature, feature-rich, well-supported etc., rather than having to sift through 80+ half-baked solutions.

I think it sums up nicely the current state of affairs, and explains why Ruby on Rails is more popular than any of Python web frameworks..

Finally Guido got some time to play with goods. The verdict is in: Django vs. Cheeta 1-0..

If you liked reading Guido's previous post on Python web frameworks (Rails was mentioned too) and discussion that followed, you should read his second installment: Web Framework Redux. Don't forget to voice your opinion in the forum..

Django-Dojo alliance was finally announced to the world by our very own Jacob Kaplan-Moss:

Starting with version 0.92 (which should be out in a few weeks, Murphy willing), Django is going to bundle Dojo with the toolkit. Specifically as part of Django's admin interface (but available to user apps as well).

Read all about it in Jacob's post. 

I am overjoyed to see such cool high quality open source projects are working together. Clearly it will make it easy to create kick-ass highly interactive web applications in Django and it will advance Dojo positions as a premier AJAX toolkit. It is a rare win-win situation for everybody involved including users of both frameworks.

Some people, who fault Django on building its own components instead of reusing existing ones, will be surprised by this decision. Let me clarify that Django community is not affected by DIY and NIH syndromes. Being a part of it I can say that a lot of options are debated, and the most compelling alternative is used. This is the secret sauce, which makes Django so solid and consistent. Selection of Dojo is just a visible evidence of this approach.

A lot of people are using Dojo in their projects. Django will bring even more users, which will lead to more ideas, more contributed code, and more polished foundation.

Django-Dojo pairing was not unexpected for active community members. Long debates demonstrated that Django community supported this decision even before it was announced. As for me I support it 100% and I will help to make it happen.

Django 0.92 is coming soon. But I already started salivating over upcoming changes to Django Admin application in Django 1.0! When you think it covers 99% of what you need with ease and grace, it is about to be upgraded!.

Finally full Snakes & Rubies video went live on Google Video! And it took only 18 days to verify it (19 days, if you count when I started to upload it). Apparently the whole process of verification depends on file size nonlinearly. It cannot depend on content because it is a combination of smaller files: Adrian's Django presentation, David's Rails presentation, and Q&A session. Oh, well.

And now is time for some stats (1/9/2006–1/27/2006):

As you can see publishing files on Google Video was worth it. It helped ~200 people. Obviously nobody was able to see the full video yet because it just went live. One interesting tidbit: viewers loved to download David's presentation. Downloading Google Video file (.gvp) gives you a small text file, which references the web site and nothing more. You can save URL with the same effect. Of course some smart alecks can use tools to download actual video (.flv), but why? Much better original is available without restrictions and it is referenced from Google Video pages. Another thing: many people watched presentations but didn't watch the Q&A session, which is equally (if not more) interesting.

BTW, if you like Adrian's speaking skills, you should hear him playing his guitar Django style (of course!): Gypsy jazz version of Super Mario Bros 2 song..

It looks like nbd was able to find the problem, which plagued many people (including me) with QoS. You can find details in this thread on OpenWrt forum. Instructions on how to install QoS package can be found in this FAQ entry. Give it a whirl and don't forget to thank Felix Fietkau (nbd).

QoS was the biggest feature on my "wanted" list. It means now I can produce an OpenWrt GUI (webui) module to deal with it. It coincides well with release of Dojo 0.2.2. But most probably I'll be using a snapshot because guys added so much new stuff to Dojo preparing for Dojo 0.3 (widget release). Given the stability of OpenWrt GUI Homunculus Alpha release, I may upgrade new release to Beta status.

As far as I know nbd plans to add firewall and QoS support to webif as well, so all you I-swear-by-my-text-browser and JavaScript-is-over-my-dead-body guys will be covered as well.

Some time ago Jacob Kaplan-Moss released his documentary about Snakes & Rubies event. It is a must see video for all serious programmers working in different fields because it gives you a rare chance to understand the motives of two successful software projects.

Pretty soon it became obvious that sending links to hefty files or torrents is not the best way to spread the word — people are lazy and distractible. I needed something, which can play almost immediately. What can be better than Google Video (Beta)? Jacob gave me his blessing and I started the process.

It's very easy to open an account. But this is the point where convenience ends.

Google Video Uploader (a local application) is used to upload files. You select files you want to upload and hit "Upload". It is as simple as that. Unfortunately connection is dropped frequently but the program doesn't have a notion of "retry". Instead it shows a dialog box. You have to close it and hit "Upload" again. The good thing it understands incomplete uploads continuing them from the last point.

During these "connection drops" I didn't experience any other connection-related problems, which makes me think that it was dropped somewhere on Google's side. It took 2 days to upload 4 files (~1.5G). Most of the time was spent showing a dialog box about "lost connection". I had to monitor the upload process constantly in order to finish it.

In general I would prefer to see two more options for video upload: by specifying a URL of existing video, and by specifying a torrent. In most cases video files are going to be shared by some other means as well, why bother to upload it twice? Why do I think you will host your video files by yourself instead of relying exclusively on Google Video? Read on.

Now I have all files uploaded but they are not "live" because I have to enter all required information about them. Fair enough. After entering description and certifying that it doesn't have any pornographic or obscene material in it, it doesn't go live either: "Submission complete; your video is currently being verified." (David did use some rather colorful expressions, but it was not the highlight of the video.)

It took 5 days to verify 3 smaller files. The video of full event is still "being verified". I have no idea what they do during verification but in any case Google doesn't have a necessary bandwidth to handle it in timely manner.

Quality-wise I was not impressed either. For example Adrian’s presentation is 146M MP4 file. It has a 320 by 240 video stream with 2 16-bit audio channels at 44.1kHz (regular stereo CD specs). After Google's recompression it became 52M file with 1 16-bit audio channel at 22kHz compressed with MP3. No big deal for this type of video material (talking heads) but music will be distorted. While video resolution is the same, the picture was significantly degraded. In general it was mostly okay (presenters are barely moving, most of the time we see static pictures), but text on slides became unreadable. You have to guess some program excerpts. Adrian's presentation was relatively undamaged but David's presentation, which consisted mostly from screens of Ruby code, was barely understandable.

I saw some evidence that internal pieces of Google Video are not synchronized properly. For example status of videos was changed to "live" before they were actually live; changes to video information take time (15-30 minutes and more) to be propagated to live servers.

To sum it up: Google Video service lives up to its "beta" status. It is far from mature and has some quality problems, which may make it unsuitable for some types of video material.

Brad Neuberg wrote a good article, which compares two different approaches to AJAX: thick client (e.g., Dojo style) and thin client (e.g., Prototype style). While it does a good job contrasting two approaches, I want to underscore that the underlying problem is a clash of two cultures between "local application" developers, and "web site" developers.

There is no doubt that local applications create the most satisfying end-user experience. Their typical weakness is in restriction of underlying data to local installation, which makes any collaboration impossible. "Connected applications" can help to alleviate this problem, but networking is hard in general and many local app programmers try to avoid it. They don't have proper culture to do it, existing network APIs are hard to combine with GUI, and so on. In general they don't get it.

On the other side of the spectrum we have web site developers. They tend to use AJAX as a crutch for improving user interface, or providing some whiz-bang eye candy. Doing so they try to be footed on familiar territory of web servers. They don't have proper culture to deal with the client side, JavaScript is notoriously hard to deal with, browser environments impose some funky (== unfamiliar) style of programming, and different browsers support JavaScript and related infrastructure (e.g., DOM, events) differently. In general they don't get it too. That's why they are desperately clinging to "it should degrade seamlessly" requirement. In majority of cases there is no rational foundation for such requirement, we are talking about pure psychology here.

Another gap is due to historical reasons. Ideas (just like applications) tend to be developed incrementally. While there is nothing wrong with that, this is the reason why we see baby steps in bridging the gap from both sides. One camp adds ability to update local programs using Internet, or even access some databases from applications claiming "new revolutionary approach". Another camp adds AJAX to web platforms as an afterthought trying to keep all logic on servers like in good old days.

In public perception there is a factor of coolness, which presently tips the balance against client-side applications. Everything not related to the web is deemed archaic. In reality there are a lot of applications, which cannot be web-based. While it is technically possible, it doesn't make any sense to do so. If your application does require a lot of data (image processing, video processing, data crunching, and so on) or relies on near real-time response (device control, some video games, and so on), it is not practical to access required data using Internet.

In order to understand what's going on let's take a look at the big picture:

• The combined processing power of client computers exceeds the computing power of all servers combined on all fronts (CPU, memory, and so on). On global level it makes sense to distribute the load as evenly as possible increasing the global throughput.
• Don't forget "the round trip problem": response delay is ultimately limited by the speed of light, and we have no practical or theoretical means to reduce it even for unlimited bandwidth connections.

Historically all robust systems tends to be self-optimized increasing "useful work" to "required resources" ratio. The first two rules of server load optimization are:

• Conserve bandwidth:
• Do not send unnecessary data.
• Cache is your best friend.
• Conserve CPU load:
• Don't do unnecessary calculations.
• Don't repeat expensive calculations, if possible. (Cache is your best friend).
• Off-load everything possible to clients. Let somebody else's computer do the work.

How can client side help to optimize a system? Easy:

• Conserve bandwidth:
• Do not ask for unnecessary data.
• Reduce bandwidth with available means (data caching, data compression, partial data reconstruction, and so on).
• Conserve CPU load:
• Don't do unnecessary calculations. In some cases it may be cheaper to ask server to provide precalculated results.
• Don't repeat expensive calculations, if possible. Other clients or servers may have done it for you already.
• Take as much processing on your side as practically possible. A server is a shared component of global infrastructure, which can be overloaded by other clients, while your client is pretty much dedicated to the task most of the time.

My point is the trends of development can be easily predicted from global trends. And I foresee a merge of existing technologies, which will bring the best of both worlds producing truly distributed applications and services.

It looks like the recent development (some call it Web 2.0) goes in the right direction: web sites provide different client-side helpers ranging from "smart" bookmarklets for web browsers (e.g., Reddit) to browser plugins (e.g., Del.icio.us) to full-blown client-side helper applications: uploaders (e.g., Flickr Uploadr), notifiers (e.g., GMail Notifier), updaters (e.g., Google Updater), and so on. Connected applications are there as well (e.g., NASA World Wind). AJAX has its place too — just take a look at all high-flying companies or new startups.

In my opinion it is not enough. We are yet to see new applications, which will leverage all existing possibilities of Internet. How long will it take? Not until developers will soak in new trends and understand how to design applications properly. New tools will emerge as well. Do we have to wait for Web 3.0 for that? I hope not.

Nowadays this question is asked frequently. A lot of guys in their 30s realize that they are the oldest guys in their groups. 20+ guys don't see wise sages around. What is going on? It was debated on /. without any productive outcome (as usual).

Let's take a look at the problem using available statistics. One nice source of data is the National Center for Educational Statistics. I made a chart using Table 280. "Earned degrees in computer and information sciences conferred by degree-granting institutions, by level of degree and sex of student: 1970-71 to 2002-03":

Click on the thumbnail to open big picture in a separate tab and continue reading.