Home › Blog › 2005 › December › 24 › Live Filtering

Live Filtering

Update on 11/25/2007: today this article presents mostly historical interest. Since Dojo 0.2 a lot of versions were published and many things were changed. At the time of this writing Dojo is at ripe 1.0. I had to disable all Ajax action in examples because I don't use Dojo 0.2 anymore.

What is Filtering? It is a selection of items using some criteria (filter). In this tutorial I am going to filter documents of my blog (made with Django, of course) matching titles against user-specified substring. Later on I'll talk about generalization of this approach.

Just like a big boy I am going to use Custom Manipulators, which can be avoided, but I want to show how to use it.

We all want to improve end user's experience. To improve usability I'll put a little "live" in it using Ajax (courtesy of Dojo) later on. You can see for yourself how simple it is.

Filtering

Django's ORM provides several ways to do a substring search: contains, icontains, startswith, istartswith, endswith, iendswith. Which one should we use? Let's do all of them. I am going to create a separate function, which will take user's input as parameters and returns a string of rendered items.

def simple_filter(module, field, input, method, template):
    object_list  = module.get_list(**{
        field + '__' + method: input,
        })
    return template_loader.get_template(template).render(Context({
        'object_list':  object_list,
        }))

As you can see the code is extremely simple. This function takes a module (e.g., documents), a field's name (e.g., 'title'), a method (e.g., 'istartswith'), and a template name (e.g., 'blog/documents_filter'). Notes:

• In real life I would put a limit on number of returned items, but I know that my blog contains ~60 documents, so I am not going to do it for the sake of clarity.
• It is possible to add extra parameters like ordering, additional restrictions (e.g., by time frame), and so on.
• It is possible to use different lookup function using parameters.

I'll leave it as an exercise for readers.

Let's take a look at the template I am going to use in this tutorial.

<div class="items">
    {% for object in object_list %}
        <div class="item">
            <div class="title"><a href="{{ object.get_absolute_url }}">{{ object.title|escape }}</a></div>
        </div>
    {% endfor %}
</div>

It is super simple. I added some style names to be able to style this output with CSS, if I need to. Basically it is a list of titles of my documents presented as a list of references to actual documents. Yes, I don't like get_absolute_url(), but decided to use it in this tutorial for simplicity — if you want to adapt my code for your objects, don't forget to define it, or use some other methods.

I think we are ready to write a view. But first we have to define a custom manipulator using a handy cheatsheet.

type_choices = (
    ("startswith", "starts with"),
    ("contains",   "contains"),
    ("endswith",   "ends with"),
    )

class FilterManipulator(formfields.Manipulator):
    def __init__(self):
        self.fields = (
            formfields.TextField(field_name="input"),
            formfields.CheckboxField(field_name="case", checked_by_default=True),
            formfields.SelectField(field_name="type", choices=type_choices),
            formfields.HiddenField(field_name="format"),
        )

I am going to present a form with a text input element (user's substring), a checkbox (case sensitivity), and a select element (type of lookup), and a hidden field (output format). Now we can write a view.

def simple_filtering(request):
    manipulator = FilterManipulator()
    new_data = {'input': '', 'case': True, 'type': 'startswith',}
    if request.GET:
        new_data = request.GET.copy()
        manipulator.do_html2python(new_data)
    form = formfields.FormWrapper(manipulator, new_data, {})
    objects = simple_filter(documents, 'title', form['input'].data,
        (form['case'].data and 'i' or '') + form['type'].data,
        'blog/documents_filter')
    return render_to_response('blog/documents_filtering', {
        'form':    form,
        'objects': objects,
        })

Again manipulator-related code is taken from Custom Manipulators. This view calls our simple_filter() function using user-submitted data constructing method parameter from case and type. The results and the form are passed to a template. Let's take a look at it.

{% extends "blog/base" %}
{% block title %}{{ block.super }} - Search documents{% endblock %}
{% load portal %}
{% load blog %}
{% block menu %}<a href="/">Home</a> &rsaquo; <a href="/blog/">Blog</a> &rsaquo; Filtering{% endblock %}

{% block content %}
<h1>Search document titles</h1>
<form id="form" action="." method="GET">
<label for="id_input">Enter substring:</label> {{ form.input }} 
{{ form.case }} <label for="id_case">case insensitive</label> 
{{ form.type }} <input type="reset" name="reset" /> 
<input type="submit" name="submit" />
</form>
<hr/>
<p><div id="results">
{{ objects }}
</div></p>
{% endblock %}

It uses my predefined template. The meat is in content block. It defines a form, which submits data to its own URL using GET. It includes 3 controls defined in our manipulator. The hidden field is not there because I don't need it. Two more buttons are defined: reset and submit. There is a placeholder for filtered objects. Let's add this view to our URL table. You can see it in action here: /blog/filter/ (will be open in separate tab).

Let's test it now:

• Type in "new" and click submit.
• Switch to "contains" mode and click submit.
• Uncheck "case insensitive" element and click submit.

It works beautifully. Using GET gives us unique URL for each unique combination of parameters. You can even bookmark it (for IE guys: place in favorites) or send it to your friend.

Aren't you tired to hit submit each time? It gets kind of boring, if you want to narrow down your search. We desperately need Live Filtering. Please note: we need it to improve usability, not for splashy effects.

Live Filtering

Let's download Dojo 0.2.1. I used so called "Widgets" build. After deploying it on our server (untar it), we need to include dojo.js file. Dojo will take care about the rest. Let's write our widget in separate file. It will be fun! I don't expect we will write our widgets frequently. Most probably we will use ready-made stuff. But in order to understand how it works, let's write this one.

First of all we need to declare what we provide: LiveFilter widget, of course. More precisely we provide HTML version of it. For simplicity I am going to use existing "dojo" namespace. Obviously Dojo has no idea how to include it automatically. Well, for now I will include this file manually when I need it.

dojo.provide("dojo.widget.LiveFilter")
dojo.provide("dojo.widget.HtmlLiveFilter")

Now let's declare what we are going to use: widget infrastructure (we are going to define a widget) and i/o facilities (we are doing Ajax!).

dojo.require("dojo.widget.Widget");
dojo.require("dojo.io");

We are ready to define a widget.

dojo.widget.HtmlLiveFilter = function() {
    dojo.widget.HtmlWidget.call(this);
    this.widgetType = "LiveFilter";

This code defines a name of our widget, initializes its base class, and sets a widget type. We need to define our public parameters. There are two things I want to control:

• When user types in a substring, it is stupid to do trips to a server after every keystroke. It may overload the server, and it is extremely distracting for users. Typical way to do it is to wait until user stops. What is a good timeout value? Everybody types with different speed. In real life commercial environment it makes sense to measure it using real users and typical situations. The problem is that short timeout wastes resources and may introduce flickering, while long timeout will give a sluggish feel to your web site. In most cases 300ms to 500ms are good values. I am going to use 300ms as my default value.
• Our widget needs to know where to put asynchronously fetched results.
  

this.delay   = 300;
this.results = "results";

Two parameters are defined:

• delay — defines a waiting period in ms (default: 300ms).
• results — defines ID of a container element (like <div>) for results (default: "results").

Now we need to define internal variables to support our business logic. Basically we have several scenarios:

1. User modifies data, no active i/o → set a timer cancelling previous timer, if one exists.
2. User modifies data, there is unfinished i/o → mark that data was modified during current request.
3. Timer completes successfully → submit data, clear internal state variables.

Additionally we need to know where our form is. Do we need to bother with overlapping i/o requests and possible modification of data during the one (rule #2)? We can live with rule #1 and #3. Yes. But let's keep it realistic.

this.formNode = null;
this.timer    = -1;
this.inflight = false;
this.resubmit = false;

Four variables are defined:

• this.formNode — our form element
• this.timer — current timer
  
• this.inflight — true, if there is an active i/o request, false otherwise.
• this.resubmit — true, if during active i/o request user modified data, false otherwise.

Now we are ready to write onchange handler — the heart of our widget.

this.onchange = function() {
    if(!this.inflight) {
        if(this.timer != -1) clearTimeout(this.timer);
        var _this = this;
        this.timer = setTimeout(function() { _this.submit(); }, this.delay);
    } else {
        this.resubmit = true;
    }
}

It implements rules #1 and #2 directly. Successful timer will call submit() method. Let's define it too.

this.submit = function() {
    this.timer    = -1;
    this.inflight = true;
    this.resubmit = false;

    var _this = this;
    dojo.io.bind({
        url:      this.formNode.action,
        method:   this.formNode.method,
        formNode: this.formNode,
        content:  {format: 'ahah'},
        load:     function(type, data)  { _this.onload(data); },
        error:    function(type, error) { _this.onerror(type, error); }
    });
}

Now this is a very interesting method. It sets internal variables to indicate that i/o is active and calls dojo.io.bind(). Let's examine all parameters of the latter.

• url — defines a callback url. I use the one from the form. But how are we going to distinguish between normall call and asynchronous call? Wait, there is more.
• method — defines how to do a callback. I use the one from the form. Of course our view assumes GET, but what if we decided to change it to POST? We don't want to rewrite the widget for such tiny change.
• formNode — the source of data — the form node. All controls of the form are going to be packed and transferred to a server. (Note from my legal department: certain restrictions are applied). It is uber cool!
  
  • We don't need to worry how many fields we have in the form
  • We don't need to worry what their types are.
  • We don't need to worry what their names are.
  • Use any form!
• content — a dictionary, which defines additional content. Aha! This is our hidden field defined in the manipulator! We don't need to put it in the form. We don't need to modify the form at all. We can add it on the fly.
  • Now we can separate regular calls, and asynchronous callbacks.
• load — calls a method, when data is available.
• error — calls a method, when something went wrong.

Let's define onload method.

this.onload = function(data) {
    dojo.byId(this.results).innerHTML = data;
    this.inflight = false;
    if(this.resubmit) {
        this.resubmit = false;
        this.onchange();
    }
}

It puts data (AHAH, or HTML fragment) in the container using dojo.byId() function, indicates that i/o request is finished, and calls onchange method, if we need to resubmit our data.

onerror method is a virtual clone of onload.

this.onerror = function(type, error) {
    alert(String(type) + " " + String(error));
    this.inflight = false;
    this.resubmit = false;
}

I think you had no trouble understanding this one.

Now it is time for a big beast: initialization of the widget. Normally it is small. But I wanted to make it generic, so here it goes:

this.buildRendering = function(args, frag) {

    // retrieve a form node
    this.formNode = frag["dojo:livefilter"]["nodeRef"];
    // sanity check
    if(this.formNode.tagName.toLowerCase() != "form"){
        dojo.raise("Attempted to use a non-form element.");
    }

    // watch controls for change to do live update                
    for(var i = 0; i < this.formNode.elements.length; i++){
        var elm = this.formNode.elements[i];

        // ignore disabled controls
        if(elm.disabled) continue;

        // process select controls
        if(elm.tagName.toLowerCase() == "select") {
            dojo.event.connect(elm, "onchange", this, "onchange");
            continue;
        }

        // process input controls
        if(elm.tagName.toLowerCase() == "input") {
            switch(elm.type.toLowerCase()) {
                case "text": case "password":
                    // watch for "key up" event
                    dojo.event.connect(elm, "onkeyup", this, "onchange");
                    break;
                case "checkbox": case "radio":
                    // watch for changes
                    dojo.event.connect(elm, "onchange", this, "onchange");
                    break;
                case "reset":
                    // watch for resets
                    dojo.event.connect(elm, "onclick", this, "onchange");
                    break;
            }
        }
    }
}

What it does, it retrieves a form node, and subscribes for relevant events of form's controls. It watches for "key up" for text controls, "change" for switchable controls, and "click" for buttons (a reset button). Basically this is the second part, which makes our widget completely independent from controlled form.

The rest is simple: we have to close the definition of our widget, complete inheritance chain, and register it with Dojo.

}

dojo.inherits(dojo.widget.HtmlLiveFilter, dojo.widget.HtmlWidget);
dojo.widget.tags.addParseTreeHandler("dojo:livefilter");

Whew! You can see the result with my comments here: /appmedia/dojo/LiveFilter.js

Now we are going to add one more view.

def simple_live_filtering(request):
    manipulator = FilterManipulator()
    new_data = {'input': '', 'case': True, 'type': 'startswith',}
    if request.GET:
        new_data = request.GET.copy()
        manipulator.do_html2python(new_data)
    form = formfields.FormWrapper(manipulator, new_data, {})
    objects = simple_filter(documents, 'title', form['input'].data,
        (form['case'].data and 'i' or '') + form['type'].data,
        'blog/documents_filter')
    if form['format'].data:
        return HttpResponse(objects)
    return render_to_response('blog/documents_live_filtering', {
        'form':    form,
        'objects': objects,
        })

You can see that it is practically identical to our previous view. I highlighted the difference. It is an if statement, to produce partial output.

Let's modify a template. 

{% extends "blog/base" %}
{% block title %}{{ block.super }} - Search documents{% endblock %}
{% load portal %}
{% load blog %}
{% block menu %}<a href="/">Home</a> &rsaquo; <a href="/blog/">Blog</a> &rsaquo; Filtering{% endblock %}

{% block js %}
{{ block.super }}
<script type="text/javascript" src="/appmedia/dojo/LiveFilter.js"></script>
{% endblock %}

{% block content %}
<h1>Search document titles</h1>
<form id="form" action="." method="GET" class="dojo-LiveFilter">
<label for="id_input">Enter substring:</label> {{ form.input }} 
{{ form.case }} <label for="id_case">case insensitive</label> 
{{ form.type }} <input type="reset" name="reset" /> <input type="submit" name="submit" />
</form>
<hr/>
<p><div id="results">
{{ objects }}
</div></p>
{% endblock %}

Again it is virtually the same! The difference is highlighted. I added a reference to my script (dojo.js is included in the super block) and marked a form as being a Dojo widget. That's it! You can see it in action here: /blog/live/ (will be open in separate tab).

Let's play with it like we did before:

• Type in "new".
• Switch to "contains" mode.
• Uncheck "case insensitive" element.
• Click "reset".
• Search for something else.
• Click "submit" — the form works like it did before!
• Turn off JavaScript (QuickJava Plugin for Firefox is very handy for such testing) and try it again — it downgrades gracefully.

Now go and add it to your web site.

Conclusion

Actually I don't have a conclusion. But I have these random notes:

• Real implementation should handle security:
  • Set a limit to number of items you return from your view.
  • Validate your input data. It is very easy to do with validators.
• I use a standard Dojo "Widgets" profile, which is overkill for this simple code. Custom build will reduce downloadable size significantly. Don't forget this option, if you plan to do something like that on your production site.
  
• Q&A session:
  
  • What if you don't want to monitor all controls? One solution is to add custom (silent) classes on controls and check them during LiveFilter initialization.
  • Can you use it in several forms? Sure, why not.
  • How to specify extra parameters, like "duration"? Just add duration="500" to your form. Purists may use dojo:duration="500", or create a special XML element for LiveFilter. I don't really care for it, so I didn't try it myself.
  • Does it work with XXX browser? I tested it with Konqueror (it has some weird annoying problems with caching), FF1.5 (works), IE6 (works but it may have some slight problem with "onchange" events), Opera9 (works). In all cases fallback mode worked without problems. I don't have Mac, I didn't try Safari.
  • It is not realistic enough! My goal was to show how to do it and what's involved. I cut corners left and right to make my code comprehesible.
  • If you show your results in a pop-up div, you will have an autosuggest widget. Yes, it is going to be a poor man's autosuggest widget. It is better to use existing ComboBox widget. It handles selections correctly. Go to look at it, and you will notice the difference.
  • I don't want to see my results as a stupid list! Nobody forces you to do so. Write another template for simple_filter() and generate a graph, a table, whatever you like.

If you have questions, suggestions, improvements, and so on, you can find me on Django mail lists. Of course, you can always write me a private e-mail. Thanks for reading.

Save/recommend this post:              Subscribe to this blog: