Introduction to using Django with NAV

Since the release of 3.4.0 parts of NAV have been using the Python web framework Django, starting with the new IP Device Info app.

This introduction to using Django with NAV assumes that the reader is familiar with Django, i.e. have read the Django tutorial and knows his way around the Django documentation.

As NAV has not used Django from the very beginning, NAV does not strictly follow the Django convention of multiple apps with their own models, views, etc. To plug NAV into the Django framework some glue is needed. This glue and other common Django-related code is located in the nav.django Python package, which is located in subsystem/lib-python/nav/django.

The usual Django settings file is located at nav.django.settings. The end-users do not need to modify this settings file when deploying NAV, as all options are derived from existing NAV configuration in the files nav.conf and db.conf.

Note that there is no INSTALLED_APPS setting. NAV does not use this setting at all, due to a different file organization than what Django expects. Following from this, Django does not have a concept of what is an app in NAV. An app is just defined in the minds of the developers. This also means that NAV to a very little degree may take advantage of the django-admin.py executable, including features like syncdb for creating database tables for new apps.

The root URL configuration for all Django-related applications in NAV is located in nav.django.urls. It is not only a module, but also a package. The __init__.py file in the package imports all submodules it can find in the directory, calls get_urlpatterns() on all the submodules, and combines the results into one urlpatterns list, as all Django URL configurations does.

When creating a new Django app, create a new submodule for your app in the subsystem/your_app/nav/django/urls/ folder, run make install in your app to install the module into the nav.django.urls package, and the NAV installation will find your app. If somebody else installs URLconf glue into nav.django.urls they will not affect the URL configuration for your app, unless they use exactly the same name for the submodule as you did. For details, check out the ipdevinfo subsystem and its Makefile.in.

There is one special submodule named nav.django.urls.urlbuilder. This is simply a Django replacement for the archaic nav.web.urlbuilder. It is used by Django apps to link to non-Django apps, like the report subsystem, through the use of the url template tag or the reverse() function, instead of using the archaic nav.web.urlbuilder. From your code's point of view, the URL configurations in nav.django.urls.urlbuilder is used as any other Django URL configuration in NAV.

Since before Django, NAV has been using the Cheetah template system. To enable the use of Django templates for Django apps, while still integrating with the existing template hierarchy of NAV, some shortcut functions have been created in nav.django.shortcuts. At the time of writing, the shortcuts are render_to_response(), object_list(), and object_detail(). render_to_response() is analogous to the well known django.shortcuts.render_to_response(), and object_list() and object_detail() are analogous to functions in django.views.generic.list_detail.

The difference between the original functions from Django and the ones provided in nav.django.shortcuts is that the NAV versions take an additional first argument, namely cheetah_template_func. cheetah_template_func is assumed to be a Cheetah template function, which returns a Cheetah template with a content_string variable. The shortcuts takes the content which Django normally would have returned, and inserts it into the content_string variable of the Cheetah template. In other words, a Django template is rendered as usual, and then the result are wrapped into a Cheetah template.

At the time of writing, nav.django.context_processors only contains one context processor: debug. If Django is run in debug mode, this context processor appends a sql_queries variable to the context of all templates, containing all SQL queries executed to generate the current page. This is useful for optimizing the use of the Django ORM.

The Django convention places models in the apps where they belong the most. In NAV, all models are located centrally in the nav.models package. Usually, one creates the models first and then generates the database from the models using the previously mentioned syncdb feature. Since NAV existed long before Django, our models was created for an existing legacy database through the use of Django's database introspection and numerous hours of hard manual labor to clean up the models, adding __unicode__() methods, etc.

The models was split in as many modules as possible, only limited by interdependencies between the models. This resulted in the following seven modules:

• cabling: models for mapping switch ports to physical rooms.
• event: models related to event and alert queues.
• manage: all models from the old manage database which were not possible to split out to own modules, including the most important models like Netbox, SwPort, and GwPort.
• msgmaint: models related to the message and maintenance subsystems.
• oid: models related to OID mappings.
• rrd: models related to RRD files and data sources.
• service: models related to services.

Of these the manage module is by far the most used and probably of the highest quality. All users of the models should have a look through their definitions to be familiar with how they are mapped to the legacy database using the db_column and db_table arguments, and what methods like get_absolute_url() and __unicode__() are available to the user.

In nav.models.__init__ the DJANGO_SETTINGS environment variable is set to nav.django.settings. As a result of this, one can simply import the models one wants to work with and just use them, without regard to database connections, etc. An example follows:

>>> from nav.models.manage import Netbox
>>> Netbox.objects.filter(sysname__endswith='uninett.no').count()
189L
>>> n = Netbox.objects.filter(sysname__endswith='uninett.no')[0]
>>> n
<Netbox: afs-gw.uninett.no>
>>> n.up_since
datetime.datetime(2008, 5, 30, 15, 36, 50, 350000)
>>> n.organization
<Organization: uninett (UNINETT)>
>>> 

To create a new NAV tool/Django app, the easiest approach is probably to start with copying an existing app, like subsystem/ipdevinfo/. This section explains what the different parts of the ipdevinfo app is, and how it is plugged into the rest of NAV, by following the life cycle of a HTTP request and response.

Using the ipdevinfo app as an example. The nav/ folder contains all runnable Python code, and nothing else. It is installed to the Python path. nav/web/ipdevinfo/ (nav.web.ipdevinfo Python package after installation) contains all the Django code for the app, including views, forms, etc., but not models, as they are centralized in subsystem/lib-python in the nav.models package. The URLconf for the apps URL scope is also located in nav/web/ipdevinfo/. For Django to find it, it must be referenced in the root URLconf as mentioned in the nav.django section above.

In addition nav/web/templates/ contains a Cheetah template (not runnable Python code, but it will be compiled to a runnable Python file during installation) which is used to wrap around the Django templates. The Django templates themselves are not runnable Python code, but HTML files with some additional syntax, and are thus available directly in the templates/ folder, not within the nav/ folder. The Django templates are installed to a templates/ folder next to the python/ folder where all installed Python code goes. Typically this is /usr/local/nav/lib/templates/.

Further, the file ipdevinfo.tool defines the name, description, icon and URL of the IP Device Info tool in NAV's toolbox. The file htaccess is the only file which is installed into the ipdevinfo/ folder in the web server's document root. It simply states that all URLs starting with http://nav.example.com/ipdevinfo/ are to be handled by a Python program, namely Django's mod_python handler, using nav.django.settings as configuration. In other words, the htaccess file should be identical for all Django apps in NAV.

Finally, the folder media/ contains all static media which is needed by the app. Typically, this should be media/style/ for CSS-files, media/js/ for JavaScript, and media/images/ for images. Files should typically be named after the app to avoid name collisions with other apps when installed, i.e. media/style/ipdevinfo.css and subfolders like media/images/ipdevinfo/.

Configuration files should be in the config/ folder in the source tree, and installed into $NAVHOME/etc/appname/.

How to install everything mentioned here is defined in Makefile.in.

1. A user enters the URL http://nav.example.com/ipdevinfo/ in her browser, which then issues a HTTP request for the URL.
2. Apache receives the request, looks up the /ipdevinfo/ subfolder within its document root which contains the mentioned .htaccess file stating that this URL scope is controlled by Django using nav.django.settings as configuration.
3. The request is passed to Django's mod_python handler, which uses the root URLconf nav.django.urls, as configured in nav.django.settings, to identify what Django view should process the request.
4. During lookup in nav.django.url, nav.web.ipdevinfo.urls is identified as responsible for the /ipdevinfo/ URL scope.
5. The matching part of the URL, that is /ipdevinfo/ is stripped and the rest is passed to nav.web.ipdevinfo.urls for further lookup.
6. As the URL now is empty, it matches the ^$ pattern in nav.web.ipdevinfo.urls and the search view, as imported from nav.web.ipdevinfo.views is called.
7. The search view does its job, i.e. either creating a search form, or if a query already has been issued, looking up IP devices matching the query.
8. Finally, the view calls nav.django.shortcuts.render_to_response() with the following arguments: a Cheetah template, a Django template, the form, the query string, the search results and a list of error mesages if any.
9. render_to_response() uses Django's own render_to_response() implementation to render the Django template. Then the result is inserted into the content_string variable of the given Cheetah template. Finally, the result of the Cheetah template is extracted and inserted into a Django HttpResponse, which is returned to the search() view.
10. The search() view returns the HttpResponse object which it gets from render_to_response().
11. The response is passed to the end user.

In addition to the steps mentioned here, Django middleware may plug into the life cycle and do work at inbetween most of these steps, except those happening inside the view function. This is not NAV specific in any way, so the reader is referenced to the Django documentation for futher details.

The integration between NAV and Django is rather new and basic. Given increased usage of Django in NAV one should strive to follow Django's conventions and, where applicable, use more of Django's capabilities instead of NAV's old homebrewn solutions. All NAV developers are encouraged to sketch out ideas for improvements below.

Django 1.0 is targeted for release September 2 2008. To jump from the version NAV are currently using (SVN r7534) to 1.0, some changes have to be made to the Django code in NAV. http://code.djangoproject.com/wiki/BackwardsIncompatibleChanges contains details of the about 30 backwards incompatible changes done to Django between our current version and Django 1.0. IMHO, we should do this jump before the release of NAV 3.5.0 in September 2008.

This was done in LP#260554.

NAV uses sessions for keeping track of logged in users. Currently, we are not using django.contrib.sessions, but are still using a session library developed specifically for NAV. This session library attaches session objects to mod_python's request objects. For now, the easiest way to get hold of the NAV session in views is to use request._req.session, where request._req is the traditional mod_python request object. In the future, one should look into using Django's authentication and session frameworks.

As more parts of NAV are (re)implemented in Django, one should look into replacing the wrapping of Django templates in Cheetah templates with pure Django templates throughout the stack. Even if this may require maintenance of base templates in both Cheetah and Django for quite some time, it would result in much more flexibility with regard to page titles, bread crumb paths, etc. when using Django templates.

Proposal in django_template

NAV should certainly start using unit testing. Tests for nav.web.ipdevinfo could for example go into nav.web.ipdevinfo.tests, but this should be synchronized with Django conventions, and one must also find a way to run the tests, most probably through a custom test runner or something. The point is, one needs to give this some thought, and the sooner the better. New Django apps in NAV should really have unit tests from the start.