User Tools

Site Tools


devel:hacking

This is an old revision of the document!


Hacker's guide to NAV

If you are contributing code to Network Administration Visualized, please read this first.

Contributing to NAV

Originally, NAV was a closed source project, initiated by the Norwegian University of Science and Technology (NTNU), and eventually sponsored by UNINETT on behalf of the Norwegian higher education community. In 2004, however, NTNU and UNINETT started distributing NAV under the GNU General Public License, making it a truly free software system.

While UNINETT and NTNU are still the main contributors to NAV, developing NAV to support the needs of the Norwegian higher education community, contributions from third parties is highly appreciated.

We communicate mainly through mailing lists, Launchpad and the #nav IRC channel on FreeNode. At times, UNINETT also arranges workshops and gatherings for its customers: Norwegian universities, university colleges and research institutions.

To contribute:

Go to http://nav.uninett.no/ and

  • Join the mailing lists. The nav-dev mailing list in

particular is for discussing NAV development. So far, this is a

  low traffic list. We can only hope this will change ;-)
* Get a copy of the latest development sources by cloning the
  Mercurial repository at http://nav.uninett.no/hg/default/.
  Most new development takes place on this branch.
* Take a look at the [[:navprojects|project reports from previous
  development projects at NTNU]] (NAVMe, NAVMore, tigaNAV and
  others) - design specifications and other useful bits of historic
  NAV information is mostly to be found in these.  Unfortunately,
  some of the oldest project documentation is in Norwegian only.  Do
  not hesitate to ask for help on the mailing lists.

If you wish to contribute code to the project, see the submitting patches section.

Directory layout

A rough guide to the source tree:

bin/ NAV 'binaries'; executable scripts and programs.
contrib/ User contributed NAV tools. NAV doesn't depend on these, and any maintenance of them is left up to the original developers. We do not offer support for these tools.
doc/ User and developer documentation
etc/ Example/initial configuration files
java/ Java source code
media/ Static media such as CSS stylesheets, images and JavaScript to be served by a webserver
packages/ Stuff to help packaging NAV for various platforms, such as RedHat, CentOS, FreeBSD, Debian and soforth. Much of this is outdated today.
python/ Python source code
sql/ SQL schema definitions and installation/sync tools
templates/ Django HTML templates
tests/ Automated tests
tools/ Tool scripts for the build and release processes.

Development languages and frameworks

Historically, NAV was written using multiple programming languages (Perl, Java, PHP and Python). While this has had an unfortunate impact on integration and maintenance over the years, we've managed to reduce this to just Python and Java in later years. We have a long-term goal to rewrite the remaining Java backend code to Python.

Currently (as of September 2012), NAV consists mostly of Python code, with one remaining backend systems written in Java (eventEngine).

  • We will only accept new code written in Python (except when it

involves patches to the existing Java code).

  • When you contribute additions to the web interface, use the Django framework.

If you wish to contribute something really useful that doesn't use Python, we may consider including it in the contrib directory.

If YOU are willing to invest in porting some of the existing Java code to Python, then you will be celebrated as a NAV hero!

Coding style

Much of the legacy NAV code was written without using any coding style guidelines. This has resulted in some chaotic combination of styles, which we hope to reduce in the future. For new code, please follow these guidelines:

  • For Java code, please refer to SUN's “Code conventions for the

Java Programming Language”: http://java.sun.com/docs/codeconv/

  • For Python code, please refer to PEP-8, “Style Guide for Python

Code” http://www.python.org/doc/peps/pep-0008/

If you see violations of these guidelines, don't hesitate to fix them. If you fix file-wide indentation problems etc., please submit this as a separate patch to make your other patches look clean and readable.

Python boilerplate headers

We will generally only accept code into NAV if it is licensed under GPL v2, but we may make individual exceptions for code licensed under compatible licenses. Each Python source code file should contain the following boilerplate at the top:

#
# Copyright (C) 2008,2009 Somebody
#
# This file is part of Network Administration Visualized (NAV).
#
# NAV is free software: you can redistribute it and/or modify it under the
# terms of the GNU General Public License version 2 as published by the Free
# Software Foundation.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
# more details.  You should have received a copy of the GNU General Public
# License along with NAV. If not, see <http://www.gnu.org/licenses/>.
#

If a file uses non-ASCII characters, it must be encoded as UTF-8, and an encoding statement should be inserted at the top:

# -*- coding: utf-8 -*-

Javascript

When writing javascript code try to focus on modules not pages. If the code is html-related, it should take selectors or objects as input and concern itself solely about those. This makes for much easier testing and reuse. And of course - write the tests first.

When the module is done you write a controller for the page that plugs the needed plugins to the page elements. This should fail gracefully if the needed elements are not present.

NAVs javascript uses require.js - use this to create modules and specify dependencies.

Pro tip is to create require_config.dev.js in media/js/ and add the following configuration to requirejs:

require.urlArgs = "bust=" +  (new Date()).getTime();

This makes sure your not using cached resources in your browser when developing, which browsers loves to do! See config-urlArgs in requirejs documentation for «details». The require_config.dev.js is added in global HG ignore.

Database

NAV uses PostgreSQL as its database backend. Namespaces (schemas) are employed to logically group tables and relations. NAV versions prior to 3.5 employed separate PostgreSQL databases instead of namespaces.

The namespaces currently in use are:

manage The core knowledge database of NAV, containing all sorts of information about the monitored IP Devices, events, alerts, network topology and machine tracking data.
profiles Contains NAV user accounts and groups, user preferences and alert profiles.
logger Anything related to NAV's syslog parser/browser system.
arnold The port detention system Arnold stores it's data here.
radius Radius accounting logs, updated directly by FreeRadius' PostgreSQL module.

Connecting to the database (Python)

Raw SQL

To obtain a connection to the NAV database, use the API accordingly, e.g.:

import nav.db
# Get a connection to the NAV database
connection = nav.db.getConnection('default')

The above code will open a connection to NAV's database, or, if a previous connection with these parameters is already open, returns the already existing connection from a connection cache.

The default parameter is there for legacy reasons; it specifies the name of a subsystem. The db.conf file allows configuration of separate database users for each subsystem (known as a script in db.conf) of NAV. The default db.conf file specifies a database user for a subsystem called default, and also specifies the same database user for all known subsystem names. At present, using a subsystem name that is not configured in db.conf will cause nav.db.getConnection() to revert to using the default name.

Django models

NAV 3.5 and on includes Django models for most database tables. If no SQL magic is needed to perform your database voodoo, it is recommended that you use these models, located in the module nav.models. You do not need to explicitly establish a database connection to use these models, as Django takes care of all that.

The models are defined in modules of the nav.models package.

Changing the schema

The baseline schema is located in sql/baseline - the syncdb.py script is responsible for running this when creating a new database. To make a schema change, you do not change the baseline, but go to the sql/changes directory and create a new schema change script there.

Schema change scripts as numbered, using the following pattern:

  • sc.<major>.<minor>.<point>.sql

The <major> and <minor> numbers usually correspond to the major and minor number of the next NAV release. The <point> number is a sequence id - pick the next free number when creating a schema change script.

Remember these points when creating a schema change script:

  • Create separate change scripts for unrelated schema changes.
  • Remember to write SQL to migrate existing data, if necessary.
  • Do not use transactional statements - the syncdb.py script will take

care of that.

To apply your change scripts, just run syncdb.py. It will look inside the schema_change_log table to see which change scripts have already been applied, and it will detect your new change script and apply this to the database.

:!: When changing the schema, don't forget to update the Django models in the nav.models package. An integration test exists to verify that the Django models can at least be used to run proper SELECTs against the database.

Legacy web code

Legacy web code interfaces directly with mod_python, and uses Cheetah for HTML templating.

All Cheetah templates are located in the python/nav/web/templates directory.

Legacy database connections in web code

Use the nav.db.getConnection() call to open or retrieve an existing database connection. All NAV web modules share the same interpreter and namespace per Apache process, which also means that database connections will be shared between the modules running in each process. Therefore, the following conventions apply for connections obtained from nav.db.getConnection():

  • Do not, under any circumstances, retain references to a database

connection between client requests. Make sure to retrieve a new

  connection at the start of each request cycle - the API will cache
  connections between requests, and will automagically re-open
  broken connections.  As the connection is shared between several
  modules, retained references may be invalid in the next request
  cycle.
* **Do not explicitly close database connections.**  Although the API
  will try to reopen any closed or broken connections, you create
  extra overhead, and you don't play nice with the other web
  modules.
* **The obtained connections will use an isolation level of //read
  committed//**, i.e. no autocommits.  Be careful to commit the
  current transaction if you modify any data.  A mod_python
  ''cleanuphandler'' will try to automatically commit all open
  transactions as the request cycle ends, but this may change
  in the future, so you must not rely on it.
* **Do not change the isolation level of a connection** without
  restoring it to its original value before the end of the request
  cycle.

The "death" of mod_python

mod_python is no longer under active development and has been placed in the Apache foundation's “Attic”. We do not accept new web tools that interface directly with mod_python.

We do, however, aim to refactor existing mod_python-interfacing code into working as Django views. A few tips for such refactorings:

  • Each mod_python handler in NAV mostly performs its own custom

URL parsing and view dispatch. It's best to refactor this into a

  Django URL configuration and separate view functions first.
* Usage of ''mod_python.utils.FieldStorage'' parse URI arguments
  must be refactored to use the ''POST'', ''GET'' or ''REQUEST''
  objects of a Django ''HttpRequest''. It's not that hard, as these
  objects behave like dictionaries, much like the ''FieldStorage''
  class does.
* Conversion from Cheetah to Django templates is not necessary to
  refactor a mod_python handler into a Django view.  It is desirable
  to do so in later refactorings, though.
* NAV's authentication and authorization scheme hooks into Apache's
  request cycle using a ''mod_python'' ''headerparserhandler''. It
  also adds session data as an attribute to the ''mod_python''
  request object.  Once there are no tools left that interface
  directly with ''mod_python'', the auth and session parts of NAV
  must be refactored to work in a pure Django setting before NAV can
  be free of its dependence on ''mod_python''.

Writing new web code

If you are writing a new web application / tool for NAV, please use the Django framework. Here's a quick primer on how Django integrates with legacy NAV.

Version Control

NAV uses Mercurial for distributed version control. Official repositories are located at http://nav.uninett.no/hg/ .

Guide to the repository jungle

The official repositories represent three types of branches

Unstable (default)

New, bleeding edge development occurs on the default branch, which is considered unstable (although we try to always keep it buildable).

Feature branches

New features that take a while (and a lot of changesets) to implement and test will often be published as separate feature branches. For all intents and purposes, the feature branches will look like the default branch with some added feature. They will merge changes from the default branch regularly. Once a feature is considered “ready”, the feature branch will be merged onto the default branch.

Stable (series)

Once we are nearing a new series release of NAV (such as 3.5 or 3.6), a new series branch is created from the default branch. Once this branch is stabilized, the first version is tagged and released. After this point, we accept only bug fixes in this branch. Further point releases in this series are tagged on this branch, and all changes are merged back onto the default branch.

When someone writes a patch for a bug, this should usually be committed to the latest active series branch which is affected by the bug. Once a new series is released, we do not usually maintain the older series branches. We may push bug fixes to these branches, but we are unlikely to create a new point release from it.

Push access

Push access to the official repositories is limited to developers employed or commisioned by UNINETT.

Testing and Continuous Integration

Much of NAV is legacy code, as defined by Michael C. Feathers: Code that has no tests. We have been making an effort to introduce automated tests into the codebase the past couple of years, and hope to improve coverage in time.

There are no tests for the legacy Java code, but many unit tests and integration tests now reside in the tests/ subdirectory.

Running tests

We use py.test to run the test suite. A bundled version is included as runtests.py in the python/ subdirectory, which is used to run the unit tests only when a make check command is issued in the python/ subdirectory.

Some of the test requirements aren't available on the Debian systems we use for development, so we often test inside a Python virtualenv. A suitable virtualenv for testing (on Debian Lenny) can be created thus:

virtualenv .env
. .env/bin/activate
easy_install pip
pip install -r tests/requirements.txt

There's also a script to create a test environment, complete with database initialization. This is used by our CI server. The following will configure and build NAV automatically, and install it into a directory called workspace/build. It will also create a suitable virtualenv in workspace/.env, which you can activate before running tests:

export PGDATABASE=testdb
export PGUSER=testuser
tests/bootstrap-test-environment.sh workspace

Javascript testing

Testing of javascript is in its infancy in NAV. We are currently using buster.js as testing toolkit.

To install buster.js install node and then:

npm install -g buster

As we use require.js you need the AMD module of buster aswell. Install it in the /media/js directory:

npm install buster-amd

To run the tests you need to

  1. Start a buster server by typing buster-server
  2. Capture browsers by pointing browsers to the buster-server (default localhost:1111)
  3. Go to /media/js
  4. Run the tests by typing buster-test

All tests are located under media/js/tests/. Create new tests there. For syntax, assertions and related stuff take a look at the tests already there and the buster docs.

Jenkins

We use Jenkins (formerly Hudson) for Continuous Integration testing of NAV. All the automated tests are run each time new changesets are pushed to the NAV repositories. Jenkins also runs pylint to create stats on code quality.

Our Jenkins installation is available on http://nav.uninett.no/jenkins .

Tips and tricks

Make fixtures for integration testing

from django.core import serializers
from nav.models.manage import Netbox

fixtures = serializers.serialize("xml", Netbox.objects.all()[:2])

Fixtures can so be used in your integration tests by extending the test case DjangoTransactionTestCase in nav.tests.cases

See nav.tests.integration.l2trace_test for an example on applying fixtures for your particular test case.

Also keep in mind you have to make sure you have the model dependency in correct order when importing. Example: Netbox contains a location to a Room where it is located, you have to make sure Room's are imported first before importing Netbox's

See https://docs.djangoproject.com/en/dev/topics/serialization/

TODO: Be able to use django-admin's management command: dumpdata to create fixtures.

Submitting patches

Unless you are submitting one-off fixes for bugs and small issues, please take the time to discuss your change proposals on the nav-dev mailing list. This will increase the chances of having your patches accepted.

Base your patches on the relevant Mercurial branches. If you are submitting a patch for an issue that affects the latest stable series, base your patch on that series branch. If you are submitting patches containing new features, base them on the default branch.

There are three common options for submitting patches:

  • The best way to submit your patches would be using Mercurial. Publish

your own Mercurial branch, and mail its URL to the nav-dev mailing

  list.
* If unable to host a public Mercurial branch, export your changes
  as a Mercurial bundle and attach it to an email addressed to the
  //nav-dev// mailing list.  
* If you have a single patch to submit, attach it to an email
  addressed to the //nav-dev// mailing list.  Please **do not
  patchbomb** the mailing list with multiple emails.
devel/hacking.1352899653.txt.gz · Last modified: 2012/11/14 13:27 by norangshol