This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
devel:hacking [2012/06/25 12:00] norangshol Make fixtures for integration testing |
devel:hacking [2014/11/05 10:57] (current) morten |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== Hacker's guide to NAV ====== | ====== Hacker's guide to NAV ====== | ||
- | If you are contributing code to Network Administration Visualized, | + | This document has moved to https://nav.uninett.no/doc/dev/hacking/hacking.html |
- | please read this first. | + | |
- | + | ||
- | + | ||
- | ====== Contributing to NAV ====== | + | |
- | Originally, NAV was a closed source project, initiated by the | + | |
- | Norwegian University of Science and Technology (NTNU), and eventually | + | |
- | sponsored by UNINETT on behalf of the Norwegian higher education | + | |
- | community. In 2004, however, NTNU and UNINETT started distributing | + | |
- | NAV under the GNU General Public License, making it a truly free | + | |
- | software system. | + | |
- | + | ||
- | While UNINETT and NTNU are still the main contributors to NAV, | + | |
- | developing NAV to support the needs of the Norwegian higher education | + | |
- | community, contributions from third parties is highly appreciated. | + | |
- | + | ||
- | We communicate mainly through mailing lists, | + | |
- | [[https://launchpad.net/nav/|Launchpad]] and the ''#nav'' IRC channel | + | |
- | on //FreeNode//. At times, UNINETT also arranges workshops and | + | |
- | gatherings for its customers: Norwegian universities, university | + | |
- | colleges and research institutions. | + | |
- | + | ||
- | To contribute: | + | |
- | + | ||
- | Go to http://nav.uninett.no/ and | + | |
- | + | ||
- | * Join the mailing lists. The //nav-dev// mailing list in | + | |
- | particular is for discussing NAV development. So far, this is a | + | |
- | low traffic list. We can only hope this will change ;-) | + | |
- | * Get a copy of the latest development sources by cloning the | + | |
- | Mercurial repository at http://nav.uninett.no/hg/default/. | + | |
- | Most new development takes place on this branch. | + | |
- | * Take a look at the [[:navprojects|project reports from previous | + | |
- | development projects at NTNU]] (NAVMe, NAVMore, tigaNAV and | + | |
- | others) - design specifications and other useful bits of historic | + | |
- | NAV information is mostly to be found in these. Unfortunately, | + | |
- | some of the oldest project documentation is in Norwegian only. Do | + | |
- | not hesitate to ask for help on the mailing lists. | + | |
- | + | ||
- | If you wish to contribute code to the project, see the [[#submitting patches]] section. | + | |
- | + | ||
- | ====== Directory layout ====== | + | |
- | A rough guide to the source tree: | + | |
- | + | ||
- | | bin/ | NAV 'binaries'; executable scripts and programs. | | + | |
- | | contrib/ | User contributed NAV tools. NAV doesn't depend on these, and any maintenance of them is left up to the original developers. We do not offer support for these tools. | | + | |
- | | doc/ | User and developer documentation | | + | |
- | | etc/ | Example/initial configuration files | | + | |
- | | java/ | Java source code | | + | |
- | | media/ | Static media such as CSS stylesheets, images and JavaScript to be served by a webserver | | + | |
- | | packages/ | Stuff to help packaging NAV for various platforms, such as RedHat, CentOS, FreeBSD, Debian and soforth. Much of this is outdated today. | | + | |
- | | python/ | Python source code | | + | |
- | | sql/ | SQL schema definitions and installation/sync tools | | + | |
- | | templates/ | Django HTML templates | | + | |
- | | tests/ | Automated tests | | + | |
- | | tools/ | Tool scripts for the build and release processes. | | + | |
- | + | ||
- | ====== Development languages and frameworks ====== | + | |
- | + | ||
- | Historically, NAV was written using multiple programming languages | + | |
- | (Perl, Java, PHP and Python). While this has had an unfortunate | + | |
- | impact on integration and maintenance over the years, we've managed to | + | |
- | reduce this to just Python and Java in later years. We have a | + | |
- | long-term goal to rewrite the remaining Java backend code to Python. | + | |
- | + | ||
- | Currently (as of February 2011), NAV consists mostly of Python code, | + | |
- | with a few remaining backend systems written in Java (there's also the | + | |
- | Netmap Java Applet). | + | |
- | + | ||
- | * We will only accept new code written in Python (except when it | + | |
- | involves patches to the existing Java code). | + | |
- | * When you contribute additions to the web interface, use the Django framework. | + | |
- | + | ||
- | If you wish to contribute something really useful that doesn't use | + | |
- | Python, we may consider including it in the //contrib// directory. | + | |
- | + | ||
- | If **YOU** are willing to invest in porting some of the existing Java | + | |
- | code to Python, then you will be celebrated as a NAV hero! | + | |
- | + | ||
- | ====== Coding style ====== | + | |
- | Much of the legacy NAV code was written without using any coding style | + | |
- | guidelines. This has resulted in some chaotic combination of styles, | + | |
- | which we hope to reduce in the future. For new code, please follow | + | |
- | these guidelines: | + | |
- | + | ||
- | * For Java code, please refer to SUN's "Code conventions for the | + | |
- | Java Programming Language": http://java.sun.com/docs/codeconv/ | + | |
- | * For Python code, please refer to PEP-8, "Style Guide for Python | + | |
- | Code" http://www.python.org/doc/peps/pep-0008/ | + | |
- | + | ||
- | If you see violations of these guidelines, don't hesitate to fix them. | + | |
- | If you fix file-wide indentation problems etc., please submit this as | + | |
- | a separate patch to make your other patches look clean and readable. | + | |
- | + | ||
- | ===== Python boilerplate headers ===== | + | |
- | We will generally only accept code into NAV if it is licensed under | + | |
- | GPL v2, but we may make individual exceptions for code licensed under | + | |
- | compatible licenses. Each Python source code file should contain the | + | |
- | following boilerplate at the top: | + | |
- | + | ||
- | <code python> | + | |
- | # | + | |
- | # Copyright (C) 2008,2009 Somebody | + | |
- | # | + | |
- | # This file is part of Network Administration Visualized (NAV). | + | |
- | # | + | |
- | # NAV is free software: you can redistribute it and/or modify it under the | + | |
- | # terms of the GNU General Public License version 2 as published by the Free | + | |
- | # Software Foundation. | + | |
- | # | + | |
- | # This program is distributed in the hope that it will be useful, but WITHOUT | + | |
- | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | + | |
- | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | + | |
- | # more details. You should have received a copy of the GNU General Public | + | |
- | # License along with NAV. If not, see <http://www.gnu.org/licenses/>. | + | |
- | # | + | |
- | </code> | + | |
- | + | ||
- | If a file uses non-ASCII characters, it **must** be encoded as UTF-8, and an | + | |
- | encoding statement should be inserted at the top: | + | |
- | + | ||
- | <code python> | + | |
- | # -*- coding: utf-8 -*- | + | |
- | </code> | + | |
- | + | ||
- | ====== Database ====== | + | |
- | NAV uses PostgreSQL as its database backend. Namespaces (schemas) are | + | |
- | employed to logically group tables and relations. NAV versions prior | + | |
- | to 3.5 employed separate PostgreSQL databases instead of namespaces. | + | |
- | + | ||
- | The namespaces currently in use are: | + | |
- | + | ||
- | | ''manage'' | The core knowledge database of NAV, containing all sorts of information about the monitored IP Devices, events, alerts, network topology and machine tracking data. | | + | |
- | | ''profiles'' | Contains NAV user accounts and groups, user preferences and alert profiles. | | + | |
- | | ''logger'' | Anything related to NAV's syslog parser/browser system. | | + | |
- | | ''arnold'' | The port detention system Arnold stores it's data here. | | + | |
- | | ''radius'' | Radius accounting logs, updated directly by FreeRadius' PostgreSQL module. | | + | |
- | + | ||
- | ===== Connecting to the database (Python) ==== | + | |
- | ==== Raw SQL ==== | + | |
- | + | ||
- | To obtain a connection to the NAV database, use the API accordingly, | + | |
- | e.g.: | + | |
- | + | ||
- | <code python> | + | |
- | import nav.db | + | |
- | # Get a connection to the NAV database | + | |
- | connection = nav.db.getConnection('default') | + | |
- | </code> | + | |
- | + | ||
- | The above code will open a connection to NAV's database, or, if a | + | |
- | previous connection with these parameters is already open, returns the | + | |
- | already existing connection from a connection cache. | + | |
- | + | ||
- | The ''default'' parameter is there for legacy reasons; it specifies | + | |
- | the name of a subsystem. The ''db.conf'' file allows configuration of | + | |
- | separate database users for each subsystem (known as a ''script'' in | + | |
- | ''db.conf'') of NAV. The default ''db.conf'' file specifies a | + | |
- | database user for a subsystem called ''default'', and also specifies | + | |
- | the same database user for all known subsystem names. At present, | + | |
- | using a subsystem name that is not configured in ''db.conf'' will cause ''nav.db.getConnection()'' to revert to using the ''default'' name. | + | |
- | + | ||
- | ==== Django models ==== | + | |
- | NAV 3.5 and on includes Django models for most database tables. If no | + | |
- | SQL magic is needed to perform your database voodoo, it is recommended | + | |
- | that you use these models, located in the module ''nav.models''. You | + | |
- | do not need to explicitly establish a database connection to use these | + | |
- | models, as Django takes care of all that. | + | |
- | + | ||
- | The models are defined in modules of the ''nav.models'' package. | + | |
- | + | ||
- | ===== Changing the schema ==== | + | |
- | + | ||
- | The baseline schema is located in ''sql/baseline'' - the ''syncdb.py'' script | + | |
- | is responsible for running this when creating a new database. To make a schema | + | |
- | change, you **do not** change the baseline, but go to the ''sql/changes'' | + | |
- | directory and create a new schema change script there. | + | |
- | + | ||
- | Schema change scripts as numbered, using the following pattern: | + | |
- | + | ||
- | * ''sc.<major>.<minor>.<point>.sql'' | + | |
- | + | ||
- | The ''<major>'' and ''<minor>'' numbers usually correspond to the major and | + | |
- | minor number of the next NAV release. The ''<point>'' number is a sequence id | + | |
- | - pick the next free number when creating a schema change script. | + | |
- | + | ||
- | Remember these points when creating a schema change script: | + | |
- | + | ||
- | * Create separate change scripts for unrelated schema changes. | + | |
- | * Remember to write SQL to //migrate// existing data, if necessary. | + | |
- | * Do not use transactional statements - the ''syncdb.py'' script will take | + | |
- | care of that. | + | |
- | + | ||
- | To apply your change scripts, just run ''syncdb.py''. It will look inside the | + | |
- | ''schema_change_log'' table to see which change scripts have already been | + | |
- | applied, and it will detect your new change script and apply this to the | + | |
- | database. | + | |
- | + | ||
- | :!: When changing the schema, don't forget to update the Django models in the | + | |
- | ''nav.models'' package. An integration test exists to verify that the Django | + | |
- | models can at least be used to run proper SELECTs against the database. | + | |
- | + | ||
- | ====== Legacy web code ====== | + | |
- | Legacy web code interfaces directly with | + | |
- | [[http://www.modpython.org/|mod_python]], and uses | + | |
- | [[http://www.cheetahtemplate.org/|Cheetah for HTML templating]]. | + | |
- | + | ||
- | All Cheetah templates are located in the ''python/nav/web/templates'' | + | |
- | directory. | + | |
- | + | ||
- | ===== Legacy database connections in web code ===== | + | |
- | Use the ''nav.db.getConnection()'' call to open or retrieve an | + | |
- | existing database connection. All NAV web modules share the same | + | |
- | interpreter and namespace per Apache process, which also means that | + | |
- | database connections will be shared between the modules running in | + | |
- | each process. Therefore, the following conventions apply for | + | |
- | connections obtained from ''nav.db.getConnection()'': | + | |
- | + | ||
- | * **Do not, under any circumstances**, retain references to a database | + | |
- | connection between client requests. Make sure to retrieve a new | + | |
- | connection at the start of each request cycle - the API will cache | + | |
- | connections between requests, and will automagically re-open | + | |
- | broken connections. As the connection is shared between several | + | |
- | modules, retained references may be invalid in the next request | + | |
- | cycle. | + | |
- | * **Do not explicitly close database connections.** Although the API | + | |
- | will try to reopen any closed or broken connections, you create | + | |
- | extra overhead, and you don't play nice with the other web | + | |
- | modules. | + | |
- | * **The obtained connections will use an isolation level of //read | + | |
- | committed//**, i.e. no autocommits. Be careful to commit the | + | |
- | current transaction if you modify any data. A mod_python | + | |
- | ''cleanuphandler'' will try to automatically commit all open | + | |
- | transactions as the request cycle ends, but this may change | + | |
- | in the future, so you must not rely on it. | + | |
- | * **Do not change the isolation level of a connection** without | + | |
- | restoring it to its original value before the end of the request | + | |
- | cycle. | + | |
- | + | ||
- | ===== The "death" of mod_python ===== | + | |
- | ''mod_python'' is no longer under active development and has been | + | |
- | placed in the Apache foundation's "Attic". We do not accept new web | + | |
- | tools that interface directly with ''mod_python''. | + | |
- | + | ||
- | We do, however, aim to refactor existing mod_python-interfacing code | + | |
- | into working as Django views. A few tips for such refactorings: | + | |
- | + | ||
- | * Each ''mod_python'' handler in NAV mostly performs its own custom | + | |
- | URL parsing and view dispatch. It's best to refactor this into a | + | |
- | Django URL configuration and separate view functions first. | + | |
- | * Usage of ''mod_python.utils.FieldStorage'' parse URI arguments | + | |
- | must be refactored to use the ''POST'', ''GET'' or ''REQUEST'' | + | |
- | objects of a Django ''HttpRequest''. It's not that hard, as these | + | |
- | objects behave like dictionaries, much like the ''FieldStorage'' | + | |
- | class does. | + | |
- | * Conversion from Cheetah to Django templates is not necessary to | + | |
- | refactor a mod_python handler into a Django view. It is desirable | + | |
- | to do so in later refactorings, though. | + | |
- | * NAV's authentication and authorization scheme hooks into Apache's | + | |
- | request cycle using a ''mod_python'' ''headerparserhandler''. It | + | |
- | also adds session data as an attribute to the ''mod_python'' | + | |
- | request object. Once there are no tools left that interface | + | |
- | directly with ''mod_python'', the auth and session parts of NAV | + | |
- | must be refactored to work in a pure Django setting before NAV can | + | |
- | be free of its dependence on ''mod_python''. | + | |
- | + | ||
- | ====== Writing new web code ====== | + | |
- | If you are writing a new web application / tool for NAV, please use | + | |
- | the Django framework. [[devel:django_introduction|Here's a quick | + | |
- | primer on how Django integrates with legacy NAV]]. | + | |
- | + | ||
- | ====== Version Control ====== | + | |
- | NAV uses [[http://www.selenic.com/mercurial/|Mercurial]] for | + | |
- | distributed version control. Official repositories are located at | + | |
- | http://nav.uninett.no/hg/ . | + | |
- | + | ||
- | ===== Guide to the repository jungle ===== | + | |
- | The official repositories represent three types of branches | + | |
- | + | ||
- | ==== Unstable (default) ==== | + | |
- | New, bleeding edge development occurs on the | + | |
- | //[[http://nav.uninett.no/hg/default/|default]]// branch, which is | + | |
- | considered unstable (although we try to always keep it buildable). | + | |
- | + | ||
- | ==== Feature branches ==== | + | |
- | New features that take a while (and a lot of changesets) to implement | + | |
- | and test will often be published as separate feature branches. For | + | |
- | all intents and purposes, the feature branches will look like the | + | |
- | //default// branch with some added feature. They will merge changes | + | |
- | from the //default// branch regularly. Once a feature is considered | + | |
- | "ready", the feature branch will be merged onto the default branch. | + | |
- | + | ||
- | ==== Stable (series) ==== | + | |
- | Once we are nearing a new series release of NAV (such as 3.5 or 3.6), | + | |
- | a new [[http://nav.uninett.no/hg/series/|series branch]] is | + | |
- | created from the //default// branch. Once this branch is stabilized, | + | |
- | the first version is tagged and released. After this point, we accept | + | |
- | only bug fixes in this branch. Further point releases in this series | + | |
- | are tagged on this branch, and all changes are merged back onto the | + | |
- | //default// branch. | + | |
- | + | ||
- | When someone writes a patch for a bug, this should usually be | + | |
- | committed to the latest active series branch which is affected by the | + | |
- | bug. Once a new series is released, we do not usually maintain the | + | |
- | older series branches. We may push bug fixes to these branches, but | + | |
- | we are unlikely to create a new point release from it. | + | |
- | + | ||
- | ===== Push access ===== | + | |
- | Push access to the official repositories is limited to developers | + | |
- | employed or commisioned by UNINETT. | + | |
- | + | ||
- | ====== Testing and Continuous Integration ====== | + | |
- | Much of NAV is **legacy code**, as defined by //Michael C. Feathers//: | + | |
- | Code that has no tests. We have been making an effort to introduce | + | |
- | automated tests into the codebase the past couple of years, and hope | + | |
- | to improve coverage in time. | + | |
- | + | ||
- | There are no tests for the legacy Java code, but many unit tests and | + | |
- | integration tests now reside in the ''tests/'' subdirectory. | + | |
- | + | ||
- | ===== Running tests ===== | + | |
- | We use ''[[http://pytest.org/|py.test]]'' to run the test suite. A bundled version is | + | |
- | included as ''runtests.py'' in the ''python/'' subdirectory, which is | + | |
- | used to run the unit tests only when a ''make check'' command is | + | |
- | issued in the ''python/'' subdirectory. | + | |
- | + | ||
- | Some of the test requirements aren't available on the Debian systems | + | |
- | we use for development, so we often test inside a Python | + | |
- | //virtualenv//. A suitable virtualenv for testing (on Debian Lenny) | + | |
- | can be created thus: | + | |
- | + | ||
- | <code bash> | + | |
- | virtualenv .env | + | |
- | . .env/bin/activate | + | |
- | easy_install pip | + | |
- | pip install -r tests/requirements.txt | + | |
- | </code> | + | |
- | + | ||
- | There's also a script to create a test environment, complete with | + | |
- | database initialization. This is used by our CI server. The | + | |
- | following will configure and build NAV automatically, and install it | + | |
- | into a directory called ''workspace/build''. It will also create a | + | |
- | suitable virtualenv in ''workspace/.env'', which you can activate | + | |
- | before running tests: | + | |
- | + | ||
- | <code bash> | + | |
- | export PGDATABASE=testdb | + | |
- | export PGUSER=testuser | + | |
- | tests/bootstrap-test-environment.sh workspace | + | |
- | </code> | + | |
- | + | ||
- | ===== Jenkins ===== | + | |
- | + | ||
- | We use //Jenkins// (formerly //Hudson//) for Continuous Integration testing of | + | |
- | NAV. All the automated tests are run each time new changesets are pushed to | + | |
- | the NAV repositories. Jenkins also runs pylint to create stats on code | + | |
- | quality. | + | |
- | + | ||
- | Our Jenkins installation is available on http://nav.uninett.no/jenkins . | + | |
- | + | ||
- | ===== Tips and tricks ===== | + | |
- | + | ||
- | ===== Make fixtures for integration testing ===== | + | |
- | + | ||
- | <code> | + | |
- | from django.core import serializers | + | |
- | from nav.models.manage import Netbox | + | |
- | + | ||
- | fixtures = serializers.serialize("xml", Netbox.objects.all()[:2]) | + | |
- | </code> | + | |
- | + | ||
- | Fixtures can so be used in your integration tests by extending | + | |
- | the test case DjangoTransactionTestCase in nav.tests.cases | + | |
- | + | ||
- | See nav.tests.integration.l2trace_test for an example on applying | + | |
- | fixtures for your particular test case. | + | |
- | + | ||
- | Also keep in mind you have to make sure you have the model | + | |
- | dependency in correct order when importing. | + | |
- | Example: Netbox contains a location to a Room where it is located, | + | |
- | you have to make sure Room's are imported first before importing | + | |
- | Netbox's | + | |
- | + | ||
- | See https://docs.djangoproject.com/en/dev/topics/serialization/ | + | |
- | + | ||
- | TODO: Be able to use [[https://docs.djangoproject.com/en/dev/ref/django-admin/#dumpdata-appname-appname-appname-model|django-admin's management command: dumpdata]] | + | |
- | to create fixtures. | + | |
- | + | ||
- | + | ||
- | + | ||
- | ====== Submitting patches ====== | + | |
- | Unless you are submitting one-off fixes for bugs and small issues, | + | |
- | please take the time to discuss your change proposals on the | + | |
- | //nav-dev// mailing list. This will increase the chances of having | + | |
- | your patches accepted. | + | |
- | + | ||
- | Base your patches on the relevant Mercurial branches. If you are | + | |
- | submitting a patch for an issue that affects the latest stable series, | + | |
- | base your patch on that series branch. If you are submitting patches | + | |
- | containing new features, base them on the default branch. | + | |
- | + | ||
- | There are three common options for submitting patches: | + | |
- | + | ||
- | * The best way to submit your patches would be using Mercurial. Publish | + | |
- | your own Mercurial branch, and mail its URL to the //nav-dev// mailing | + | |
- | list. | + | |
- | * If unable to host a public Mercurial branch, export your changes | + | |
- | as a Mercurial bundle and attach it to an email addressed to the | + | |
- | //nav-dev// mailing list. | + | |
- | * If you have a single patch to submit, attach it to an email | + | |
- | addressed to the //nav-dev// mailing list. Please **do not | + | |
- | patchbomb** the mailing list with multiple emails. | + |