NAV Wiki

User Tools

Site Tools

Trace: • source_file_headers • hacking
devel:hacking

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
devel:hacking [2012/09/18 07:52] – [Running tests] bredaldevel:hacking [2014/11/05 10:57] (current) – morten
Line 1: Line 1:
 ====== Hacker's guide to NAV ====== ====== Hacker's guide to NAV ======
  
-If you are contributing code to Network Administration Visualized, +This document has moved to https://nav.uninett.no/doc/dev/hacking/hacking.html
-please read this first. +
- +
- +
-====== Contributing to NAV ====== +
-Originally, NAV was a closed source project, initiated by the +
-Norwegian University of Science and Technology (NTNU), and eventually +
-sponsored by UNINETT on behalf of the Norwegian higher education +
-community.  In 2004, however, NTNU and UNINETT started distributing +
-NAV under the GNU General Public License, making it a truly free +
-software system. +
- +
-While UNINETT and NTNU are still the main contributors to NAV, +
-developing NAV to support the needs of the Norwegian higher education +
-community, contributions from third parties is highly appreciated. +
- +
-We communicate mainly through mailing lists, +
-[[https://launchpad.net/nav/|Launchpad]] and the ''#nav'' IRC channel +
-on //FreeNode//.  At times, UNINETT also arranges workshops and +
-gatherings for its customers: Norwegian universities, university +
-colleges and research institutions. +
- +
-To contribute: +
- +
-Go to http://nav.uninett.no/ and +
- +
-  * Join the mailing lists.  The //nav-dev// mailing list in +
-    particular is for discussing NAV development.  So far, this is a +
-    low traffic list. We can only hope this will change ;-) +
-  * Get a copy of the latest development sources by cloning the +
-    Mercurial repository at http://nav.uninett.no/hg/default/. +
-    Most new development takes place on this branch. +
-  * Take a look at the [[:navprojects|project reports from previous +
-    development projects at NTNU]] (NAVMe, NAVMore, tigaNAV and +
-    others) - design specifications and other useful bits of historic +
-    NAV information is mostly to be found in these.  Unfortunately, +
-    some of the oldest project documentation is in Norwegian only.  Do +
-    not hesitate to ask for help on the mailing lists. +
- +
-If you wish to contribute code to the project, see the [[#submitting patches]] section. +
- +
-====== Directory layout ====== +
-A rough guide to the source tree: +
- +
-| bin/       | NAV 'binaries'; executable scripts and programs. | +
-| contrib/   | User contributed NAV tools. NAV doesn't depend on these, and any maintenance of them is left up to the original developers.  We do not offer support for these tools. | +
-| doc/       | User and developer documentation | +
-| etc/       | Example/initial configuration files | +
-| java/      | Java source code | +
-| media/     | Static media such as CSS stylesheets, images and JavaScript to be served by a webserver | +
-| packages/  | Stuff to help packaging NAV for various platforms, such as RedHat, CentOS, FreeBSD, Debian and soforth. Much of this is outdated today. | +
-| python/    | Python source code | +
-| sql/       | SQL schema definitions and installation/sync tools | +
-| templates/ | Django HTML templates | +
-| tests/     | Automated tests | +
-| tools/     | Tool scripts for the build and release processes. | +
- +
-====== Development languages and frameworks ====== +
- +
-Historically, NAV was written using multiple programming languages +
-(Perl, Java, PHP and Python).  While this has had an unfortunate +
-impact on integration and maintenance over the years, we've managed to +
-reduce this to just Python and Java in later years. We have a +
-long-term goal to rewrite the remaining Java backend code to Python. +
- +
-Currently (as of September 2012), NAV consists mostly of Python code, +
-with one remaining backend systems written in Java (eventEngine). +
- +
-  * We will only accept new code written in Python (except when it +
-    involves patches to the existing Java code). +
-  * When you contribute additions to the web interface, use the Django framework. +
- +
-If you wish to contribute something really useful that doesn't use +
-Python, we may consider including it in the //contrib// directory. +
- +
-If **YOU** are willing to invest in porting some of the existing Java +
-code to Python, then you will be celebrated as a NAV hero! +
- +
-====== Coding style ======  +
-Much of the legacy NAV code was written without using any coding style +
-guidelines.  This has resulted in some chaotic combination of styles, +
-which we hope to reduce in the future.  For new code, please follow +
-these guidelines: +
- +
-  * For Java code, please refer to SUN's "Code conventions for the +
-    Java Programming Language": http://java.sun.com/docs/codeconv/ +
-  * For Python code, please refer to PEP-8, "Style Guide for Python +
-    Code" http://www.python.org/doc/peps/pep-0008/ +
- +
-If you see violations of these guidelines, don't hesitate to fix them. +
-If you fix file-wide indentation problems etc., please submit this as +
-a separate patch to make your other patches look clean and readable. +
- +
-===== Python boilerplate headers ===== +
-We will generally only accept code into NAV if it is licensed under +
-GPL v2, but we may make individual exceptions for code licensed under +
-compatible licenses.  Each Python source code file should contain the +
-following boilerplate at the top: +
- +
-<code python> +
-# +
-# Copyright (C) 2008,2009 Somebody +
-# +
-# This file is part of Network Administration Visualized (NAV). +
-# +
-# NAV is free software: you can redistribute it and/or modify it under the +
-# terms of the GNU General Public License version 2 as published by the Free +
-# Software Foundation. +
-# +
-# This program is distributed in the hope that it will be useful, but WITHOUT +
-# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +
-# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +
-# more details.  You should have received a copy of the GNU General Public +
-# License along with NAV. If not, see <http://www.gnu.org/licenses/>. +
-# +
-</code> +
- +
-If a file uses non-ASCII characters, it **must** be encoded as UTF-8, and an +
-encoding statement should be inserted at the top:  +
- +
-<code python> +
-# -*- coding: utf-8 -*- +
-</code> +
- +
-====== Database ====== +
-NAV uses PostgreSQL as its database backend.  Namespaces (schemas) are +
-employed to logically group tables and relations.  NAV versions prior +
-to 3.5 employed separate PostgreSQL databases instead of namespaces. +
- +
-The namespaces currently in use are: +
- +
-| ''manage''      | The core knowledge database of NAV, containing all sorts of information about the monitored IP Devices, events, alerts, network topology and machine tracking data. | +
-| ''profiles''    | Contains NAV user accounts and groups, user preferences and alert profiles. | +
-| ''logger''      | Anything related to NAV's syslog parser/browser system. | +
-| ''arnold''      | The port detention system Arnold stores it's data here. | +
-| ''radius''      | Radius accounting logs, updated directly by FreeRadius' PostgreSQL module. | +
- +
-===== Connecting to the database (Python) ==== +
-==== Raw SQL ==== +
- +
-To obtain a connection to the NAV database, use the API accordingly, +
-e.g.: +
- +
-<code python> +
-import nav.db +
-# Get a connection to the NAV database +
-connection = nav.db.getConnection('default') +
-</code> +
- +
-The above code will open a connection to NAV's database, or, if a +
-previous connection with these parameters is already open, returns the +
-already existing connection from a connection cache. +
- +
-The ''default'' parameter is there for legacy reasons; it specifies +
-the name of a subsystem.  The ''db.conf'' file allows configuration of +
-separate database users for each subsystem (known as a ''script'' in +
-''db.conf'') of NAV.  The default ''db.conf'' file specifies a +
-database user for a subsystem called ''default'', and also specifies +
-the same database user for all known subsystem names.  At present, +
-using a subsystem name that is not configured in ''db.conf'' will cause ''nav.db.getConnection()'' to revert to using the ''default'' name. +
- +
-==== Django models ==== +
-NAV 3.5 and on includes Django models for most database tables.  If no +
-SQL magic is needed to perform your database voodoo, it is recommended +
-that you use these models, located in the module ''nav.models''.  You +
-do not need to explicitly establish a database connection to use these +
-models, as Django takes care of all that. +
- +
-The models are defined in modules of the ''nav.models'' package. +
- +
-===== Changing the schema ==== +
- +
-The baseline schema is located in ''sql/baseline'' - the ''syncdb.py'' script +
-is responsible for running this when creating a new database. To make a schema +
-change, you **do not** change the baseline, but go to the ''sql/changes'' +
-directory and create a new schema change script there. +
- +
-Schema change scripts as numbered, using the following pattern: +
- +
-  * ''sc.<major>.<minor>.<point>.sql'' +
- +
-The ''<major>'' and ''<minor>'' numbers usually correspond to the major and +
-minor number of the next NAV release.  The ''<point>'' number is a sequence id +
-- pick the next free number when creating a schema change script. +
- +
-Remember these points when creating a schema change script: +
- +
-  * Create separate change scripts for unrelated schema changes. +
-  * Remember to write SQL to //migrate// existing data, if necessary. +
-  * Do not use transactional statements - the ''syncdb.py'' script will take +
-    care of that. +
- +
-To apply your change scripts, just run ''syncdb.py''.  It will look inside the +
-''schema_change_log'' table to see which change scripts have already been +
-applied, and it will detect your new change script and apply this to the +
-database. +
- +
-:!: When changing the schema, don't forget to update the Django models in the +
-''nav.models'' package.  An integration test exists to verify that the Django +
-models can at least be used to run proper SELECTs against the database. +
- +
-====== Legacy web code ====== +
-Legacy web code interfaces directly with +
-[[http://www.modpython.org/|mod_python]], and uses +
-[[http://www.cheetahtemplate.org/|Cheetah for HTML templating]]. +
- +
-All Cheetah templates are located in the ''python/nav/web/templates'' +
-directory. +
- +
-===== Legacy database connections in web code ===== +
-Use the ''nav.db.getConnection()'' call to open or retrieve an +
-existing database connection.  All NAV web modules share the same +
-interpreter and namespace per Apache process, which also means that +
-database connections will be shared between the modules running in +
-each process.  Therefore, the following conventions apply for +
-connections obtained from ''nav.db.getConnection()'': +
- +
-  * **Do not, under any circumstances**, retain references to a database +
-    connection between client requests.  Make sure to retrieve a new +
-    connection at the start of each request cycle - the API will cache +
-    connections between requests, and will automagically re-open +
-    broken connections.  As the connection is shared between several +
-    modules, retained references may be invalid in the next request +
-    cycle. +
-  * **Do not explicitly close database connections.**  Although the API +
-    will try to reopen any closed or broken connections, you create +
-    extra overhead, and you don't play nice with the other web +
-    modules. +
-  * **The obtained connections will use an isolation level of //read +
-    committed//**, i.e. no autocommits.  Be careful to commit the +
-    current transaction if you modify any data.  A mod_python +
-    ''cleanuphandler'' will try to automatically commit all open +
-    transactions as the request cycle ends, but this may change +
-    in the future, so you must not rely on it. +
-  * **Do not change the isolation level of a connection** without +
-    restoring it to its original value before the end of the request +
-    cycle. +
- +
-===== The "death" of mod_python ===== +
-''mod_python'' is no longer under active development and has been +
-placed in the Apache foundation's "Attic".  We do not accept new web +
-tools that interface directly with ''mod_python''. +
- +
-We do, however, aim to refactor existing mod_python-interfacing code +
-into working as Django views.  A few tips for such refactorings: +
- +
-  * Each ''mod_python'' handler in NAV mostly performs its own custom +
-    URL parsing and view dispatch.  It's best to refactor this into a +
-    Django URL configuration and separate view functions first. +
-  * Usage of ''mod_python.utils.FieldStorage'' parse URI arguments +
-    must be refactored to use the ''POST'', ''GET'' or ''REQUEST'' +
-    objects of a Django ''HttpRequest''. It's not that hard, as these +
-    objects behave like dictionaries, much like the ''FieldStorage'' +
-    class does. +
-  * Conversion from Cheetah to Django templates is not necessary to +
-    refactor a mod_python handler into a Django view.  It is desirable +
-    to do so in later refactorings, though. +
-  * NAV's authentication and authorization scheme hooks into Apache's +
-    request cycle using a ''mod_python'' ''headerparserhandler''. It +
-    also adds session data as an attribute to the ''mod_python'' +
-    request object.  Once there are no tools left that interface +
-    directly with ''mod_python'', the auth and session parts of NAV +
-    must be refactored to work in a pure Django setting before NAV can +
-    be free of its dependence on ''mod_python''. +
- +
-====== Writing new web code ====== +
-If you are writing a new web application / tool for NAV, please use +
-the Django framework.  [[devel:django_introduction|Here's a quick +
-primer on how Django integrates with legacy NAV]]. +
- +
-====== Version Control ====== +
-NAV uses [[http://www.selenic.com/mercurial/|Mercurial]] for +
-distributed version control.  Official repositories are located at +
-http://nav.uninett.no/hg/ . +
- +
-===== Guide to the repository jungle ===== +
-The official repositories represent three types of branches +
- +
-==== Unstable (default) ==== +
-New, bleeding edge development occurs on the +
-//[[http://nav.uninett.no/hg/default/|default]]// branch, which is +
-considered unstable (although we try to always keep it buildable). +
- +
-==== Feature branches ==== +
-New features that take a while (and a lot of changesets) to implement +
-and test will often be published as separate feature branches.  For +
-all intents and purposes, the feature branches will look like the +
-//default// branch with some added feature.  They will merge changes +
-from the //default// branch regularly.  Once a feature is considered +
-"ready", the feature branch will be merged onto the default branch. +
- +
-==== Stable (series) ==== +
-Once we are nearing a new series release of NAV (such as 3.5 or 3.6), +
-a new [[http://nav.uninett.no/hg/series/|series branch]] is +
-created from the //default// branch.  Once this branch is stabilized, +
-the first version is tagged and released.  After this point, we accept +
-only bug fixes in this branch.  Further point releases in this series +
-are tagged on this branch, and all changes are merged back onto the +
-//default// branch. +
- +
-When someone writes a patch for a bug, this should usually be +
-committed to the latest active series branch which is affected by the +
-bug.  Once a new series is released, we do not usually maintain the +
-older series branches.  We may push bug fixes to these branches, but +
-we are unlikely to create a new point release from it. +
- +
-===== Push access ===== +
-Push access to the official repositories is limited to developers +
-employed or commisioned by UNINETT. +
- +
-====== Testing and Continuous Integration ====== +
-Much of NAV is **legacy code**, as defined by //Michael C. Feathers//: +
-Code that has no tests.  We have been making an effort to introduce +
-automated tests into the codebase the past couple of years, and hope +
-to improve coverage in time. +
- +
-There are no tests for the legacy Java code, but many unit tests and +
-integration tests now reside in the ''tests/'' subdirectory. +
- +
-===== Running tests ===== +
-We use ''[[http://pytest.org/|py.test]]'' to run the test suite.  A bundled version is +
-included as ''runtests.py'' in the ''python/'' subdirectory, which is +
-used to run the unit tests only when a ''make check'' command is +
-issued in the ''python/'' subdirectory. +
- +
-Some of the test requirements aren't available on the Debian systems +
-we use for development, so we often test inside a Python +
-//virtualenv//.  A suitable virtualenv for testing (on Debian Lenny) +
-can be created thus: +
- +
-<code bash> +
-virtualenv .env +
-. .env/bin/activate +
-easy_install pip +
-pip install -r tests/requirements.txt +
-</code> +
- +
-There's also a script to create a test environment, complete with +
-database initialization.  This is used by our CI server.  The +
-following will configure and build NAV automatically, and install it +
-into a directory called ''workspace/build''.  It will also create a +
-suitable virtualenv in ''workspace/.env'', which you can activate +
-before running tests: +
- +
-<code bash> +
-export PGDATABASE=testdb +
-export PGUSER=testuser +
-tests/bootstrap-test-environment.sh workspace +
-</code> +
- +
- +
-===== Javascript testing ===== +
- +
-blapp +
-===== Jenkins ===== +
- +
-We use //Jenkins// (formerly //Hudson//) for Continuous Integration testing of +
-NAV.  All the automated tests are run each time new changesets are pushed to +
-the NAV repositories.  Jenkins also runs pylint to create stats on code +
-quality. +
- +
-Our Jenkins installation is available on http://nav.uninett.no/jenkins . +
- +
-===== Tips and tricks ===== +
- +
-===== Make fixtures for integration testing ===== +
- +
-<code> +
-from django.core import serializers +
-from nav.models.manage import Netbox +
- +
-fixtures = serializers.serialize("xml", Netbox.objects.all()[:2]) +
-</code> +
- +
-Fixtures can so be used in your integration tests by extending +
-the test case DjangoTransactionTestCase in nav.tests.cases +
- +
-See nav.tests.integration.l2trace_test for an example on applying +
-fixtures for your particular test case.  +
- +
-Also keep in mind you have to make sure you have the model +
-dependency in correct order when importing. +
-Example: Netbox contains a location to a Room where it is located, +
-you have to make sure Room's are imported first before importing +
-Netbox's +
- +
-See https://docs.djangoproject.com/en/dev/topics/serialization/ +
- +
-TODO: Be able to use [[https://docs.djangoproject.com/en/dev/ref/django-admin/#dumpdata-appname-appname-appname-model|django-admin's management command: dumpdata]] +
-to create fixtures.  +
- +
- +
- +
-====== Submitting patches ====== +
-Unless you are submitting one-off fixes for bugs and small issues, +
-please take the time to discuss your change proposals on the +
-//nav-dev// mailing list.  This will increase the chances of having +
-your patches accepted. +
- +
-Base your patches on the relevant Mercurial branches.  If you are +
-submitting a patch for an issue that affects the latest stable series, +
-base your patch on that series branch.  If you are submitting patches +
-containing new features, base them on the default branch. +
- +
-There are three common options for submitting patches: +
- +
-  * The best way to submit your patches would be using Mercurial. Publish +
-    your own Mercurial branch, and mail its URL to the //nav-dev// mailing +
-    list. +
-  * If unable to host a public Mercurial branch, export your changes +
-    as a Mercurial bundle and attach it to an email addressed to the +
-    //nav-dev// mailing list.   +
-  * If you have a single patch to submit, attach it to an email +
-    addressed to the //nav-dev// mailing list.  Please **do not +
-    patchbomb** the mailing list with multiple emails.+
devel/hacking.1347954769.txt.gz · Last modified: by bredal
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki