User Tools

Site Tools


devel:hacking

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
devel:hacking [2012/11/14 13:22]
norangshol [Javascript] Updated info about require_config.dev.js.
devel:hacking [2014/11/05 10:57] (current)
morten
Line 1: Line 1:
 ====== Hacker'​s guide to NAV ====== ====== Hacker'​s guide to NAV ======
  
-If you are contributing code to Network Administration Visualized,​ +This document has moved to https://​nav.uninett.no/​doc/​dev/​hacking/hacking.html
-please read this first. +
- +
- +
-====== Contributing to NAV ====== +
-Originally, NAV was a closed source project, initiated by the +
-Norwegian University of Science and Technology (NTNU), and eventually +
-sponsored by UNINETT on behalf of the Norwegian higher education +
-community. ​ In 2004, however, NTNU and UNINETT started distributing +
-NAV under the GNU General Public License, making it a truly free +
-software system. +
- +
-While UNINETT and NTNU are still the main contributors to NAV, +
-developing NAV to support the needs of the Norwegian higher education +
-community, contributions from third parties is highly appreciated. +
- +
-We communicate mainly through mailing lists, +
-[[https://​launchpad.net/​nav/​|Launchpad]] and the ''#​nav''​ IRC channel +
-on //​FreeNode//​. ​ At times, UNINETT also arranges workshops and +
-gatherings for its customers: Norwegian universities,​ university +
-colleges and research institutions. +
- +
-To contribute:​ +
- +
-Go to http://​nav.uninett.no/ ​and +
- +
-  * Join the mailing lists. ​ The //nav-dev// mailing list in +
-    particular is for discussing NAV development. ​ So far, this is a +
-    low traffic list. We can only hope this will change ;-) +
-  * Get a copy of the latest development sources by cloning the +
-    Mercurial repository at http://​nav.uninett.no/​hg/​default/​. +
-    Most new development takes place on this branch. +
-  * Take a look at the [[:​navprojects|project reports from previous +
-    development projects at NTNU]] (NAVMe, NAVMore, tigaNAV and +
-    others) - design specifications and other useful bits of historic +
-    NAV information is mostly to be found in these. ​ Unfortunately,​ +
-    some of the oldest project documentation is in Norwegian only.  Do +
-    not hesitate to ask for help on the mailing lists. +
- +
-If you wish to contribute code to the project, see the [[#​submitting patches]] section. +
- +
-====== Directory layout ====== +
-A rough guide to the source tree: +
- +
-| bin/       | NAV '​binaries';​ executable scripts and programs. | +
-| contrib/ ​  | User contributed NAV tools. NAV doesn'​t depend on these, and any maintenance of them is left up to the original developers. ​ We do not offer support for these tools. | +
-doc/       | User and developer documentation | +
-| etc/       | Example/​initial configuration files | +
-| java/      | Java source code | +
-| media/ ​    | Static media such as CSS stylesheets,​ images and JavaScript to be served by a webserver | +
-| packages/ ​ | Stuff to help packaging NAV for various platforms, such as RedHat, CentOS, FreeBSD, Debian and soforth. Much of this is outdated today. | +
-| python/ ​   | Python source code | +
-| sql/       | SQL schema definitions and installation/​sync tools | +
-| templates/ | Django HTML templates | +
-| tests/ ​    | Automated tests | +
-| tools/ ​    | Tool scripts for the build and release processes. | +
- +
-====== Development languages and frameworks ====== +
- +
-Historically,​ NAV was written using multiple programming languages +
-(Perl, Java, PHP and Python). ​ While this has had an unfortunate +
-impact on integration and maintenance over the years, we've managed to +
-reduce this to just Python and Java in later years. We have a +
-long-term goal to rewrite the remaining Java backend code to Python. +
- +
-Currently (as of September 2012), NAV consists mostly of Python code, +
-with one remaining backend systems written in Java (eventEngine). +
- +
-  * We will only accept new code written in Python (except when it +
-    involves patches to the existing Java code). +
-  * When you contribute additions to the web interface, use the Django framework. +
- +
-If you wish to contribute something really useful that doesn'​t use +
-Python, we may consider including it in the //contrib// directory. +
- +
-If **YOU** are willing to invest in porting some of the existing Java +
-code to Python, then you will be celebrated as a NAV hero! +
- +
-====== Coding style ======  +
-Much of the legacy NAV code was written without using any coding style +
-guidelines. ​ This has resulted in some chaotic combination of styles, +
-which we hope to reduce in the future. ​ For new code, please follow +
-these guidelines:​ +
- +
-  * For Java code, please refer to SUN's "Code conventions for the +
-    Java Programming Language":​ http://​java.sun.com/​docs/​codeconv/​ +
-  * For Python code, please refer to PEP-8, "Style Guide for Python +
-    Code" http://​www.python.org/​doc/​peps/​pep-0008/​ +
- +
-If you see violations of these guidelines, don't hesitate to fix them. +
-If you fix file-wide indentation problems etc., please submit this as +
-a separate patch to make your other patches look clean and readable. +
- +
-===== Python boilerplate headers ===== +
-We will generally only accept code into NAV if it is licensed under +
-GPL v2, but we may make individual exceptions for code licensed under +
-compatible licenses. ​ Each Python source code file should contain the +
-following boilerplate at the top: +
- +
-<code python>​ +
-+
-# Copyright (C) 2008,2009 Somebody +
-+
-# This file is part of Network Administration Visualized (NAV). +
-+
-# NAV is free software: you can redistribute it and/or modify it under the +
-# terms of the GNU General Public License version 2 as published by the Free +
-# Software Foundation. +
-+
-# This program is distributed in the hope that it will be useful, but WITHOUT +
-# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or +
-# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for +
-# more details. ​ You should have received a copy of the GNU General Public +
-# License along with NAV. If not, see <​http://​www.gnu.org/​licenses/>​. +
-+
-</​code>​ +
- +
-If a file uses non-ASCII characters, it **must** be encoded as UTF-8, and an +
-encoding statement should be inserted at the top:  +
- +
-<code python>​ +
-# -*- coding: utf-8 -*- +
-</​code>​ +
- +
-===== Javascript ===== +
- +
-When writing javascript code try to focus on modules not pages. If the code is html-related,​ it should take selectors or objects as input and concern itself solely about those. This makes for much easier testing and reuse. And of course - write the tests first.  +
- +
-When the module is done you write a controller for the page that plugs the needed plugins to the page elements. This should fail gracefully if the needed elements are not present. +
- +
-NAVs javascript uses [[http://​requirejs.org/​|require.js]] - use this to create modules and specify dependencies.  +
- +
-Pro tip is to create ''​require_config.dev.js''​ in ''​media/js/''​ and add the following configuration to requirejs:  +
-<​code>​require.urlArgs = "​bust="​ +  (new Date()).getTime();</​code>​ This makes sure your not using cached resources in your browser when developing, which browser loves to do! See [[http://​requirejs.org/​docs/​api.html#​config-urlArgs|config-urlArgs]] in requirejs documentation for «details» +
-====== Database ====== +
-NAV uses PostgreSQL as its database backend. ​ Namespaces (schemas) are +
-employed to logically group tables and relations. ​ NAV versions prior +
-to 3.5 employed separate PostgreSQL databases instead of namespaces. +
- +
-The namespaces currently in use are: +
- +
-| ''​manage'' ​     | The core knowledge database of NAV, containing all sorts of information about the monitored IP Devices, events, alerts, network topology and machine tracking data. | +
-| ''​profiles'' ​   | Contains NAV user accounts and groups, user preferences and alert profiles. | +
-| ''​logger'' ​     | Anything related to NAV's syslog parser/​browser system. | +
-| ''​arnold'' ​     | The port detention system Arnold stores it's data here. | +
-| ''​radius'' ​     | Radius accounting logs, updated directly by FreeRadius'​ PostgreSQL module. | +
- +
-===== Connecting to the database (Python) ==== +
-==== Raw SQL ==== +
- +
-To obtain a connection to the NAV database, use the API accordingly,​ +
-e.g.: +
- +
-<code python>​ +
-import nav.db +
-# Get a connection to the NAV database +
-connection = nav.db.getConnection('​default'​) +
-</​code>​ +
- +
-The above code will open a connection to NAV's database, or, if a +
-previous connection with these parameters is already open, returns the +
-already existing connection from a connection cache. +
- +
-The ''​default''​ parameter is there for legacy reasons; it specifies +
-the name of a subsystem. ​ The ''​db.conf''​ file allows configuration of +
-separate database users for each subsystem (known as a ''​script''​ in +
-''​db.conf''​) of NAV.  The default ''​db.conf''​ file specifies a +
-database user for a subsystem called ''​default'',​ and also specifies +
-the same database user for all known subsystem names. ​ At present, +
-using a subsystem name that is not configured in ''​db.conf''​ will cause ''​nav.db.getConnection()''​ to revert to using the ''​default''​ name. +
- +
-==== Django models ==== +
-NAV 3.5 and on includes Django models for most database tables. ​ If no +
-SQL magic is needed to perform your database voodoo, it is recommended +
-that you use these models, located in the module ''​nav.models''​. ​ You +
-do not need to explicitly establish a database connection to use these +
-models, as Django takes care of all that. +
- +
-The models are defined in modules of the ''​nav.models''​ package. +
- +
-===== Changing the schema ==== +
- +
-The baseline schema is located in ''​sql/​baseline''​ - the ''​syncdb.py''​ script +
-is responsible for running this when creating a new database. To make a schema +
-change, you **do not** change the baseline, but go to the ''​sql/​changes''​ +
-directory and create a new schema change script there. +
- +
-Schema change scripts as numbered, using the following pattern: +
- +
-  * ''​sc.<​major>​.<​minor>​.<​point>​.sql''​ +
- +
-The ''<​major>''​ and ''<​minor>''​ numbers usually correspond to the major and +
-minor number of the next NAV release. ​ The ''<​point>''​ number is a sequence id +
-- pick the next free number when creating a schema change script. +
- +
-Remember these points when creating a schema change script: +
- +
-  * Create separate change scripts for unrelated schema changes. +
-  * Remember to write SQL to //migrate// existing data, if necessary. +
-  * Do not use transactional statements - the ''​syncdb.py''​ script will take +
-    care of that. +
- +
-To apply your change scripts, just run ''​syncdb.py''​. ​ It will look inside the +
-''​schema_change_log''​ table to see which change scripts have already been +
-applied, and it will detect your new change script and apply this to the +
-database. +
- +
-:!: When changing the schema, don't forget to update the Django models in the +
-''​nav.models''​ package. ​ An integration test exists to verify that the Django +
-models can at least be used to run proper SELECTs against the database. +
- +
-====== Legacy web code ====== +
-Legacy web code interfaces directly with +
-[[http://​www.modpython.org/​|mod_python]],​ and uses +
-[[http://​www.cheetahtemplate.org/​|Cheetah for HTML templating]]. +
- +
-All Cheetah templates are located in the ''​python/​nav/​web/​templates''​ +
-directory. +
- +
-===== Legacy database connections in web code ===== +
-Use the ''​nav.db.getConnection()''​ call to open or retrieve an +
-existing database connection. ​ All NAV web modules share the same +
-interpreter and namespace per Apache process, which also means that +
-database connections will be shared between the modules running in +
-each process. ​ Therefore, the following conventions apply for +
-connections obtained from ''​nav.db.getConnection()'':​ +
- +
-  * **Do not, under any circumstances**,​ retain references to a database +
-    connection between client requests. ​ Make sure to retrieve a new +
-    connection at the start of each request cycle - the API will cache +
-    connections between requests, and will automagically re-open +
-    broken connections. ​ As the connection is shared between several +
-    modules, retained references may be invalid in the next request +
-    cycle. +
-  * **Do not explicitly close database connections.** ​ Although the API +
-    will try to reopen any closed or broken connections,​ you create +
-    extra overhead, and you don't play nice with the other web +
-    modules. +
-  * **The obtained connections will use an isolation level of //read +
-    committed//​**,​ i.e. no autocommits. ​ Be careful to commit the +
-    current transaction if you modify any data.  A mod_python +
-    ''​cleanuphandler''​ will try to automatically commit all open +
-    transactions as the request cycle ends, but this may change +
-    in the future, so you must not rely on it. +
-  * **Do not change the isolation level of a connection** without +
-    restoring it to its original value before the end of the request +
-    cycle. +
- +
-===== The "​death"​ of mod_python ===== +
-''​mod_python''​ is no longer under active development and has been +
-placed in the Apache foundation'​s "​Attic"​. ​ We do not accept new web +
-tools that interface directly with ''​mod_python''​. +
- +
-We do, however, aim to refactor existing mod_python-interfacing code +
-into working as Django views. ​ A few tips for such refactorings:​ +
- +
-  * Each ''​mod_python''​ handler in NAV mostly performs its own custom +
-    URL parsing and view dispatch. ​ It's best to refactor this into a +
-    Django URL configuration and separate view functions first. +
-  * Usage of ''​mod_python.utils.FieldStorage''​ parse URI arguments +
-    must be refactored to use the ''​POST'',​ ''​GET''​ or ''​REQUEST''​ +
-    objects of a Django ''​HttpRequest''​. It's not that hard, as these +
-    objects behave like dictionaries,​ much like the ''​FieldStorage''​ +
-    class does. +
-  * Conversion from Cheetah to Django templates is not necessary to +
-    refactor a mod_python handler into a Django view.  It is desirable +
-    to do so in later refactorings,​ though. +
-  * NAV's authentication and authorization scheme hooks into Apache'​s +
-    request cycle using a ''​mod_python''​ ''​headerparserhandler''​. It +
-    also adds session data as an attribute to the ''​mod_python''​ +
-    request object. ​ Once there are no tools left that interface +
-    directly with ''​mod_python'',​ the auth and session parts of NAV +
-    must be refactored to work in a pure Django setting before NAV can +
-    be free of its dependence on ''​mod_python''​. +
- +
-====== Writing new web code ====== +
-If you are writing a new web application / tool for NAV, please use +
-the Django framework. ​ [[devel:​django_introduction|Here'​s a quick +
-primer on how Django integrates with legacy NAV]]. +
- +
-====== Version Control ====== +
-NAV uses [[http://​www.selenic.com/​mercurial/​|Mercurial]] for +
-distributed version control. ​ Official repositories are located at +
-http://​nav.uninett.no/​hg/​ . +
- +
-===== Guide to the repository jungle ===== +
-The official repositories represent three types of branches +
- +
-==== Unstable (default) ==== +
-New, bleeding edge development occurs on the +
-//​[[http://​nav.uninett.no/​hg/​default/​|default]]//​ branch, which is +
-considered unstable (although we try to always keep it buildable). +
- +
-==== Feature branches ==== +
-New features that take a while (and a lot of changesets) to implement +
-and test will often be published as separate feature branches. ​ For +
-all intents and purposes, the feature branches will look like the +
-//default// branch with some added feature. ​ They will merge changes +
-from the //default// branch regularly. ​ Once a feature is considered +
-"​ready",​ the feature branch will be merged onto the default branch. +
- +
-==== Stable (series) ==== +
-Once we are nearing a new series release of NAV (such as 3.5 or 3.6), +
-a new [[http://​nav.uninett.no/​hg/​series/​|series branch]] is +
-created from the //default// branch. ​ Once this branch is stabilized,​ +
-the first version is tagged and released. ​ After this point, we accept +
-only bug fixes in this branch. ​ Further point releases in this series +
-are tagged on this branch, and all changes are merged back onto the +
-//default// branch. +
- +
-When someone writes a patch for a bug, this should usually be +
-committed to the latest active series branch which is affected by the +
-bug.  Once a new series is released, we do not usually maintain the +
-older series branches. ​ We may push bug fixes to these branches, but +
-we are unlikely to create a new point release from it. +
- +
-===== Push access ===== +
-Push access to the official repositories is limited to developers +
-employed or commisioned by UNINETT. +
- +
-====== Testing and Continuous Integration ====== +
-Much of NAV is **legacy code**, as defined by //Michael C. Feathers//:​ +
-Code that has no tests. ​ We have been making an effort to introduce +
-automated tests into the codebase the past couple of years, and hope +
-to improve coverage in time. +
- +
-There are no tests for the legacy Java code, but many unit tests and +
-integration tests now reside in the ''​tests/''​ subdirectory. +
- +
-===== Running tests ===== +
-We use ''​[[http://​pytest.org/​|py.test]]''​ to run the test suite. ​ A bundled version is +
-included as ''​runtests.py''​ in the ''​python/''​ subdirectory,​ which is +
-used to run the unit tests only when a ''​make check''​ command is +
-issued in the ''​python/''​ subdirectory. +
- +
-Some of the test requirements aren't available on the Debian systems +
-we use for development,​ so we often test inside a Python +
-//​virtualenv//​. ​ A suitable virtualenv for testing (on Debian Lenny) +
-can be created thus: +
- +
-<code bash> +
-virtualenv .env +
-. .env/​bin/​activate +
-easy_install pip +
-pip install -r tests/​requirements.txt +
-</​code>​ +
- +
-There'​s also a script to create a test environment,​ complete with +
-database initialization. ​ This is used by our CI server. ​ The +
-following will configure and build NAV automatically,​ and install it +
-into a directory called ''​workspace/​build''​. ​ It will also create a +
-suitable virtualenv in ''​workspace/​.env'',​ which you can activate +
-before running tests: +
- +
-<code bash> +
-export PGDATABASE=testdb +
-export PGUSER=testuser +
-tests/​bootstrap-test-environment.sh workspace +
-</​code>​ +
- +
- +
-===== Javascript testing ===== +
- +
-Testing of javascript is in its infancy in NAV. We are currently using [[http://​busterjs.org/​|buster.js]] as testing toolkit. +
- +
-To install buster.js install [[http://​nodejs.org/​|node]] and then: +
-<​code>​ +
-npm install -g buster +
-</​code>​ +
- +
-As we use [[http://​requirejs.org/​|require.js]] you need the AMD module of buster aswell. Install it in the /media/js directory:​ +
-<​code>​ +
-npm install buster-amd +
-</​code>​ +
- +
-To run the tests you need to +
-  - Start a buster server by typing ''​buster-server''​ +
-  - Capture browsers by pointing browsers to the buster-server (default localhost:​1111) +
-  - Go to /media/js +
-  - Run the tests by typing ''​buster-test''​ +
- +
-All tests are located under ''​media/​js/​tests/''​. Create new tests there. For syntax, assertions and related stuff take a look at the tests already there and [[http://​busterjs.org/​docs/​|the buster docs]]. +
-===== Jenkins ===== +
- +
-We use //Jenkins// (formerly //Hudson//) for Continuous Integration testing of +
-NAV.  All the automated tests are run each time new changesets are pushed to +
-the NAV repositories. ​ Jenkins also runs pylint to create stats on code +
-quality. +
- +
-Our Jenkins installation is available on http://​nav.uninett.no/​jenkins . +
- +
-===== Tips and tricks ===== +
- +
-===== Make fixtures for integration testing ===== +
- +
-<​code>​ +
-from django.core import serializers +
-from nav.models.manage import Netbox +
- +
-fixtures = serializers.serialize("​xml",​ Netbox.objects.all()[:​2]) +
-</​code>​ +
- +
-Fixtures can so be used in your integration tests by extending +
-the test case DjangoTransactionTestCase in nav.tests.cases +
- +
-See nav.tests.integration.l2trace_test for an example on applying +
-fixtures for your particular test case.  +
- +
-Also keep in mind you have to make sure you have the model +
-dependency in correct order when importing. +
-Example: Netbox contains a location to a Room where it is located, +
-you have to make sure Room's are imported first before importing +
-Netbox'​s +
- +
-See https://​docs.djangoproject.com/​en/​dev/​topics/​serialization/​ +
- +
-TODO: Be able to use [[https://​docs.djangoproject.com/​en/​dev/​ref/​django-admin/#​dumpdata-appname-appname-appname-model|django-admin'​s management command: dumpdata]] +
-to create fixtures.  +
- +
- +
- +
-====== Submitting patches ====== +
-Unless you are submitting one-off fixes for bugs and small issues, +
-please take the time to discuss your change proposals on the +
-//nav-dev// mailing list.  This will increase the chances of having +
-your patches accepted. +
- +
-Base your patches on the relevant Mercurial branches. ​ If you are +
-submitting a patch for an issue that affects the latest stable series, +
-base your patch on that series branch. ​ If you are submitting patches +
-containing new features, base them on the default branch. +
- +
-There are three common options for submitting patches: +
- +
-  * The best way to submit your patches would be using Mercurial. Publish +
-    your own Mercurial branch, and mail its URL to the //nav-dev// mailing +
-    list. +
-  * If unable to host a public Mercurial branch, export your changes +
-    as a Mercurial bundle and attach it to an email addressed to the +
-    //nav-dev// mailing list.   +
-  * If you have a single patch to submit, attach it to an email +
-    addressed to the //nav-dev// mailing list.  Please **do not +
-    patchbomb** the mailing list with multiple emails.+
devel/hacking.1352899350.txt.gz · Last modified: 2012/11/14 13:22 by norangshol