This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Last revision Both sides next revision | ||
devel:hacking [2009/02/09 11:24] morten break list item lines (installed mllist plugin to enable this) |
devel:hacking [2013/06/19 07:45] morten fix jenkins url |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | :!: **Be warned, this information is becoming very out of date, it is currently being updated** :!: | ||
====== Hacker's guide to NAV ====== | ====== Hacker's guide to NAV ====== | ||
Line 6: | Line 5: | ||
- | ===== Participating in the community ===== | + | ====== Contributing to NAV ====== |
Originally, NAV was a closed source project, initiated by the | Originally, NAV was a closed source project, initiated by the | ||
Norwegian University of Science and Technology (NTNU), and eventually | Norwegian University of Science and Technology (NTNU), and eventually | ||
sponsored by UNINETT on behalf of the Norwegian higher education | sponsored by UNINETT on behalf of the Norwegian higher education | ||
community. In 2004, however, NTNU and UNINETT started distributing | community. In 2004, however, NTNU and UNINETT started distributing | ||
- | NAV under the GNU General Public License, making it a true open source | + | NAV under the GNU General Public License, making it a truly free |
- | project. | + | software system. |
- | While NTNU and UNINETT still are the main contributors to NAV, | + | While UNINETT and NTNU are still the main contributors to NAV, |
developing NAV to support the needs of the Norwegian higher education | developing NAV to support the needs of the Norwegian higher education | ||
- | community, volunteer work from other interested parties is highly | + | community, contributions from third parties is highly appreciated. |
- | appreciated. | + | |
- | The community exists mainly through mailing lists, a wiki and a | + | We communicate mainly through mailing lists, |
- | Subversion repository, although UNINETT also arranges seminars and | + | [[https://launchpad.net/nav/|Launchpad]] and the ''#nav'' IRC channel |
- | gatherings for its target audience: Norwegian universities and | + | on //FreeNode//. At times, UNINETT also arranges workshops and |
- | university colleges. To participate: | + | gatherings for its customers: Norwegian universities, university |
+ | colleges and research institutions. | ||
- | Go to http://metanav.uninett.no/ and | + | To contribute: |
- | * Join the mailing lists. The nav-dev mailing list in particular is | + | Go to http://nav.uninett.no/ and |
- | for discussing NAV development. So far, this is a low traffic | + | |
- | list. We can only hope this will change ;-) | + | |
- | * Get a copy of the latest development sources from | + | |
- | https://svn.itea.ntnu.no/repos/nav/navme/trunk/. New development | + | |
- | always takes place on trunk. Bugfixes, enhancements, and new | + | |
- | features are backported from there to the various release | + | |
- | branches. | + | |
- | * Take a look at the project reports from previous development | + | |
- | projects at NTNU (NAVMe, NAVMore, tigaNAV and others) - design | + | |
- | specifications and other useful bits of historic NAV information | + | |
- | is mostly to be found in these. Unfortunately, some of the older | + | |
- | project documentation is in Norwegian only. Do not hesitate to | + | |
- | ask for help on the mailing lists. | + | |
- | If you wish to contribute code to the project, please tell us about it | + | * Join the mailing lists. The //nav-dev// mailing list in |
- | on the nav-dev mailing list. It is always a good idea to check if | + | particular is for discussing NAV development. So far, this is a |
- | someone is already working on something similar, and to get some | + | low traffic list. We can only hope this will change ;-) |
- | helpful tips on how to integrate your code with the rest of the | + | * Get a copy of the latest development sources by cloning the |
- | project. If you already went ahead and wrote a patch, announce it on | + | Mercurial repository at http://nav.uninett.no/hg/default/. |
- | nav-dev and provide a link to the patch so it can be studied for | + | Most new development takes place on this branch. |
- | possible inclusion into NAV. | + | * Take a look at the [[:navprojects|project reports from previous |
+ | development projects at NTNU]] (NAVMe, NAVMore, tigaNAV and | ||
+ | others) - design specifications and other useful bits of historic | ||
+ | NAV information is mostly to be found in these. Unfortunately, | ||
+ | some of the oldest project documentation is in Norwegian only. Do | ||
+ | not hesitate to ask for help on the mailing lists. | ||
+ | If you wish to contribute code to the project, see the [[#submitting patches]] section. | ||
- | ===== Directory layout ===== | + | ====== Directory layout ====== |
A rough guide to the source tree: | A rough guide to the source tree: | ||
- | | conf/ | Files related to the autoconf build system. | | + | | bin/ | NAV 'binaries'; executable scripts and programs. | |
- | | doc/ | User and developer documentation, SQL scripts and example NAV configuration files. | | + | | contrib/ | User contributed NAV tools. NAV doesn't depend on these, and any maintenance of them is left up to the original developers. We do not offer support for these tools. | |
- | | tools/ | Tool scripts for the build process. | | + | | doc/ | User and developer documentation | |
- | | contrib/ | Stuff that works with NAV, but that NAV doesn't depend on, and that is maintained by individuals who may or may not participate in NAV development. | | + | | etc/ | Example/initial configuration files | |
- | | packages/ | Stuff to help packaging systems, like rpm and dpkg. | | + | | java/ | Java source code | |
- | | src/ | Source code to Java subsystems of NAV (still here for historic reasons). | | + | | media/ | Static media such as CSS stylesheets, images and JavaScript to be served by a webserver | |
- | | subsystem/ | Source code to the rest of NAV - a lot of Python. NAV is loosely divided into subsystems, and each one of these has its own subdirectory in here. | | + | | packages/ | Stuff to help packaging NAV for various platforms, such as RedHat, CentOS, FreeBSD, Debian and soforth. Much of this is outdated today. | |
- | | subsystem/lib-python/ | Python libraries & APIs. Please check what's already there before you roll your own. | | + | | python/ | Python source code | |
- | | subsystem/lib-perl/ | Perl libraries & APIs. Please check what's already there before you roll your own. | | + | | sql/ | SQL schema definitions and installation/sync tools | |
- | | subsystem/webfront/ | Python libraries for the web interface and front-page handler modules for mod_python. | | + | | templates/ | Django HTML templates | |
+ | | tests/ | Automated tests | | ||
+ | | tools/ | Tool scripts for the build and release processes. | | ||
+ | ====== Development languages and frameworks ====== | ||
- | ===== Development languages ===== | + | Historically, NAV was written using multiple programming languages |
- | For historic reasons, NAV today is made up of several programming | + | (Perl, Java, PHP and Python). While this has had an unfortunate |
- | languages - these are Python, Java and Perl. Although this is | + | impact on integration and maintenance over the years, we've managed to |
- | unfortunate in many ways, no-one has been willing to invest in the | + | reduce this to just Python and Java in later years. We have a |
- | time needed to rewrite stuff to reduce the number of languages. | + | long-term goal to rewrite the remaining Java backend code to Python. |
- | When contributing patches to existing code, or plugins to existing | + | Currently (as of September 2012), NAV consists mostly of Python code, |
- | subsystems, use the language that subsystem was written in. | + | with one remaining backend systems written in Java (eventEngine). |
- | When writing entirely new subsystems, the following rules apply: | + | * We will only accept new code written in Python (except when it |
+ | involves patches to the existing Java code). | ||
+ | * When you contribute additions to the web interface, use the Django framework. | ||
- | * If your subsystem is a new tool for the web interface, use | + | If you wish to contribute something really useful that doesn't use |
- | Python. The web interface is built using mod_python. Also, the | + | Python, we may consider including it in the //contrib// directory. |
- | integration with Cricket uses Cricket's web front-end, which is | + | |
- | Perl CGI. | + | |
- | * If your subsystem is a new back-end tool/daemon, please use | + | |
- | Python. The Python API created for NAV is more complete than for | + | |
- | any of the other languages, and you will receive a lot for free. | + | |
- | You may use Perl if you absolutely abhor Python, but then you will | + | |
- | be frowned upon. | + | |
- | If *YOU* are willing to invest in porting some of the existing code to | + | If **YOU** are willing to invest in porting some of the existing Java |
- | Python, then you will be celebrated as a NAV hero! | + | code to Python, then you will be celebrated as a NAV hero! |
- | + | ====== Coding style ====== | |
- | ===== Coding style ===== | + | Much of the legacy NAV code was written without using any coding style |
- | NAV has not previously bothered with having coding style guidelines. | + | guidelines. This has resulted in some chaotic combination of styles, |
- | This has resulted in some chaotic combination of styles, which we hope | + | which we hope to reduce in the future. For new code, please follow |
- | to reduce in the future. For new code, please follow these | + | these guidelines: |
- | guidelines: | + | |
* For Java code, please refer to SUN's "Code conventions for the | * For Java code, please refer to SUN's "Code conventions for the | ||
Line 101: | Line 90: | ||
Code" http://www.python.org/doc/peps/pep-0008/ | Code" http://www.python.org/doc/peps/pep-0008/ | ||
- | If you see violations of these guidelines, don't hesitate to report | + | If you see violations of these guidelines, don't hesitate to fix them. |
- | them and/or fix them :) If you fix file-wide indentation problems | + | If you fix file-wide indentation problems etc., please submit this as |
- | etc., please submit this as a separate patch to make your other | + | a separate patch to make your other patches look clean and readable. |
- | patches look clean and readable. | + | |
+ | ===== Python boilerplate headers ===== | ||
+ | We will generally only accept code into NAV if it is licensed under | ||
+ | GPL v2, but we may make individual exceptions for code licensed under | ||
+ | compatible licenses. Each Python source code file should contain the | ||
+ | following boilerplate at the top: | ||
- | ===== Database connections ===== | + | <code python> |
- | NOTE: The following is Python-specific, more info should be added for | + | # |
- | the other languages used in NAV. | + | # Copyright (C) 2008,2009 Somebody |
+ | # | ||
+ | # This file is part of Network Administration Visualized (NAV). | ||
+ | # | ||
+ | # NAV is free software: you can redistribute it and/or modify it under the | ||
+ | # terms of the GNU General Public License version 2 as published by the Free | ||
+ | # Software Foundation. | ||
+ | # | ||
+ | # This program is distributed in the hope that it will be useful, but WITHOUT | ||
+ | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
+ | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
+ | # more details. You should have received a copy of the GNU General Public | ||
+ | # License along with NAV. If not, see <http://www.gnu.org/licenses/>. | ||
+ | # | ||
+ | </code> | ||
- | The NAV database actually consists of four PostgreSQL databases | + | If a file uses non-ASCII characters, it **must** be encoded as UTF-8, and an |
- | (although there are plans to merge them into a single database using | + | encoding statement should be inserted at the top: |
- | PostgreSQL's support for schemas). The databases are: | + | |
+ | <code python> | ||
+ | # -*- coding: utf-8 -*- | ||
+ | </code> | ||
+ | |||
+ | ===== Javascript ===== | ||
+ | |||
+ | When writing javascript code try to focus on modules not pages. If the code is html-related, it should take selectors or objects as input and concern itself solely about those. This makes for much easier testing and reuse. And of course - write the tests first. | ||
+ | |||
+ | When the module is done you write a controller for the page that plugs the needed plugins to the page elements. This should fail gracefully if the needed elements are not present. | ||
+ | |||
+ | NAVs javascript uses [[http://requirejs.org/|require.js]] - use this to create modules and specify dependencies. | ||
+ | |||
+ | Pro tip is to create ''require_config.dev.js'' in ''media/js/'' and add the following configuration to requirejs: | ||
+ | <code>require.urlArgs = "bust=" + (new Date()).getTime();</code> This makes sure your not using cached resources in your browser when developing, which browsers loves to do! See [[http://requirejs.org/docs/api.html#config-urlArgs|config-urlArgs]] in requirejs documentation for «details». The ''require_config.dev.js'' is added in global HG ignore. | ||
+ | |||
+ | ==== Accessing resources with ajax requiring authentication ==== | ||
+ | |||
+ | As your authenticated session might have timed out due to idle (no activity), resources will return 500 Internal Error if you do not supply the important ''//X-NAV-AJAX//'' header on your ajax requests. | ||
+ | |||
+ | So make sure to include the ''**X-NAV-AJAX**'' header so you will get proper HTTP response code in the reponse from your request. | ||
+ | |||
+ | NAV has a shortcut for fixing this in ''default.js'' (adds a function in the public namespace NAV) which attaches the required handlers for jQuery by doing: | ||
+ | <code>NAV.addGlobalAjaxHandlers()</code> | ||
+ | |||
+ | ====== Database ====== | ||
+ | NAV uses PostgreSQL as its database backend. Namespaces (schemas) are | ||
+ | employed to logically group tables and relations. NAV versions prior | ||
+ | to 3.5 employed separate PostgreSQL databases instead of namespaces. | ||
+ | |||
+ | The namespaces currently in use are: | ||
| ''manage'' | The core knowledge database of NAV, containing all sorts of information about the monitored IP Devices, events, alerts, network topology and machine tracking data. | | | ''manage'' | The core knowledge database of NAV, containing all sorts of information about the monitored IP Devices, events, alerts, network topology and machine tracking data. | | ||
- | | ''navprofiles'' | Contains NAV user accounts and groups, user preferences and alert profiles. | | + | | ''profiles'' | Contains NAV user accounts and groups, user preferences and alert profiles. | |
- | | ''logger'' | Contains syslog entries collected by the syslog parser/browser system. | | + | | ''logger'' | Anything related to NAV's syslog parser/browser system. | |
| ''arnold'' | The port detention system Arnold stores it's data here. | | | ''arnold'' | The port detention system Arnold stores it's data here. | | ||
+ | | ''radius'' | Radius accounting logs, updated directly by FreeRadius' PostgreSQL module. | | ||
+ | |||
+ | ===== Connecting to the database (Python) ==== | ||
+ | ==== Raw SQL ==== | ||
To obtain a connection to the NAV database, use the API accordingly, | To obtain a connection to the NAV database, use the API accordingly, | ||
Line 125: | Line 166: | ||
<code python> | <code python> | ||
import nav.db | import nav.db | ||
- | # Get a connection to the manage database | + | # Get a connection to the NAV database |
- | connection = nav.db.getConnection('default', 'manage') | + | connection = nav.db.getConnection('default') |
</code> | </code> | ||
- | The above code will open a connection to the manage database, or, if | + | The above code will open a connection to NAV's database, or, if a |
- | the connection had already been opened during the lifetime of the | + | previous connection with these parameters is already open, returns the |
- | current process, returns the already existing connection from a | + | already existing connection from a connection cache. |
- | connection cache. | + | |
- | The 'default' parameter is there for legacy reasons; the db.conf file | + | The ''default'' parameter is there for legacy reasons; it specifies |
- | allows one to configure separate database users for each subsystem | + | the name of a subsystem. The ''db.conf'' file allows configuration of |
- | (known as a script in db.conf) of NAV, but it is recommended to use | + | separate database users for each subsystem (known as a ''script'' in |
- | only one user for all NAV database connections - hence the default | + | ''db.conf'') of NAV. The default ''db.conf'' file specifies a |
- | "subsystem". | + | database user for a subsystem called ''default'', and also specifies |
+ | the same database user for all known subsystem names. At present, | ||
+ | using a subsystem name that is not configured in ''db.conf'' will cause ''nav.db.getConnection()'' to revert to using the ''default'' name. | ||
+ | ==== Django models ==== | ||
+ | NAV 3.5 and on includes Django models for most database tables. If no | ||
+ | SQL magic is needed to perform your database voodoo, it is recommended | ||
+ | that you use these models, located in the module ''nav.models''. You | ||
+ | do not need to explicitly establish a database connection to use these | ||
+ | models, as Django takes care of all that. | ||
- | ===== Web interface ===== | + | The models are defined in modules of the ''nav.models'' package. |
- | When programming for NAV's web interface, a few special considerations | + | |
- | need to be made. | + | |
- | ==== Mod_python ==== | + | ===== Changing the schema ==== |
- | NAV uses mod_python to interface with the Apache web server. See | + | |
- | http://www.modpython.org/. | + | |
- | ==== Cheetah Templates ==== | + | The baseline schema is located in ''sql/baseline'' - the ''syncdb.py'' script |
- | The NAV web interface makes extensive use of Cheetah templates for | + | is responsible for running this when creating a new database. To make a schema |
- | generating its HTML output, see http://www.cheetahtemplate.org/ . | + | change, you **do not** change the baseline, but go to the ''sql/changes'' |
+ | directory and create a new schema change script there. | ||
- | Most of the existing Cheetah templates are to be found in | + | Schema change scripts as numbered, using the following pattern: |
- | subsystem/webfront/nav/web/templates, although some of NAV's subsystems | + | |
- | store their templates along with their code in their respective subsystem | + | |
- | subdirectories. The compiled templates should be placed in the | + | |
- | nav.web.templates package. | + | |
- | If you are making a new web module for NAV, your module's Cheetah | + | * ''sc.<major>.<minor>.<point>.sql'' |
- | template should subclass MainTemplate.tmpl found in | + | |
- | subsystem/webfront/nav/web/templates/. See other templates for code | + | |
- | examples of how to inherit from this template. | + | |
- | ==== Database connections in the web interface ==== | + | The ''<major>'' and ''<minor>'' numbers usually correspond to the major and |
- | As stated above, use the nav.db.getConnection function to open or | + | minor number of the next NAV release. The ''<point>'' number is a sequence id |
- | retrieve an existing database connection. All NAV web modules share | + | - pick the next free number when creating a schema change script. |
- | the same interpreter and namespace per Apache process, which also | + | |
- | means that database connections will be shared between the modules | + | |
- | running in each process. Therefore, the following conventions apply | + | |
- | for connections obtained from nav.db.getConnection: | + | |
- | * Do not, under any circumstances, retain references to a database | + | Remember these points when creating a schema change script: |
+ | |||
+ | * Create separate change scripts for unrelated schema changes. | ||
+ | * Remember to write SQL to //migrate// existing data, if necessary. | ||
+ | * Do not use transactional statements - the ''syncdb.py'' script will take | ||
+ | care of that. | ||
+ | |||
+ | To apply your change scripts, just run ''syncdb.py''. It will look inside the | ||
+ | ''schema_change_log'' table to see which change scripts have already been | ||
+ | applied, and it will detect your new change script and apply this to the | ||
+ | database. | ||
+ | |||
+ | :!: When changing the schema, don't forget to update the Django models in the | ||
+ | ''nav.models'' package. An integration test exists to verify that the Django | ||
+ | models can at least be used to run proper SELECTs against the database. | ||
+ | |||
+ | ====== Legacy web code ====== | ||
+ | Legacy web code interfaces directly with | ||
+ | [[http://www.modpython.org/|mod_python]], and uses | ||
+ | [[http://www.cheetahtemplate.org/|Cheetah for HTML templating]]. | ||
+ | |||
+ | All Cheetah templates are located in the ''python/nav/web/templates'' | ||
+ | directory. | ||
+ | |||
+ | ===== Legacy database connections in web code ===== | ||
+ | Use the ''nav.db.getConnection()'' call to open or retrieve an | ||
+ | existing database connection. All NAV web modules share the same | ||
+ | interpreter and namespace per Apache process, which also means that | ||
+ | database connections will be shared between the modules running in | ||
+ | each process. Therefore, the following conventions apply for | ||
+ | connections obtained from ''nav.db.getConnection()'': | ||
+ | |||
+ | * **Do not, under any circumstances**, retain references to a database | ||
connection between client requests. Make sure to retrieve a new | connection between client requests. Make sure to retrieve a new | ||
connection at the start of each request cycle - the API will cache | connection at the start of each request cycle - the API will cache | ||
Line 179: | Line 245: | ||
modules, retained references may be invalid in the next request | modules, retained references may be invalid in the next request | ||
cycle. | cycle. | ||
- | * Do not explicitly close database connections. Although the API | + | * **Do not explicitly close database connections.** Although the API |
will try to reopen any closed or broken connections, you create | will try to reopen any closed or broken connections, you create | ||
extra overhead, and you don't play nice with the other web | extra overhead, and you don't play nice with the other web | ||
modules. | modules. | ||
- | * Do not enable/disable autocommit or alter a connection's | + | * **The obtained connections will use an isolation level of //read |
- | transaction isolation level, unless you make pretty darn sure to | + | committed//**, i.e. no autocommits. Be careful to commit the |
- | reset them to their original states at the end of a request cycle. | + | current transaction if you modify any data. A mod_python |
- | * NAV 3.0 makes connections autocommit by default, whereas NAV 3.1 | + | ''cleanuphandler'' will try to automatically commit all open |
- | will not - make sure to commit your transactions when needed. NAV | + | transactions as the request cycle ends, but this may change |
- | will help unfortunate souls by attempting to commit transactions | + | in the future, so you must not rely on it. |
- | in a mod_python cleanuphandler, but you should nevertheless | + | * **Do not change the isolation level of a connection** without |
- | explicitly call connection.commit to avoid having your | + | restoring it to its original value before the end of the request |
- | transactions accidentally rolled back. | + | cycle. |
+ | |||
+ | ===== The "death" of mod_python ===== | ||
+ | ''mod_python'' is no longer under active development and has been | ||
+ | placed in the Apache foundation's "Attic". We do not accept new web | ||
+ | tools that interface directly with ''mod_python''. | ||
+ | |||
+ | We do, however, aim to refactor existing mod_python-interfacing code | ||
+ | into working as Django views. A few tips for such refactorings: | ||
+ | |||
+ | * Each ''mod_python'' handler in NAV mostly performs its own custom | ||
+ | URL parsing and view dispatch. It's best to refactor this into a | ||
+ | Django URL configuration and separate view functions first. | ||
+ | * Usage of ''mod_python.utils.FieldStorage'' parse URI arguments | ||
+ | must be refactored to use the ''POST'', ''GET'' or ''REQUEST'' | ||
+ | objects of a Django ''HttpRequest''. It's not that hard, as these | ||
+ | objects behave like dictionaries, much like the ''FieldStorage'' | ||
+ | class does. | ||
+ | * Conversion from Cheetah to Django templates is not necessary to | ||
+ | refactor a mod_python handler into a Django view. It is desirable | ||
+ | to do so in later refactorings, though. | ||
+ | * NAV's authentication and authorization scheme hooks into Apache's | ||
+ | request cycle using a ''mod_python'' ''headerparserhandler''. It | ||
+ | also adds session data as an attribute to the ''mod_python'' | ||
+ | request object. Once there are no tools left that interface | ||
+ | directly with ''mod_python'', the auth and session parts of NAV | ||
+ | must be refactored to work in a pure Django setting before NAV can | ||
+ | be free of its dependence on ''mod_python''. | ||
+ | |||
+ | ====== Writing new web code ====== | ||
+ | If you are writing a new web application / tool for NAV, please use | ||
+ | the Django framework. [[devel:django_introduction|Here's a quick | ||
+ | primer on how Django integrates with legacy NAV]]. | ||
+ | |||
+ | ====== Version Control ====== | ||
+ | NAV uses [[http://www.selenic.com/mercurial/|Mercurial]] for | ||
+ | distributed version control. Official repositories are located at | ||
+ | http://nav.uninett.no/hg/ . | ||
+ | |||
+ | ===== Guide to the repository jungle ===== | ||
+ | The official repositories represent three types of branches | ||
+ | |||
+ | ==== Unstable (default) ==== | ||
+ | New, bleeding edge development occurs on the | ||
+ | //[[http://nav.uninett.no/hg/default/|default]]// branch, which is | ||
+ | considered unstable (although we try to always keep it buildable). | ||
+ | |||
+ | ==== Feature branches ==== | ||
+ | New features that take a while (and a lot of changesets) to implement | ||
+ | and test will often be published as separate feature branches. For | ||
+ | all intents and purposes, the feature branches will look like the | ||
+ | //default// branch with some added feature. They will merge changes | ||
+ | from the //default// branch regularly. Once a feature is considered | ||
+ | "ready", the feature branch will be merged onto the default branch. | ||
+ | |||
+ | ==== Stable (series) ==== | ||
+ | Once we are nearing a new series release of NAV (such as 3.5 or 3.6), | ||
+ | a new [[http://nav.uninett.no/hg/series/|series branch]] is | ||
+ | created from the //default// branch. Once this branch is stabilized, | ||
+ | the first version is tagged and released. After this point, we accept | ||
+ | only bug fixes in this branch. Further point releases in this series | ||
+ | are tagged on this branch, and all changes are merged back onto the | ||
+ | //default// branch. | ||
+ | |||
+ | When someone writes a patch for a bug, this should usually be | ||
+ | committed to the latest active series branch which is affected by the | ||
+ | bug. Once a new series is released, we do not usually maintain the | ||
+ | older series branches. We may push bug fixes to these branches, but | ||
+ | we are unlikely to create a new point release from it. | ||
+ | |||
+ | ===== Push access ===== | ||
+ | Push access to the official repositories is limited to developers | ||
+ | employed or commisioned by UNINETT. | ||
+ | |||
+ | ====== Testing and Continuous Integration ====== | ||
+ | Much of NAV is **legacy code**, as defined by //Michael C. Feathers//: | ||
+ | Code that has no tests. We have been making an effort to introduce | ||
+ | automated tests into the codebase the past couple of years, and hope | ||
+ | to improve coverage in time. | ||
+ | |||
+ | There are no tests for the legacy Java code, but many unit tests and | ||
+ | integration tests now reside in the ''tests/'' subdirectory. | ||
+ | |||
+ | ===== Running tests ===== | ||
+ | We use ''[[http://pytest.org/|py.test]]'' to run the test suite. A bundled version is | ||
+ | included as ''runtests.py'' in the ''python/'' subdirectory, which is | ||
+ | used to run the unit tests only when a ''make check'' command is | ||
+ | issued in the ''python/'' subdirectory. | ||
+ | |||
+ | Some of the test requirements aren't available on the Debian systems | ||
+ | we use for development, so we often test inside a Python | ||
+ | //virtualenv//. A suitable virtualenv for testing (on Debian Lenny) | ||
+ | can be created thus: | ||
+ | |||
+ | <code bash> | ||
+ | virtualenv .env | ||
+ | . .env/bin/activate | ||
+ | easy_install pip | ||
+ | pip install -r tests/requirements.txt | ||
+ | </code> | ||
+ | |||
+ | There's also a script to create a test environment, complete with | ||
+ | database initialization. This is used by our CI server. The | ||
+ | following will configure and build NAV automatically, and install it | ||
+ | into a directory called ''workspace/build''. It will also create a | ||
+ | suitable virtualenv in ''workspace/.env'', which you can activate | ||
+ | before running tests: | ||
+ | |||
+ | <code bash> | ||
+ | export PGDATABASE=testdb | ||
+ | export PGUSER=testuser | ||
+ | tests/bootstrap-test-environment.sh workspace | ||
+ | </code> | ||
+ | |||
+ | |||
+ | ===== Javascript testing ===== | ||
+ | |||
+ | Testing of javascript is in its infancy in NAV. We are currently using [[http://busterjs.org/|buster.js]] as testing toolkit. | ||
+ | |||
+ | To install buster.js install [[http://nodejs.org/|node]] and then: | ||
+ | <code> | ||
+ | npm install -g buster | ||
+ | </code> | ||
+ | |||
+ | As we use [[http://requirejs.org/|require.js]] you need the AMD module of buster aswell. Install it in the /media/js directory: | ||
+ | <code> | ||
+ | npm install buster-amd | ||
+ | </code> | ||
+ | |||
+ | To run the tests you need to | ||
+ | - Start a buster server by typing ''buster-server'' | ||
+ | - Capture browsers by pointing browsers to the buster-server (default localhost:1111) | ||
+ | - Go to /media/js | ||
+ | - Run the tests by typing ''buster-test'' | ||
+ | |||
+ | All tests are located under ''media/js/tests/''. Create new tests there. For syntax, assertions and related stuff take a look at the tests already there and [[http://busterjs.org/docs/|the buster docs]]. | ||
+ | ===== Jenkins ===== | ||
+ | |||
+ | We use //Jenkins// (formerly //Hudson//) for Continuous Integration testing of | ||
+ | NAV. All the automated tests are run each time new changesets are pushed to | ||
+ | the NAV repositories. Jenkins also runs pylint to create stats on code | ||
+ | quality. | ||
+ | |||
+ | Our Jenkins installation is available on https://ci.nav.uninett.no/ . | ||
+ | |||
+ | ===== Tips and tricks ===== | ||
+ | |||
+ | ===== Make fixtures for integration testing ===== | ||
+ | |||
+ | <code> | ||
+ | from django.core import serializers | ||
+ | from nav.models.manage import Netbox | ||
+ | |||
+ | fixtures = serializers.serialize("xml", Netbox.objects.all()[:2]) | ||
+ | </code> | ||
+ | |||
+ | Fixtures can so be used in your integration tests by extending | ||
+ | the test case DjangoTransactionTestCase in nav.tests.cases | ||
+ | |||
+ | See nav.tests.integration.l2trace_test for an example on applying | ||
+ | fixtures for your particular test case. | ||
+ | |||
+ | Also keep in mind you have to make sure you have the model | ||
+ | dependency in correct order when importing. | ||
+ | Example: Netbox contains a location to a Room where it is located, | ||
+ | you have to make sure Room's are imported first before importing | ||
+ | Netbox's | ||
+ | |||
+ | See https://docs.djangoproject.com/en/dev/topics/serialization/ | ||
+ | |||
+ | TODO: Be able to use [[https://docs.djangoproject.com/en/dev/ref/django-admin/#dumpdata-appname-appname-appname-model|django-admin's management command: dumpdata]] | ||
+ | to create fixtures. | ||
+ | |||
+ | |||
+ | |||
+ | ====== Submitting patches ====== | ||
+ | Unless you are submitting one-off fixes for bugs and small issues, | ||
+ | please take the time to discuss your change proposals on the | ||
+ | //nav-dev// mailing list. This will increase the chances of having | ||
+ | your patches accepted. | ||
+ | |||
+ | Base your patches on the relevant Mercurial branches. If you are | ||
+ | submitting a patch for an issue that affects the latest stable series, | ||
+ | base your patch on that series branch. If you are submitting patches | ||
+ | containing new features, base them on the default branch. | ||
+ | |||
+ | There are three common options for submitting patches: | ||
+ | * The best way to submit your patches would be using Mercurial. Publish | ||
+ | your own Mercurial branch, and mail its URL to the //nav-dev// mailing | ||
+ | list. | ||
+ | * If unable to host a public Mercurial branch, export your changes | ||
+ | as a Mercurial bundle and attach it to an email addressed to the | ||
+ | //nav-dev// mailing list. | ||
+ | * If you have a single patch to submit, attach it to an email | ||
+ | addressed to the //nav-dev// mailing list. Please **do not | ||
+ | patchbomb** the mailing list with multiple emails. |