Differences

This shows you the differences between two versions of the page.

--- devel:hacking [2009/02/09 14:32]
morten
+++ devel:hacking [2013/06/19 07:45]
morten fix jenkins url
@@ Line 1: / Line 1: @@
-:!: **This information is currently being updated, check back often.**
 ====== Hacker's guide to NAV ======
@@ Line 19: / Line 18: @@
 We communicate mainly through mailing lists,
-[[https://launchpad.net/nav/|Launchpad]], and the wiki and Mercurial
+[[https://launchpad.net/nav/|Launchpad]] and the ''#nav'' IRC channel
-repositories hosted at http://metanav.uninett.no/ .  At times, UNINETT
+on //FreeNode//.  At times, UNINETT also arranges workshops and
-also arranges workshops and gatherings for its customers: Norwegian
+gatherings for its customers: Norwegian universities, university
-universities, university colleges and research institutions.
+colleges and research institutions.
 To contribute:
-Go to http://metanav.uninett.no/ and
+Go to http://nav.uninett.no/ and
   * Join the mailing lists.  The //nav-dev// mailing list in
@@ Line 32: / Line 31: @@
     low traffic list. We can only hope this will change ;-)
   * Get a copy of the latest development sources by cloning the
-    Mercurial repository at http://metanav.uninett.no/hg/default/.
+    Mercurial repository at http://nav.uninett.no/hg/default/.
     Most new development takes place on this branch.
   * Take a look at the [[:navprojects|project reports from previous
@@ Line 41: / Line 40: @@
     not hesitate to ask for help on the mailing lists.
-If you wish to contribute code to the project, please tell us about it
+If you wish to contribute code to the project, see the [[#submitting patches]] section.
-on the nav-dev mailing list.  It is always a good idea to check if
-someone is already working on something similar, and to get some
-helpful tips on how to integrate your code with the rest of the
-project.  If you already went ahead and wrote a patch, announce it on
-nav-dev and provide a link to the patch so it can be studied for
-possible inclusion into NAV.
 ====== Directory layout ======
 A rough guide to the source tree:
-| conf/      | Files related to the autoconf build system. |
+| bin/       | NAV 'binaries'; executable scripts and programs. |
-| doc/       | User and developer documentation, SQL scripts and example NAV configuration files. |
+| contrib/   | User contributed NAV tools. NAV doesn't depend on these, and any maintenance of them is left up to the original developers.  We do not offer support for these tools. |
+| doc/       | User and developer documentation |
+| etc/       | Example/initial configuration files |
+| java/      | Java source code |
+| media/     | Static media such as CSS stylesheets, images and JavaScript to be served by a webserver |
+| packages/  | Stuff to help packaging NAV for various platforms, such as RedHat, CentOS, FreeBSD, Debian and soforth. Much of this is outdated today. |
+| python/    | Python source code |
+| sql/       | SQL schema definitions and installation/sync tools |
+| templates/ | Django HTML templates |
+| tests/     | Automated tests |
 | tools/     | Tool scripts for the build and release processes. |
-| contrib/   | User contributed NAV tools. NAV doesn't depend on these, and any maintenance of them is left up to the original developers.  We do not offer support for these tools. |
-| packages/  | Stuff to help packaging NAV for various platforms, such as RedHat, CentOS, FreeBSD, Debian and soforth. |
-| src/       | Source code to Java subsystems of NAV (here for historic reasons). |
-| subsystem/ | Source code to the rest of NAV - most of it Python.  NAV is loosely divided into subsystems, and each one of these has its own subdirectory in here. |
-| subsystem/lib-python/ | Python libraries & APIs.  Please check what's already there before you roll your own. |
-| subsystem/lib-perl/ | Perl libraries & APIs.  Only a single piece of Perl code remains in NAV; once this has been replaced, this directory will cease to exist. |
-| subsystem/webfront/ | Python libraries for the web interface and front-page handler modules for mod_python. |
 ====== Development languages and frameworks ======
-For historic reasons, different parts of NAV are written in different
-programming languages (Perl, Java, PHP and Python).  This has been
-unfortunate in may ways, not at least for the sake of consistency and
-maintenance, but fortunately we have a long-term goal of reducing the
-number of languages and dependencies.  The last few years we've spent
-a significant amount of time rewriting parts that were written in Perl
-and PHP to Python, which is the language we are currently gravitating
-towards.
-Currently (as of February 2009), major parts of NAV are written in
-Python and Java, while only a single Perl program remains
-(''makecricketconfig.pl'').
-When contributing patches to existing code, or plugins to existing
-subsystems, use the language that subsystem was written in.
-When writing entirely new subsystems, the following rules apply:
+Historically, NAV was written using multiple programming languages
+(Perl, Java, PHP and Python).  While this has had an unfortunate
+impact on integration and maintenance over the years, we've managed to
+reduce this to just Python and Java in later years. We have a
+long-term goal to rewrite the remaining Java backend code to Python.
-  * If your subsystem is a new tool for the web interface, use Python
+Currently (as of September 2012), NAV consists mostly of Python code,
-    and Django.  The legacy parts of the web system interface directly
+with one remaining backend systems written in Java (eventEngine).
-    with ''mod_python'', using Cheetah for HTML templating.  Anything
-    written in 2008 and later uses Django.  There is also
-    [[devel:django_introduction|a guide for interfacing Django
-    applications with the legacy web code]].
-  * If your subsystem is a new back-end tool/daemon, please use
+  * We will only accept new code written in Python (except when it
-    Python.  The NAV Python API is more complete than for any of the
+    involves patches to the existing Java code).
-    other languages, and you will receive a lot for free.
+  * When you contribute additions to the web interface, use the Django framework.
-If these rules are not followed, your patches will not be accepted
+If you wish to contribute something really useful that doesn't use
-into NAV (but if they are really good, we will consider including them
+Python, we may consider including it in the //contrib// directory.
-in the //contrib// directory).
-If **YOU** are willing to invest in porting some of the existing
-Java/Perl code to Python, then you will be celebrated as a NAV hero!
+If **YOU** are willing to invest in porting some of the existing Java
+code to Python, then you will be celebrated as a NAV hero!
 ====== Coding style ======
@@ Line 112: / Line 89: @@
   * For Python code, please refer to PEP-8, "Style Guide for Python
     Code" http://www.python.org/doc/peps/pep-0008/
-FIXME Add info about header guidelines for Python files.
 If you see violations of these guidelines, don't hesitate to fix them.
@@ Line 119: / Line 94: @@
 a separate patch to make your other patches look clean and readable.
+===== Python boilerplate headers =====
+We will generally only accept code into NAV if it is licensed under
+GPL v2, but we may make individual exceptions for code licensed under
+compatible licenses.  Each Python source code file should contain the
+following boilerplate at the top:
-====== Database connections ======
+<code python>
-NOTE: The following is Python-specific, more info should be added for
+#
-the other languages used in NAV.
+# Copyright (C) 2008,2009 Somebody
+#
+# This file is part of Network Administration Visualized (NAV).
+#
+# NAV is free software: you can redistribute it and/or modify it under the
+# terms of the GNU General Public License version 2 as published by the Free
+# Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+# more details.  You should have received a copy of the GNU General Public
+# License along with NAV. If not, see <http://www.gnu.org/licenses/>.
+#
+</code>
+If a file uses non-ASCII characters, it **must** be encoded as UTF-8, and an
+encoding statement should be inserted at the top:
+<code python>
+# -*- coding: utf-8 -*-
+</code>
+===== Javascript =====
+When writing javascript code try to focus on modules not pages. If the code is html-related, it should take selectors or objects as input and concern itself solely about those. This makes for much easier testing and reuse. And of course - write the tests first.
+When the module is done you write a controller for the page that plugs the needed plugins to the page elements. This should fail gracefully if the needed elements are not present.
+NAVs javascript uses [[http://requirejs.org/|require.js]] - use this to create modules and specify dependencies.
+Pro tip is to create ''require_config.dev.js'' in ''media/js/'' and add the following configuration to requirejs:
+<code>require.urlArgs = "bust=" +  (new Date()).getTime();</code> This makes sure your not using cached resources in your browser when developing, which browsers loves to do! See [[http://requirejs.org/docs/api.html#config-urlArgs|config-urlArgs]] in requirejs documentation for «details». The ''require_config.dev.js'' is added in global HG ignore.
+==== Accessing resources with ajax requiring authentication ====
+As your authenticated session might have timed out due to idle (no activity), resources will return 500 Internal Error if you do not supply the important ''//X-NAV-AJAX//'' header on your ajax requests.
+So make sure to include the ''**X-NAV-AJAX**'' header so you will get proper HTTP response code in the reponse from your request.
+NAV has a shortcut for fixing this in ''default.js'' (adds a function in the public namespace NAV) which attaches the required handlers for jQuery by doing:
+<code>NAV.addGlobalAjaxHandlers()</code>
+====== Database ======
 NAV uses PostgreSQL as its database backend.  Namespaces (schemas) are
 employed to logically group tables and relations.  NAV versions prior
@@ Line 134: / Line 156: @@
 | ''logger''      | Anything related to NAV's syslog parser/browser system. |
 | ''arnold''      | The port detention system Arnold stores it's data here. |
+| ''radius''      | Radius accounting logs, updated directly by FreeRadius' PostgreSQL module. |
+===== Connecting to the database (Python) ====
+==== Raw SQL ====
 To obtain a connection to the NAV database, use the API accordingly,
@@ Line 154: / Line 180: @@
 database user for a subsystem called ''default'', and also specifies
 the same database user for all known subsystem names.  At present,
-using a subsystem name that is not configured in ''db.conf'' will
+using a subsystem name that is not configured in ''db.conf'' will cause ''nav.db.getConnection()'' to revert to using the ''default'' name.
-raise an exception in ''nav.db.getConnection()''.
-====== Legacy web code ======
+==== Django models ====
-When making changes to NAV's legacy web code, a few special
+NAV 3.5 and on includes Django models for most database tables.  If no
-considerations need to be made.
+SQL magic is needed to perform your database voodoo, it is recommended
+that you use these models, located in the module ''nav.models''.  You
+do not need to explicitly establish a database connection to use these
+models, as Django takes care of all that.
+The models are defined in modules of the ''nav.models'' package.
+===== Changing the schema ====
+The baseline schema is located in ''sql/baseline'' - the ''syncdb.py'' script
+is responsible for running this when creating a new database. To make a schema
+change, you **do not** change the baseline, but go to the ''sql/changes''
+directory and create a new schema change script there.
+Schema change scripts as numbered, using the following pattern:
+  * ''sc.<major>.<minor>.<point>.sql''
+The ''<major>'' and ''<minor>'' numbers usually correspond to the major and
+minor number of the next NAV release.  The ''<point>'' number is a sequence id
+- pick the next free number when creating a schema change script.
+Remember these points when creating a schema change script:
+  * Create separate change scripts for unrelated schema changes.
+  * Remember to write SQL to //migrate// existing data, if necessary.
+  * Do not use transactional statements - the ''syncdb.py'' script will take
+    care of that.
+To apply your change scripts, just run ''syncdb.py''.  It will look inside the
+''schema_change_log'' table to see which change scripts have already been
+applied, and it will detect your new change script and apply this to the
+database.
+:!: When changing the schema, don't forget to update the Django models in the
+''nav.models'' package.  An integration test exists to verify that the Django
+models can at least be used to run proper SELECTs against the database.
+====== Legacy web code ======
 Legacy web code interfaces directly with
 [[http://www.modpython.org/|mod_python]], and uses
 [[http://www.cheetahtemplate.org/|Cheetah for HTML templating]].
-===== Cheetah templates =====
+All Cheetah templates are located in the ''python/nav/web/templates''
-Global Cheetah templates are found in
+directory.
-''subsystem/webfront/nav/web/templates/'', while local templates are
-located in the subdirectories of their respective subsystems.  All
-Python modules compiled from Cheetah templates should be installed in the
-''nav.web.templates'' package.
-===== Database connections =====
+===== Legacy database connections in web code =====
 Use the ''nav.db.getConnection()'' call to open or retrieve an
 existing database connection.  All NAV web modules share the same
@@ Line 195: / Line 252: @@
     committed//**, i.e. no autocommits.  Be careful to commit the
     current transaction if you modify any data.  A mod_python
-    cleanuphandler will try to automatically commit all open
+    ''cleanuphandler'' will try to automatically commit all open
     transactions as the request cycle ends, but this may change
     in the future, so you must not rely on it.
@@ Line 201: / Line 258: @@
     restoring it to its original value before the end of the request
     cycle.
+===== The "death" of mod_python =====
+''mod_python'' is no longer under active development and has been
+placed in the Apache foundation's "Attic".  We do not accept new web
+tools that interface directly with ''mod_python''.
+We do, however, aim to refactor existing mod_python-interfacing code
+into working as Django views.  A few tips for such refactorings:
+  * Each ''mod_python'' handler in NAV mostly performs its own custom
+    URL parsing and view dispatch.  It's best to refactor this into a
+    Django URL configuration and separate view functions first.
+  * Usage of ''mod_python.utils.FieldStorage'' parse URI arguments
+    must be refactored to use the ''POST'', ''GET'' or ''REQUEST''
+    objects of a Django ''HttpRequest''. It's not that hard, as these
+    objects behave like dictionaries, much like the ''FieldStorage''
+    class does.
+  * Conversion from Cheetah to Django templates is not necessary to
+    refactor a mod_python handler into a Django view.  It is desirable
+    to do so in later refactorings, though.
+  * NAV's authentication and authorization scheme hooks into Apache's
+    request cycle using a ''mod_python'' ''headerparserhandler''. It
+    also adds session data as an attribute to the ''mod_python''
+    request object.  Once there are no tools left that interface
+    directly with ''mod_python'', the auth and session parts of NAV
+    must be refactored to work in a pure Django setting before NAV can
+    be free of its dependence on ''mod_python''.
 ====== Writing new web code ======
 If you are writing a new web application / tool for NAV, please use
 the Django framework.  [[devel:django_introduction|Here's a quick
-primer]].
+primer on how Django integrates with legacy NAV]].
+====== Version Control ======
+NAV uses [[http://www.selenic.com/mercurial/|Mercurial]] for
+distributed version control.  Official repositories are located at
+http://nav.uninett.no/hg/ .
+===== Guide to the repository jungle =====
+The official repositories represent three types of branches
+==== Unstable (default) ====
+New, bleeding edge development occurs on the
+//[[http://nav.uninett.no/hg/default/|default]]// branch, which is
+considered unstable (although we try to always keep it buildable).
+==== Feature branches ====
+New features that take a while (and a lot of changesets) to implement
+and test will often be published as separate feature branches.  For
+all intents and purposes, the feature branches will look like the
+//default// branch with some added feature.  They will merge changes
+from the //default// branch regularly.  Once a feature is considered
+"ready", the feature branch will be merged onto the default branch.
+==== Stable (series) ====
+Once we are nearing a new series release of NAV (such as 3.5 or 3.6),
+a new [[http://nav.uninett.no/hg/series/|series branch]] is
+created from the //default// branch.  Once this branch is stabilized,
+the first version is tagged and released.  After this point, we accept
+only bug fixes in this branch.  Further point releases in this series
+are tagged on this branch, and all changes are merged back onto the
+//default// branch.
+When someone writes a patch for a bug, this should usually be
+committed to the latest active series branch which is affected by the
+bug.  Once a new series is released, we do not usually maintain the
+older series branches.  We may push bug fixes to these branches, but
+we are unlikely to create a new point release from it.
+===== Push access =====
+Push access to the official repositories is limited to developers
+employed or commisioned by UNINETT.
+====== Testing and Continuous Integration ======
+Much of NAV is **legacy code**, as defined by //Michael C. Feathers//:
+Code that has no tests.  We have been making an effort to introduce
+automated tests into the codebase the past couple of years, and hope
+to improve coverage in time.
+There are no tests for the legacy Java code, but many unit tests and
+integration tests now reside in the ''tests/'' subdirectory.
+===== Running tests =====
+We use ''[[http://pytest.org/|py.test]]'' to run the test suite.  A bundled version is
+included as ''runtests.py'' in the ''python/'' subdirectory, which is
+used to run the unit tests only when a ''make check'' command is
+issued in the ''python/'' subdirectory.
+Some of the test requirements aren't available on the Debian systems
+we use for development, so we often test inside a Python
+//virtualenv//.  A suitable virtualenv for testing (on Debian Lenny)
+can be created thus:
+<code bash>
+virtualenv .env
+. .env/bin/activate
+easy_install pip
+pip install -r tests/requirements.txt
+</code>
+There's also a script to create a test environment, complete with
+database initialization.  This is used by our CI server.  The
+following will configure and build NAV automatically, and install it
+into a directory called ''workspace/build''.  It will also create a
+suitable virtualenv in ''workspace/.env'', which you can activate
+before running tests:
+<code bash>
+export PGDATABASE=testdb
+export PGUSER=testuser
+tests/bootstrap-test-environment.sh workspace
+</code>
+===== Javascript testing =====
+Testing of javascript is in its infancy in NAV. We are currently using [[http://busterjs.org/|buster.js]] as testing toolkit.
+To install buster.js install [[http://nodejs.org/|node]] and then:
+<code>
+npm install -g buster
+</code>
+As we use [[http://requirejs.org/|require.js]] you need the AMD module of buster aswell. Install it in the /media/js directory:
+<code>
+npm install buster-amd
+</code>
+To run the tests you need to
+  - Start a buster server by typing ''buster-server''
+  - Capture browsers by pointing browsers to the buster-server (default localhost:1111)
+  - Go to /media/js
+  - Run the tests by typing ''buster-test''
+All tests are located under ''media/js/tests/''. Create new tests there. For syntax, assertions and related stuff take a look at the tests already there and [[http://busterjs.org/docs/|the buster docs]].
+===== Jenkins =====
+We use //Jenkins// (formerly //Hudson//) for Continuous Integration testing of
+NAV.  All the automated tests are run each time new changesets are pushed to
+the NAV repositories.  Jenkins also runs pylint to create stats on code
+quality.
+Our Jenkins installation is available on https://ci.nav.uninett.no/ .
+===== Tips and tricks =====
+===== Make fixtures for integration testing =====
+<code>
+from django.core import serializers
+from nav.models.manage import Netbox
+fixtures = serializers.serialize("xml", Netbox.objects.all()[:2])
+</code>
+Fixtures can so be used in your integration tests by extending
+the test case DjangoTransactionTestCase in nav.tests.cases
+See nav.tests.integration.l2trace_test for an example on applying
+fixtures for your particular test case.
+Also keep in mind you have to make sure you have the model
+dependency in correct order when importing.
+Example: Netbox contains a location to a Room where it is located,
+you have to make sure Room's are imported first before importing
+Netbox's
+See https://docs.djangoproject.com/en/dev/topics/serialization/
+TODO: Be able to use [[https://docs.djangoproject.com/en/dev/ref/django-admin/#dumpdata-appname-appname-appname-model|django-admin's management command: dumpdata]]
+to create fixtures.
+====== Submitting patches ======
+Unless you are submitting one-off fixes for bugs and small issues,
+please take the time to discuss your change proposals on the
+//nav-dev// mailing list.  This will increase the chances of having
+your patches accepted.
+Base your patches on the relevant Mercurial branches.  If you are
+submitting a patch for an issue that affects the latest stable series,
+base your patch on that series branch.  If you are submitting patches
+containing new features, base them on the default branch.
+There are three common options for submitting patches:
+  * The best way to submit your patches would be using Mercurial. Publish
+    your own Mercurial branch, and mail its URL to the //nav-dev// mailing
+    list.
+  * If unable to host a public Mercurial branch, export your changes
+    as a Mercurial bundle and attach it to an email addressed to the
+    //nav-dev// mailing list.
+  * If you have a single patch to submit, attach it to an email
+    addressed to the //nav-dev// mailing list.  Please **do not
+    patchbomb** the mailing list with multiple emails.

NAV Wiki

User Tools

Site Tools

Differences

Page Tools