Differences

This shows you the differences between two versions of the page.

--- devel:hacking [2009/05/27 08:53]
morten add radius to list of namespaces
+++ devel:hacking [2013/06/19 07:45]
morten fix jenkins url
@@ Line 18: / Line 18: @@
 We communicate mainly through mailing lists,
-[[https://launchpad.net/nav/|Launchpad]], and the wiki and Mercurial
+[[https://launchpad.net/nav/|Launchpad]] and the ''#nav'' IRC channel
-repositories hosted at http://metanav.uninett.no/ .  At times, UNINETT
+on //FreeNode//.  At times, UNINETT also arranges workshops and
-also arranges workshops and gatherings for its customers: Norwegian
+gatherings for its customers: Norwegian universities, university
-universities, university colleges and research institutions.
+colleges and research institutions.
 To contribute:
-Go to http://metanav.uninett.no/ and
+Go to http://nav.uninett.no/ and
   * Join the mailing lists.  The //nav-dev// mailing list in
@@ Line 31: / Line 31: @@
     low traffic list. We can only hope this will change ;-)
   * Get a copy of the latest development sources by cloning the
-    Mercurial repository at http://metanav.uninett.no/hg/default/.
+    Mercurial repository at http://nav.uninett.no/hg/default/.
     Most new development takes place on this branch.
   * Take a look at the [[:navprojects|project reports from previous
@@ Line 45: / Line 45: @@
 A rough guide to the source tree:
-| conf/      | Files related to the autoconf build system. |
+| bin/       | NAV 'binaries'; executable scripts and programs. |
-| doc/       | User and developer documentation, SQL scripts and example NAV configuration files. |
+| contrib/   | User contributed NAV tools. NAV doesn't depend on these, and any maintenance of them is left up to the original developers.  We do not offer support for these tools. |
+| doc/       | User and developer documentation |
+| etc/       | Example/initial configuration files |
+| java/      | Java source code |
+| media/     | Static media such as CSS stylesheets, images and JavaScript to be served by a webserver |
+| packages/  | Stuff to help packaging NAV for various platforms, such as RedHat, CentOS, FreeBSD, Debian and soforth. Much of this is outdated today. |
+| python/    | Python source code |
+| sql/       | SQL schema definitions and installation/sync tools |
+| templates/ | Django HTML templates |
+| tests/     | Automated tests |
 | tools/     | Tool scripts for the build and release processes. |
-| contrib/   | User contributed NAV tools. NAV doesn't depend on these, and any maintenance of them is left up to the original developers.  We do not offer support for these tools. |
-| packages/  | Stuff to help packaging NAV for various platforms, such as RedHat, CentOS, FreeBSD, Debian and soforth. |
-| src/       | Source code to Java subsystems of NAV (here for historic reasons). |
-| subsystem/ | Source code to the rest of NAV - most of it Python.  NAV is loosely divided into subsystems, and each one of these has its own subdirectory in here. |
-| subsystem/lib-python/ | Python libraries & APIs.  Please check what's already there before you roll your own. |
-| subsystem/lib-perl/ | Perl libraries & APIs.  Only a single piece of Perl code remains in NAV; once this has been replaced, this directory will cease to exist. |
-| subsystem/webfront/ | Python libraries for the web interface and front-page handler modules for mod_python. |
 ====== Development languages and frameworks ======
-For historic reasons, different parts of NAV are written in different
-programming languages (Perl, Java, PHP and Python).  This has been
-unfortunate in may ways, not at least for the sake of consistency and
-maintenance, but fortunately we have a long-term goal of reducing the
-number of languages and dependencies.  The last few years we've spent
-a significant amount of time rewriting parts that were written in Perl
-and PHP to Python, which is the language we are currently gravitating
-towards.
-Currently (as of February 2009), major parts of NAV are written in
+Historically, NAV was written using multiple programming languages
-Python and Java, while only a single Perl program remains
+(Perl, Java, PHP and Python).  While this has had an unfortunate
-(''makecricketconfig.pl'').
+impact on integration and maintenance over the years, we've managed to
+reduce this to just Python and Java in later years. We have a
+long-term goal to rewrite the remaining Java backend code to Python.
-  * When contributing patches to existing code, or plugins to existing
+Currently (as of September 2012), NAV consists mostly of Python code,
-    subsystems, use the language that subsystem was written in.
+with one remaining backend systems written in Java (eventEngine).
-  * When writing entirely new subsystems, the following rules apply:
-    - If your subsystem is a new tool for the web interface, use
-      Python and Django.  The legacy parts of the web system interface
-      directly with ''mod_python'', using Cheetah for HTML templating.
-      Anything written in 2008 and later uses Django.  There is also
-      [[devel:django_introduction|a guide for interfacing Django
-      applications with the legacy web code]].
-    - If your subsystem is a new back-end tool/daemon, please use
-      Python.  The NAV Python API is more complete than for any of the
-      other languages, and you will receive a lot for free.
-If these rules are not followed, your patches will not be accepted
+  * We will only accept new code written in Python (except when it
-into NAV (but if they are really good, we will consider including them
+    involves patches to the existing Java code).
-in the //contrib// directory).
+  * When you contribute additions to the web interface, use the Django framework.
-If **YOU** are willing to invest in porting some of the existing
+If you wish to contribute something really useful that doesn't use
-Java/Perl code to Python, then you will be celebrated as a NAV hero!
+Python, we may consider including it in the //contrib// directory.
+If **YOU** are willing to invest in porting some of the existing Java
+code to Python, then you will be celebrated as a NAV hero!
 ====== Coding style ======
@@ Line 138: / Line 125: @@
 </code>
-====== Database connections ======
+===== Javascript =====
+When writing javascript code try to focus on modules not pages. If the code is html-related, it should take selectors or objects as input and concern itself solely about those. This makes for much easier testing and reuse. And of course - write the tests first.
+When the module is done you write a controller for the page that plugs the needed plugins to the page elements. This should fail gracefully if the needed elements are not present.
+NAVs javascript uses [[http://requirejs.org/|require.js]] - use this to create modules and specify dependencies.
+Pro tip is to create ''require_config.dev.js'' in ''media/js/'' and add the following configuration to requirejs:
+<code>require.urlArgs = "bust=" +  (new Date()).getTime();</code> This makes sure your not using cached resources in your browser when developing, which browsers loves to do! See [[http://requirejs.org/docs/api.html#config-urlArgs|config-urlArgs]] in requirejs documentation for «details». The ''require_config.dev.js'' is added in global HG ignore.
+==== Accessing resources with ajax requiring authentication ====
+As your authenticated session might have timed out due to idle (no activity), resources will return 500 Internal Error if you do not supply the important ''//X-NAV-AJAX//'' header on your ajax requests.
+So make sure to include the ''**X-NAV-AJAX**'' header so you will get proper HTTP response code in the reponse from your request.
+NAV has a shortcut for fixing this in ''default.js'' (adds a function in the public namespace NAV) which attaches the required handlers for jQuery by doing:
+<code>NAV.addGlobalAjaxHandlers()</code>
+====== Database ======
 NAV uses PostgreSQL as its database backend.  Namespaces (schemas) are
 employed to logically group tables and relations.  NAV versions prior
@@ Line 151: / Line 158: @@
 | ''radius''      | Radius accounting logs, updated directly by FreeRadius' PostgreSQL module. |
-**NOTE**: The following is Python-specific, more info should be added for
+===== Connecting to the database (Python) ====
-the other languages used in NAV.
+==== Raw SQL ====
-===== Raw SQL =====
 To obtain a connection to the NAV database, use the API accordingly,
@@ Line 176: / Line 180: @@
 database user for a subsystem called ''default'', and also specifies
 the same database user for all known subsystem names.  At present,
-using a subsystem name that is not configured in ''db.conf'' will
+using a subsystem name that is not configured in ''db.conf'' will cause ''nav.db.getConnection()'' to revert to using the ''default'' name.
-raise an exception in ''nav.db.getConnection()''.
-===== Django models =====
+==== Django models ====
-NAV 3.5 and on includes Django models for the most widely used
+NAV 3.5 and on includes Django models for most database tables.  If no
-database tables.  If no SQL magic is needed to perform your database
+SQL magic is needed to perform your database voodoo, it is recommended
-voodoo, it is recommended that you use these models, located in the
+that you use these models, located in the module ''nav.models''.  You
-module ''nav.models''.  You do not need to explicitly establish a db
+do not need to explicitly establish a database connection to use these
-connection to use these models, as Django takes care of all that.
+models, as Django takes care of all that.
+The models are defined in modules of the ''nav.models'' package.
-====== Legacy web code ======
+===== Changing the schema ====
-When making changes to NAV's legacy web code, a few special
-considerations need to be made.
+The baseline schema is located in ''sql/baseline'' - the ''syncdb.py'' script
+is responsible for running this when creating a new database. To make a schema
+change, you **do not** change the baseline, but go to the ''sql/changes''
+directory and create a new schema change script there.
+Schema change scripts as numbered, using the following pattern:
+  * ''sc.<major>.<minor>.<point>.sql''
+The ''<major>'' and ''<minor>'' numbers usually correspond to the major and
+minor number of the next NAV release.  The ''<point>'' number is a sequence id
+- pick the next free number when creating a schema change script.
+Remember these points when creating a schema change script:
+  * Create separate change scripts for unrelated schema changes.
+  * Remember to write SQL to //migrate// existing data, if necessary.
+  * Do not use transactional statements - the ''syncdb.py'' script will take
+    care of that.
+To apply your change scripts, just run ''syncdb.py''.  It will look inside the
+''schema_change_log'' table to see which change scripts have already been
+applied, and it will detect your new change script and apply this to the
+database.
+:!: When changing the schema, don't forget to update the Django models in the
+''nav.models'' package.  An integration test exists to verify that the Django
+models can at least be used to run proper SELECTs against the database.
+====== Legacy web code ======
 Legacy web code interfaces directly with
 [[http://www.modpython.org/|mod_python]], and uses
 [[http://www.cheetahtemplate.org/|Cheetah for HTML templating]].
-===== Cheetah templates =====
+All Cheetah templates are located in the ''python/nav/web/templates''
-Global Cheetah templates are found in
+directory.
-''subsystem/webfront/nav/web/templates/'', while local templates are
-located in the subdirectories of their respective subsystems.  All
-Python modules compiled from Cheetah templates should be installed in the
-''nav.web.templates'' package.
+===== Legacy database connections in web code =====
-===== Database connections =====
 Use the ''nav.db.getConnection()'' call to open or retrieve an
 existing database connection.  All NAV web modules share the same
@@ Line 225: / Line 252: @@
     committed//**, i.e. no autocommits.  Be careful to commit the
     current transaction if you modify any data.  A mod_python
-    cleanuphandler will try to automatically commit all open
+    ''cleanuphandler'' will try to automatically commit all open
     transactions as the request cycle ends, but this may change
     in the future, so you must not rely on it.
@@ Line 231: / Line 258: @@
     restoring it to its original value before the end of the request
     cycle.
+===== The "death" of mod_python =====
+''mod_python'' is no longer under active development and has been
+placed in the Apache foundation's "Attic".  We do not accept new web
+tools that interface directly with ''mod_python''.
+We do, however, aim to refactor existing mod_python-interfacing code
+into working as Django views.  A few tips for such refactorings:
+  * Each ''mod_python'' handler in NAV mostly performs its own custom
+    URL parsing and view dispatch.  It's best to refactor this into a
+    Django URL configuration and separate view functions first.
+  * Usage of ''mod_python.utils.FieldStorage'' parse URI arguments
+    must be refactored to use the ''POST'', ''GET'' or ''REQUEST''
+    objects of a Django ''HttpRequest''. It's not that hard, as these
+    objects behave like dictionaries, much like the ''FieldStorage''
+    class does.
+  * Conversion from Cheetah to Django templates is not necessary to
+    refactor a mod_python handler into a Django view.  It is desirable
+    to do so in later refactorings, though.
+  * NAV's authentication and authorization scheme hooks into Apache's
+    request cycle using a ''mod_python'' ''headerparserhandler''. It
+    also adds session data as an attribute to the ''mod_python''
+    request object.  Once there are no tools left that interface
+    directly with ''mod_python'', the auth and session parts of NAV
+    must be refactored to work in a pure Django setting before NAV can
+    be free of its dependence on ''mod_python''.
 ====== Writing new web code ======
 If you are writing a new web application / tool for NAV, please use
 the Django framework.  [[devel:django_introduction|Here's a quick
-primer]].
+primer on how Django integrates with legacy NAV]].
 ====== Version Control ======
 NAV uses [[http://www.selenic.com/mercurial/|Mercurial]] for
 distributed version control.  Official repositories are located at
-http://metanav.uninett.no/hg/ .
+http://nav.uninett.no/hg/ .
 ===== Guide to the repository jungle =====
@@ Line 247: / Line 301: @@
 ==== Unstable (default) ====
 New, bleeding edge development occurs on the
-//[[http://metanav.uninett.no/hg/default/|default]]// branch, which is
+//[[http://nav.uninett.no/hg/default/|default]]// branch, which is
 considered unstable (although we try to always keep it buildable).
@@ Line 260: / Line 314: @@
 ==== Stable (series) ====
 Once we are nearing a new series release of NAV (such as 3.5 or 3.6),
-a new [[http://metanav.uninett.no/hg/series/|series branch]] is
+a new [[http://nav.uninett.no/hg/series/|series branch]] is
 created from the //default// branch.  Once this branch is stabilized,
 the first version is tagged and released.  After this point, we accept
@@ Line 276: / Line 330: @@
 Push access to the official repositories is limited to developers
 employed or commisioned by UNINETT.
+====== Testing and Continuous Integration ======
+Much of NAV is **legacy code**, as defined by //Michael C. Feathers//:
+Code that has no tests.  We have been making an effort to introduce
+automated tests into the codebase the past couple of years, and hope
+to improve coverage in time.
+There are no tests for the legacy Java code, but many unit tests and
+integration tests now reside in the ''tests/'' subdirectory.
+===== Running tests =====
+We use ''[[http://pytest.org/|py.test]]'' to run the test suite.  A bundled version is
+included as ''runtests.py'' in the ''python/'' subdirectory, which is
+used to run the unit tests only when a ''make check'' command is
+issued in the ''python/'' subdirectory.
+Some of the test requirements aren't available on the Debian systems
+we use for development, so we often test inside a Python
+//virtualenv//.  A suitable virtualenv for testing (on Debian Lenny)
+can be created thus:
+<code bash>
+virtualenv .env
+. .env/bin/activate
+easy_install pip
+pip install -r tests/requirements.txt
+</code>
+There's also a script to create a test environment, complete with
+database initialization.  This is used by our CI server.  The
+following will configure and build NAV automatically, and install it
+into a directory called ''workspace/build''.  It will also create a
+suitable virtualenv in ''workspace/.env'', which you can activate
+before running tests:
+<code bash>
+export PGDATABASE=testdb
+export PGUSER=testuser
+tests/bootstrap-test-environment.sh workspace
+</code>
+===== Javascript testing =====
+Testing of javascript is in its infancy in NAV. We are currently using [[http://busterjs.org/|buster.js]] as testing toolkit.
+To install buster.js install [[http://nodejs.org/|node]] and then:
+<code>
+npm install -g buster
+</code>
+As we use [[http://requirejs.org/|require.js]] you need the AMD module of buster aswell. Install it in the /media/js directory:
+<code>
+npm install buster-amd
+</code>
+To run the tests you need to
+  - Start a buster server by typing ''buster-server''
+  - Capture browsers by pointing browsers to the buster-server (default localhost:1111)
+  - Go to /media/js
+  - Run the tests by typing ''buster-test''
+All tests are located under ''media/js/tests/''. Create new tests there. For syntax, assertions and related stuff take a look at the tests already there and [[http://busterjs.org/docs/|the buster docs]].
+===== Jenkins =====
+We use //Jenkins// (formerly //Hudson//) for Continuous Integration testing of
+NAV.  All the automated tests are run each time new changesets are pushed to
+the NAV repositories.  Jenkins also runs pylint to create stats on code
+quality.
+Our Jenkins installation is available on https://ci.nav.uninett.no/ .
+===== Tips and tricks =====
+===== Make fixtures for integration testing =====
+<code>
+from django.core import serializers
+from nav.models.manage import Netbox
+fixtures = serializers.serialize("xml", Netbox.objects.all()[:2])
+</code>
+Fixtures can so be used in your integration tests by extending
+the test case DjangoTransactionTestCase in nav.tests.cases
+See nav.tests.integration.l2trace_test for an example on applying
+fixtures for your particular test case.
+Also keep in mind you have to make sure you have the model
+dependency in correct order when importing.
+Example: Netbox contains a location to a Room where it is located,
+you have to make sure Room's are imported first before importing
+Netbox's
+See https://docs.djangoproject.com/en/dev/topics/serialization/
+TODO: Be able to use [[https://docs.djangoproject.com/en/dev/ref/django-admin/#dumpdata-appname-appname-appname-model|django-admin's management command: dumpdata]]
+to create fixtures.
 ====== Submitting patches ======
@@ Line 299: / Line 454: @@
     addressed to the //nav-dev// mailing list.  Please **do not
     patchbomb** the mailing list with multiple emails.

NAV Wiki

User Tools

Site Tools

Differences

Page Tools