This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
devel:hacking [2009/05/27 08:52] morten |
devel:hacking [2012/09/18 08:25] bredal [Python boilerplate headers] |
||
---|---|---|---|
Line 18: | Line 18: | ||
We communicate mainly through mailing lists, | We communicate mainly through mailing lists, | ||
- | [[https://launchpad.net/nav/|Launchpad]], and the wiki and Mercurial | + | [[https://launchpad.net/nav/|Launchpad]] and the ''#nav'' IRC channel |
- | repositories hosted at http://metanav.uninett.no/ . At times, UNINETT | + | on //FreeNode//. At times, UNINETT also arranges workshops and |
- | also arranges workshops and gatherings for its customers: Norwegian | + | gatherings for its customers: Norwegian universities, university |
- | universities, university colleges and research institutions. | + | colleges and research institutions. |
To contribute: | To contribute: | ||
- | Go to http://metanav.uninett.no/ and | + | Go to http://nav.uninett.no/ and |
* Join the mailing lists. The //nav-dev// mailing list in | * Join the mailing lists. The //nav-dev// mailing list in | ||
Line 31: | Line 31: | ||
low traffic list. We can only hope this will change ;-) | low traffic list. We can only hope this will change ;-) | ||
* Get a copy of the latest development sources by cloning the | * Get a copy of the latest development sources by cloning the | ||
- | Mercurial repository at http://metanav.uninett.no/hg/default/. | + | Mercurial repository at http://nav.uninett.no/hg/default/. |
Most new development takes place on this branch. | Most new development takes place on this branch. | ||
* Take a look at the [[:navprojects|project reports from previous | * Take a look at the [[:navprojects|project reports from previous | ||
Line 45: | Line 45: | ||
A rough guide to the source tree: | A rough guide to the source tree: | ||
- | | conf/ | Files related to the autoconf build system. | | + | | bin/ | NAV 'binaries'; executable scripts and programs. | |
- | | doc/ | User and developer documentation, SQL scripts and example NAV configuration files. | | + | | contrib/ | User contributed NAV tools. NAV doesn't depend on these, and any maintenance of them is left up to the original developers. We do not offer support for these tools. | |
+ | | doc/ | User and developer documentation | | ||
+ | | etc/ | Example/initial configuration files | | ||
+ | | java/ | Java source code | | ||
+ | | media/ | Static media such as CSS stylesheets, images and JavaScript to be served by a webserver | | ||
+ | | packages/ | Stuff to help packaging NAV for various platforms, such as RedHat, CentOS, FreeBSD, Debian and soforth. Much of this is outdated today. | | ||
+ | | python/ | Python source code | | ||
+ | | sql/ | SQL schema definitions and installation/sync tools | | ||
+ | | templates/ | Django HTML templates | | ||
+ | | tests/ | Automated tests | | ||
| tools/ | Tool scripts for the build and release processes. | | | tools/ | Tool scripts for the build and release processes. | | ||
- | | contrib/ | User contributed NAV tools. NAV doesn't depend on these, and any maintenance of them is left up to the original developers. We do not offer support for these tools. | | ||
- | | packages/ | Stuff to help packaging NAV for various platforms, such as RedHat, CentOS, FreeBSD, Debian and soforth. | | ||
- | | src/ | Source code to Java subsystems of NAV (here for historic reasons). | | ||
- | | subsystem/ | Source code to the rest of NAV - most of it Python. NAV is loosely divided into subsystems, and each one of these has its own subdirectory in here. | | ||
- | | subsystem/lib-python/ | Python libraries & APIs. Please check what's already there before you roll your own. | | ||
- | | subsystem/lib-perl/ | Perl libraries & APIs. Only a single piece of Perl code remains in NAV; once this has been replaced, this directory will cease to exist. | | ||
- | | subsystem/webfront/ | Python libraries for the web interface and front-page handler modules for mod_python. | | ||
- | |||
====== Development languages and frameworks ====== | ====== Development languages and frameworks ====== | ||
- | For historic reasons, different parts of NAV are written in different | ||
- | programming languages (Perl, Java, PHP and Python). This has been | ||
- | unfortunate in may ways, not at least for the sake of consistency and | ||
- | maintenance, but fortunately we have a long-term goal of reducing the | ||
- | number of languages and dependencies. The last few years we've spent | ||
- | a significant amount of time rewriting parts that were written in Perl | ||
- | and PHP to Python, which is the language we are currently gravitating | ||
- | towards. | ||
- | Currently (as of February 2009), major parts of NAV are written in | + | Historically, NAV was written using multiple programming languages |
- | Python and Java, while only a single Perl program remains | + | (Perl, Java, PHP and Python). While this has had an unfortunate |
- | (''makecricketconfig.pl''). | + | impact on integration and maintenance over the years, we've managed to |
+ | reduce this to just Python and Java in later years. We have a | ||
+ | long-term goal to rewrite the remaining Java backend code to Python. | ||
- | * When contributing patches to existing code, or plugins to existing | + | Currently (as of September 2012), NAV consists mostly of Python code, |
- | subsystems, use the language that subsystem was written in. | + | with one remaining backend systems written in Java (eventEngine). |
- | * When writing entirely new subsystems, the following rules apply: | + | |
- | - If your subsystem is a new tool for the web interface, use | + | |
- | Python and Django. The legacy parts of the web system interface | + | |
- | directly with ''mod_python'', using Cheetah for HTML templating. | + | |
- | Anything written in 2008 and later uses Django. There is also | + | |
- | [[devel:django_introduction|a guide for interfacing Django | + | |
- | applications with the legacy web code]]. | + | |
- | - If your subsystem is a new back-end tool/daemon, please use | + | |
- | Python. The NAV Python API is more complete than for any of the | + | |
- | other languages, and you will receive a lot for free. | + | |
- | If these rules are not followed, your patches will not be accepted | + | * We will only accept new code written in Python (except when it |
- | into NAV (but if they are really good, we will consider including them | + | involves patches to the existing Java code). |
- | in the //contrib// directory). | + | * When you contribute additions to the web interface, use the Django framework. |
- | If **YOU** are willing to invest in porting some of the existing | + | If you wish to contribute something really useful that doesn't use |
- | Java/Perl code to Python, then you will be celebrated as a NAV hero! | + | Python, we may consider including it in the //contrib// directory. |
+ | If **YOU** are willing to invest in porting some of the existing Java | ||
+ | code to Python, then you will be celebrated as a NAV hero! | ||
====== Coding style ====== | ====== Coding style ====== | ||
Line 138: | Line 125: | ||
</code> | </code> | ||
- | ====== Database connections ====== | + | ===== Headline ===== |
+ | |||
+ | ====== Database ====== | ||
NAV uses PostgreSQL as its database backend. Namespaces (schemas) are | NAV uses PostgreSQL as its database backend. Namespaces (schemas) are | ||
employed to logically group tables and relations. NAV versions prior | employed to logically group tables and relations. NAV versions prior | ||
Line 149: | Line 138: | ||
| ''logger'' | Anything related to NAV's syslog parser/browser system. | | | ''logger'' | Anything related to NAV's syslog parser/browser system. | | ||
| ''arnold'' | The port detention system Arnold stores it's data here. | | | ''arnold'' | The port detention system Arnold stores it's data here. | | ||
+ | | ''radius'' | Radius accounting logs, updated directly by FreeRadius' PostgreSQL module. | | ||
- | **NOTE**: The following is Python-specific, more info should be added for | + | ===== Connecting to the database (Python) ==== |
- | the other languages used in NAV. | + | ==== Raw SQL ==== |
- | + | ||
- | + | ||
- | ===== Raw SQL ===== | + | |
To obtain a connection to the NAV database, use the API accordingly, | To obtain a connection to the NAV database, use the API accordingly, | ||
Line 175: | Line 162: | ||
database user for a subsystem called ''default'', and also specifies | database user for a subsystem called ''default'', and also specifies | ||
the same database user for all known subsystem names. At present, | the same database user for all known subsystem names. At present, | ||
- | using a subsystem name that is not configured in ''db.conf'' will | + | using a subsystem name that is not configured in ''db.conf'' will cause ''nav.db.getConnection()'' to revert to using the ''default'' name. |
- | raise an exception in ''nav.db.getConnection()''. | + | |
- | ===== Django models ===== | + | ==== Django models ==== |
- | NAV 3.5 and on includes Django models for the most widely used | + | NAV 3.5 and on includes Django models for most database tables. If no |
- | database tables. If no SQL magic is needed to perform your database | + | SQL magic is needed to perform your database voodoo, it is recommended |
- | voodoo, it is recommended that you use these models, located in the | + | that you use these models, located in the module ''nav.models''. You |
- | module ''nav.models''. You do not need to explicitly establish a db | + | do not need to explicitly establish a database connection to use these |
- | connection to use these models, as Django takes care of all that. | + | models, as Django takes care of all that. |
+ | The models are defined in modules of the ''nav.models'' package. | ||
- | ====== Legacy web code ====== | + | ===== Changing the schema ==== |
- | When making changes to NAV's legacy web code, a few special | + | |
- | considerations need to be made. | + | |
+ | The baseline schema is located in ''sql/baseline'' - the ''syncdb.py'' script | ||
+ | is responsible for running this when creating a new database. To make a schema | ||
+ | change, you **do not** change the baseline, but go to the ''sql/changes'' | ||
+ | directory and create a new schema change script there. | ||
+ | |||
+ | Schema change scripts as numbered, using the following pattern: | ||
+ | |||
+ | * ''sc.<major>.<minor>.<point>.sql'' | ||
+ | |||
+ | The ''<major>'' and ''<minor>'' numbers usually correspond to the major and | ||
+ | minor number of the next NAV release. The ''<point>'' number is a sequence id | ||
+ | - pick the next free number when creating a schema change script. | ||
+ | |||
+ | Remember these points when creating a schema change script: | ||
+ | |||
+ | * Create separate change scripts for unrelated schema changes. | ||
+ | * Remember to write SQL to //migrate// existing data, if necessary. | ||
+ | * Do not use transactional statements - the ''syncdb.py'' script will take | ||
+ | care of that. | ||
+ | |||
+ | To apply your change scripts, just run ''syncdb.py''. It will look inside the | ||
+ | ''schema_change_log'' table to see which change scripts have already been | ||
+ | applied, and it will detect your new change script and apply this to the | ||
+ | database. | ||
+ | |||
+ | :!: When changing the schema, don't forget to update the Django models in the | ||
+ | ''nav.models'' package. An integration test exists to verify that the Django | ||
+ | models can at least be used to run proper SELECTs against the database. | ||
+ | |||
+ | ====== Legacy web code ====== | ||
Legacy web code interfaces directly with | Legacy web code interfaces directly with | ||
[[http://www.modpython.org/|mod_python]], and uses | [[http://www.modpython.org/|mod_python]], and uses | ||
[[http://www.cheetahtemplate.org/|Cheetah for HTML templating]]. | [[http://www.cheetahtemplate.org/|Cheetah for HTML templating]]. | ||
- | ===== Cheetah templates ===== | + | All Cheetah templates are located in the ''python/nav/web/templates'' |
- | Global Cheetah templates are found in | + | directory. |
- | ''subsystem/webfront/nav/web/templates/'', while local templates are | + | |
- | located in the subdirectories of their respective subsystems. All | + | |
- | Python modules compiled from Cheetah templates should be installed in the | + | |
- | ''nav.web.templates'' package. | + | |
- | + | ===== Legacy database connections in web code ===== | |
- | ===== Database connections ===== | + | |
Use the ''nav.db.getConnection()'' call to open or retrieve an | Use the ''nav.db.getConnection()'' call to open or retrieve an | ||
existing database connection. All NAV web modules share the same | existing database connection. All NAV web modules share the same | ||
Line 224: | Line 234: | ||
committed//**, i.e. no autocommits. Be careful to commit the | committed//**, i.e. no autocommits. Be careful to commit the | ||
current transaction if you modify any data. A mod_python | current transaction if you modify any data. A mod_python | ||
- | cleanuphandler will try to automatically commit all open | + | ''cleanuphandler'' will try to automatically commit all open |
transactions as the request cycle ends, but this may change | transactions as the request cycle ends, but this may change | ||
in the future, so you must not rely on it. | in the future, so you must not rely on it. | ||
Line 230: | Line 240: | ||
restoring it to its original value before the end of the request | restoring it to its original value before the end of the request | ||
cycle. | cycle. | ||
+ | |||
+ | ===== The "death" of mod_python ===== | ||
+ | ''mod_python'' is no longer under active development and has been | ||
+ | placed in the Apache foundation's "Attic". We do not accept new web | ||
+ | tools that interface directly with ''mod_python''. | ||
+ | |||
+ | We do, however, aim to refactor existing mod_python-interfacing code | ||
+ | into working as Django views. A few tips for such refactorings: | ||
+ | |||
+ | * Each ''mod_python'' handler in NAV mostly performs its own custom | ||
+ | URL parsing and view dispatch. It's best to refactor this into a | ||
+ | Django URL configuration and separate view functions first. | ||
+ | * Usage of ''mod_python.utils.FieldStorage'' parse URI arguments | ||
+ | must be refactored to use the ''POST'', ''GET'' or ''REQUEST'' | ||
+ | objects of a Django ''HttpRequest''. It's not that hard, as these | ||
+ | objects behave like dictionaries, much like the ''FieldStorage'' | ||
+ | class does. | ||
+ | * Conversion from Cheetah to Django templates is not necessary to | ||
+ | refactor a mod_python handler into a Django view. It is desirable | ||
+ | to do so in later refactorings, though. | ||
+ | * NAV's authentication and authorization scheme hooks into Apache's | ||
+ | request cycle using a ''mod_python'' ''headerparserhandler''. It | ||
+ | also adds session data as an attribute to the ''mod_python'' | ||
+ | request object. Once there are no tools left that interface | ||
+ | directly with ''mod_python'', the auth and session parts of NAV | ||
+ | must be refactored to work in a pure Django setting before NAV can | ||
+ | be free of its dependence on ''mod_python''. | ||
====== Writing new web code ====== | ====== Writing new web code ====== | ||
If you are writing a new web application / tool for NAV, please use | If you are writing a new web application / tool for NAV, please use | ||
the Django framework. [[devel:django_introduction|Here's a quick | the Django framework. [[devel:django_introduction|Here's a quick | ||
- | primer]]. | + | primer on how Django integrates with legacy NAV]]. |
====== Version Control ====== | ====== Version Control ====== | ||
NAV uses [[http://www.selenic.com/mercurial/|Mercurial]] for | NAV uses [[http://www.selenic.com/mercurial/|Mercurial]] for | ||
distributed version control. Official repositories are located at | distributed version control. Official repositories are located at | ||
- | http://metanav.uninett.no/hg/ . | + | http://nav.uninett.no/hg/ . |
===== Guide to the repository jungle ===== | ===== Guide to the repository jungle ===== | ||
Line 246: | Line 283: | ||
==== Unstable (default) ==== | ==== Unstable (default) ==== | ||
New, bleeding edge development occurs on the | New, bleeding edge development occurs on the | ||
- | //[[http://metanav.uninett.no/hg/default/|default]]// branch, which is | + | //[[http://nav.uninett.no/hg/default/|default]]// branch, which is |
considered unstable (although we try to always keep it buildable). | considered unstable (although we try to always keep it buildable). | ||
Line 259: | Line 296: | ||
==== Stable (series) ==== | ==== Stable (series) ==== | ||
Once we are nearing a new series release of NAV (such as 3.5 or 3.6), | Once we are nearing a new series release of NAV (such as 3.5 or 3.6), | ||
- | a new [[http://metanav.uninett.no/hg/series/|series branch]] is | + | a new [[http://nav.uninett.no/hg/series/|series branch]] is |
created from the //default// branch. Once this branch is stabilized, | created from the //default// branch. Once this branch is stabilized, | ||
the first version is tagged and released. After this point, we accept | the first version is tagged and released. After this point, we accept | ||
Line 275: | Line 312: | ||
Push access to the official repositories is limited to developers | Push access to the official repositories is limited to developers | ||
employed or commisioned by UNINETT. | employed or commisioned by UNINETT. | ||
+ | |||
+ | ====== Testing and Continuous Integration ====== | ||
+ | Much of NAV is **legacy code**, as defined by //Michael C. Feathers//: | ||
+ | Code that has no tests. We have been making an effort to introduce | ||
+ | automated tests into the codebase the past couple of years, and hope | ||
+ | to improve coverage in time. | ||
+ | |||
+ | There are no tests for the legacy Java code, but many unit tests and | ||
+ | integration tests now reside in the ''tests/'' subdirectory. | ||
+ | |||
+ | ===== Running tests ===== | ||
+ | We use ''[[http://pytest.org/|py.test]]'' to run the test suite. A bundled version is | ||
+ | included as ''runtests.py'' in the ''python/'' subdirectory, which is | ||
+ | used to run the unit tests only when a ''make check'' command is | ||
+ | issued in the ''python/'' subdirectory. | ||
+ | |||
+ | Some of the test requirements aren't available on the Debian systems | ||
+ | we use for development, so we often test inside a Python | ||
+ | //virtualenv//. A suitable virtualenv for testing (on Debian Lenny) | ||
+ | can be created thus: | ||
+ | |||
+ | <code bash> | ||
+ | virtualenv .env | ||
+ | . .env/bin/activate | ||
+ | easy_install pip | ||
+ | pip install -r tests/requirements.txt | ||
+ | </code> | ||
+ | |||
+ | There's also a script to create a test environment, complete with | ||
+ | database initialization. This is used by our CI server. The | ||
+ | following will configure and build NAV automatically, and install it | ||
+ | into a directory called ''workspace/build''. It will also create a | ||
+ | suitable virtualenv in ''workspace/.env'', which you can activate | ||
+ | before running tests: | ||
+ | |||
+ | <code bash> | ||
+ | export PGDATABASE=testdb | ||
+ | export PGUSER=testuser | ||
+ | tests/bootstrap-test-environment.sh workspace | ||
+ | </code> | ||
+ | |||
+ | |||
+ | ===== Javascript testing ===== | ||
+ | |||
+ | Testing of javascript is in its infancy in NAV. We are currently using [[http://busterjs.org/|buster.js]] as testing toolkit. | ||
+ | |||
+ | To install buster.js install [[http://nodejs.org/|node]] and then: | ||
+ | <code> | ||
+ | npm install -g buster | ||
+ | </code> | ||
+ | |||
+ | As we use [[http://requirejs.org/|require.js]] you need the AMD module of buster aswell. Install it in the /media/js directory: | ||
+ | <code> | ||
+ | npm install buster-amd | ||
+ | </code> | ||
+ | |||
+ | To run the tests you need to | ||
+ | - Start a buster server by typing ''buster-server'' | ||
+ | - Capture browsers by pointing browsers to the buster-server (default localhost:1111) | ||
+ | - Go to /media/js | ||
+ | - Run the tests by typing ''buster-test'' | ||
+ | |||
+ | All tests are located under ''media/js/tests/''. Create new tests there. For syntax, assertions and related stuff take a look at the tests already there and [[http://busterjs.org/docs/|the buster docs]]. | ||
+ | ===== Jenkins ===== | ||
+ | |||
+ | We use //Jenkins// (formerly //Hudson//) for Continuous Integration testing of | ||
+ | NAV. All the automated tests are run each time new changesets are pushed to | ||
+ | the NAV repositories. Jenkins also runs pylint to create stats on code | ||
+ | quality. | ||
+ | |||
+ | Our Jenkins installation is available on http://nav.uninett.no/jenkins . | ||
+ | |||
+ | ===== Tips and tricks ===== | ||
+ | |||
+ | ===== Make fixtures for integration testing ===== | ||
+ | |||
+ | <code> | ||
+ | from django.core import serializers | ||
+ | from nav.models.manage import Netbox | ||
+ | |||
+ | fixtures = serializers.serialize("xml", Netbox.objects.all()[:2]) | ||
+ | </code> | ||
+ | |||
+ | Fixtures can so be used in your integration tests by extending | ||
+ | the test case DjangoTransactionTestCase in nav.tests.cases | ||
+ | |||
+ | See nav.tests.integration.l2trace_test for an example on applying | ||
+ | fixtures for your particular test case. | ||
+ | |||
+ | Also keep in mind you have to make sure you have the model | ||
+ | dependency in correct order when importing. | ||
+ | Example: Netbox contains a location to a Room where it is located, | ||
+ | you have to make sure Room's are imported first before importing | ||
+ | Netbox's | ||
+ | |||
+ | See https://docs.djangoproject.com/en/dev/topics/serialization/ | ||
+ | |||
+ | TODO: Be able to use [[https://docs.djangoproject.com/en/dev/ref/django-admin/#dumpdata-appname-appname-appname-model|django-admin's management command: dumpdata]] | ||
+ | to create fixtures. | ||
+ | |||
+ | |||
====== Submitting patches ====== | ====== Submitting patches ====== | ||
Line 298: | Line 436: | ||
addressed to the //nav-dev// mailing list. Please **do not | addressed to the //nav-dev// mailing list. Please **do not | ||
patchbomb** the mailing list with multiple emails. | patchbomb** the mailing list with multiple emails. | ||
- |