This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
devel:hacking [2011/02/03 12:44] morten api updtate |
devel:hacking [2013/01/30 09:15] norangshol [Javascript] X-NAV-AJAX doc |
||
---|---|---|---|
Line 25: | Line 25: | ||
To contribute: | To contribute: | ||
- | Go to http://metanav.uninett.no/ and | + | Go to http://nav.uninett.no/ and |
* Join the mailing lists. The //nav-dev// mailing list in | * Join the mailing lists. The //nav-dev// mailing list in | ||
Line 31: | Line 31: | ||
low traffic list. We can only hope this will change ;-) | low traffic list. We can only hope this will change ;-) | ||
* Get a copy of the latest development sources by cloning the | * Get a copy of the latest development sources by cloning the | ||
- | Mercurial repository at http://metanav.uninett.no/hg/default/. | + | Mercurial repository at http://nav.uninett.no/hg/default/. |
Most new development takes place on this branch. | Most new development takes place on this branch. | ||
* Take a look at the [[:navprojects|project reports from previous | * Take a look at the [[:navprojects|project reports from previous | ||
Line 66: | Line 66: | ||
long-term goal to rewrite the remaining Java backend code to Python. | long-term goal to rewrite the remaining Java backend code to Python. | ||
- | Currently (as of February 2011), NAV consists mostly of Python code, | + | Currently (as of September 2012), NAV consists mostly of Python code, |
- | with a few remaining backend systems written in Java (there's also the | + | with one remaining backend systems written in Java (eventEngine). |
- | Netmap Java Applet). | + | |
* We will only accept new code written in Python (except when it | * We will only accept new code written in Python (except when it | ||
Line 126: | Line 125: | ||
</code> | </code> | ||
- | ====== Database connections ====== | + | ===== Javascript ===== |
+ | |||
+ | When writing javascript code try to focus on modules not pages. If the code is html-related, it should take selectors or objects as input and concern itself solely about those. This makes for much easier testing and reuse. And of course - write the tests first. | ||
+ | |||
+ | When the module is done you write a controller for the page that plugs the needed plugins to the page elements. This should fail gracefully if the needed elements are not present. | ||
+ | |||
+ | NAVs javascript uses [[http://requirejs.org/|require.js]] - use this to create modules and specify dependencies. | ||
+ | |||
+ | Pro tip is to create ''require_config.dev.js'' in ''media/js/'' and add the following configuration to requirejs: | ||
+ | <code>require.urlArgs = "bust=" + (new Date()).getTime();</code> This makes sure your not using cached resources in your browser when developing, which browsers loves to do! See [[http://requirejs.org/docs/api.html#config-urlArgs|config-urlArgs]] in requirejs documentation for «details». The ''require_config.dev.js'' is added in global HG ignore. | ||
+ | |||
+ | ==== Accessing resources with ajax requiring authentication ==== | ||
+ | |||
+ | As your authenticated session might have timed out due to idle (no activity), resources will return 500 Internal Error if you do not supply the important ''//X-NAV-AJAX//'' header on your ajax requests. | ||
+ | |||
+ | So make sure to include the ''**X-NAV-AJAX**'' header so you will get proper HTTP response code in the reponse from your request. | ||
+ | ====== Database ====== | ||
NAV uses PostgreSQL as its database backend. Namespaces (schemas) are | NAV uses PostgreSQL as its database backend. Namespaces (schemas) are | ||
employed to logically group tables and relations. NAV versions prior | employed to logically group tables and relations. NAV versions prior | ||
Line 139: | Line 154: | ||
| ''radius'' | Radius accounting logs, updated directly by FreeRadius' PostgreSQL module. | | | ''radius'' | Radius accounting logs, updated directly by FreeRadius' PostgreSQL module. | | ||
- | **NOTE**: The following is Python-specific, more info should be added for | + | ===== Connecting to the database (Python) ==== |
- | the other languages used in NAV. | + | ==== Raw SQL ==== |
- | + | ||
- | + | ||
- | ===== Raw SQL ===== | + | |
To obtain a connection to the NAV database, use the API accordingly, | To obtain a connection to the NAV database, use the API accordingly, | ||
Line 166: | Line 178: | ||
using a subsystem name that is not configured in ''db.conf'' will cause ''nav.db.getConnection()'' to revert to using the ''default'' name. | using a subsystem name that is not configured in ''db.conf'' will cause ''nav.db.getConnection()'' to revert to using the ''default'' name. | ||
- | ===== Django models ===== | + | ==== Django models ==== |
- | NAV 3.5 and on includes Django models for the most widely used | + | NAV 3.5 and on includes Django models for most database tables. If no |
- | database tables. If no SQL magic is needed to perform your database | + | SQL magic is needed to perform your database voodoo, it is recommended |
- | voodoo, it is recommended that you use these models, located in the | + | that you use these models, located in the module ''nav.models''. You |
- | module ''nav.models''. You do not need to explicitly establish a db | + | do not need to explicitly establish a database connection to use these |
- | connection to use these models, as Django takes care of all that. | + | models, as Django takes care of all that. |
+ | The models are defined in modules of the ''nav.models'' package. | ||
- | ====== Legacy web code ====== | + | ===== Changing the schema ==== |
- | When making changes to NAV's legacy web code, a few special | + | |
- | considerations need to be made. | + | |
+ | The baseline schema is located in ''sql/baseline'' - the ''syncdb.py'' script | ||
+ | is responsible for running this when creating a new database. To make a schema | ||
+ | change, you **do not** change the baseline, but go to the ''sql/changes'' | ||
+ | directory and create a new schema change script there. | ||
+ | |||
+ | Schema change scripts as numbered, using the following pattern: | ||
+ | |||
+ | * ''sc.<major>.<minor>.<point>.sql'' | ||
+ | |||
+ | The ''<major>'' and ''<minor>'' numbers usually correspond to the major and | ||
+ | minor number of the next NAV release. The ''<point>'' number is a sequence id | ||
+ | - pick the next free number when creating a schema change script. | ||
+ | |||
+ | Remember these points when creating a schema change script: | ||
+ | |||
+ | * Create separate change scripts for unrelated schema changes. | ||
+ | * Remember to write SQL to //migrate// existing data, if necessary. | ||
+ | * Do not use transactional statements - the ''syncdb.py'' script will take | ||
+ | care of that. | ||
+ | |||
+ | To apply your change scripts, just run ''syncdb.py''. It will look inside the | ||
+ | ''schema_change_log'' table to see which change scripts have already been | ||
+ | applied, and it will detect your new change script and apply this to the | ||
+ | database. | ||
+ | |||
+ | :!: When changing the schema, don't forget to update the Django models in the | ||
+ | ''nav.models'' package. An integration test exists to verify that the Django | ||
+ | models can at least be used to run proper SELECTs against the database. | ||
+ | |||
+ | ====== Legacy web code ====== | ||
Legacy web code interfaces directly with | Legacy web code interfaces directly with | ||
[[http://www.modpython.org/|mod_python]], and uses | [[http://www.modpython.org/|mod_python]], and uses | ||
[[http://www.cheetahtemplate.org/|Cheetah for HTML templating]]. | [[http://www.cheetahtemplate.org/|Cheetah for HTML templating]]. | ||
- | ===== Cheetah templates ===== | + | All Cheetah templates are located in the ''python/nav/web/templates'' |
- | Global Cheetah templates are found in | + | directory. |
- | ''subsystem/webfront/nav/web/templates/'', while local templates are | + | |
- | located in the subdirectories of their respective subsystems. All | + | |
- | Python modules compiled from Cheetah templates should be installed in the | + | |
- | ''nav.web.templates'' package. | + | |
- | ===== Database connections ===== | + | ===== Legacy database connections in web code ===== |
Use the ''nav.db.getConnection()'' call to open or retrieve an | Use the ''nav.db.getConnection()'' call to open or retrieve an | ||
existing database connection. All NAV web modules share the same | existing database connection. All NAV web modules share the same | ||
Line 212: | Line 248: | ||
committed//**, i.e. no autocommits. Be careful to commit the | committed//**, i.e. no autocommits. Be careful to commit the | ||
current transaction if you modify any data. A mod_python | current transaction if you modify any data. A mod_python | ||
- | cleanuphandler will try to automatically commit all open | + | ''cleanuphandler'' will try to automatically commit all open |
transactions as the request cycle ends, but this may change | transactions as the request cycle ends, but this may change | ||
in the future, so you must not rely on it. | in the future, so you must not rely on it. | ||
Line 218: | Line 254: | ||
restoring it to its original value before the end of the request | restoring it to its original value before the end of the request | ||
cycle. | cycle. | ||
+ | |||
+ | ===== The "death" of mod_python ===== | ||
+ | ''mod_python'' is no longer under active development and has been | ||
+ | placed in the Apache foundation's "Attic". We do not accept new web | ||
+ | tools that interface directly with ''mod_python''. | ||
+ | |||
+ | We do, however, aim to refactor existing mod_python-interfacing code | ||
+ | into working as Django views. A few tips for such refactorings: | ||
+ | |||
+ | * Each ''mod_python'' handler in NAV mostly performs its own custom | ||
+ | URL parsing and view dispatch. It's best to refactor this into a | ||
+ | Django URL configuration and separate view functions first. | ||
+ | * Usage of ''mod_python.utils.FieldStorage'' parse URI arguments | ||
+ | must be refactored to use the ''POST'', ''GET'' or ''REQUEST'' | ||
+ | objects of a Django ''HttpRequest''. It's not that hard, as these | ||
+ | objects behave like dictionaries, much like the ''FieldStorage'' | ||
+ | class does. | ||
+ | * Conversion from Cheetah to Django templates is not necessary to | ||
+ | refactor a mod_python handler into a Django view. It is desirable | ||
+ | to do so in later refactorings, though. | ||
+ | * NAV's authentication and authorization scheme hooks into Apache's | ||
+ | request cycle using a ''mod_python'' ''headerparserhandler''. It | ||
+ | also adds session data as an attribute to the ''mod_python'' | ||
+ | request object. Once there are no tools left that interface | ||
+ | directly with ''mod_python'', the auth and session parts of NAV | ||
+ | must be refactored to work in a pure Django setting before NAV can | ||
+ | be free of its dependence on ''mod_python''. | ||
====== Writing new web code ====== | ====== Writing new web code ====== | ||
If you are writing a new web application / tool for NAV, please use | If you are writing a new web application / tool for NAV, please use | ||
the Django framework. [[devel:django_introduction|Here's a quick | the Django framework. [[devel:django_introduction|Here's a quick | ||
- | primer]]. | + | primer on how Django integrates with legacy NAV]]. |
====== Version Control ====== | ====== Version Control ====== | ||
NAV uses [[http://www.selenic.com/mercurial/|Mercurial]] for | NAV uses [[http://www.selenic.com/mercurial/|Mercurial]] for | ||
distributed version control. Official repositories are located at | distributed version control. Official repositories are located at | ||
- | http://metanav.uninett.no/hg/ . | + | http://nav.uninett.no/hg/ . |
===== Guide to the repository jungle ===== | ===== Guide to the repository jungle ===== | ||
Line 234: | Line 297: | ||
==== Unstable (default) ==== | ==== Unstable (default) ==== | ||
New, bleeding edge development occurs on the | New, bleeding edge development occurs on the | ||
- | //[[http://metanav.uninett.no/hg/default/|default]]// branch, which is | + | //[[http://nav.uninett.no/hg/default/|default]]// branch, which is |
considered unstable (although we try to always keep it buildable). | considered unstable (although we try to always keep it buildable). | ||
Line 247: | Line 310: | ||
==== Stable (series) ==== | ==== Stable (series) ==== | ||
Once we are nearing a new series release of NAV (such as 3.5 or 3.6), | Once we are nearing a new series release of NAV (such as 3.5 or 3.6), | ||
- | a new [[http://metanav.uninett.no/hg/series/|series branch]] is | + | a new [[http://nav.uninett.no/hg/series/|series branch]] is |
created from the //default// branch. Once this branch is stabilized, | created from the //default// branch. Once this branch is stabilized, | ||
the first version is tagged and released. After this point, we accept | the first version is tagged and released. After this point, we accept | ||
Line 263: | Line 326: | ||
Push access to the official repositories is limited to developers | Push access to the official repositories is limited to developers | ||
employed or commisioned by UNINETT. | employed or commisioned by UNINETT. | ||
+ | |||
+ | ====== Testing and Continuous Integration ====== | ||
+ | Much of NAV is **legacy code**, as defined by //Michael C. Feathers//: | ||
+ | Code that has no tests. We have been making an effort to introduce | ||
+ | automated tests into the codebase the past couple of years, and hope | ||
+ | to improve coverage in time. | ||
+ | |||
+ | There are no tests for the legacy Java code, but many unit tests and | ||
+ | integration tests now reside in the ''tests/'' subdirectory. | ||
+ | |||
+ | ===== Running tests ===== | ||
+ | We use ''[[http://pytest.org/|py.test]]'' to run the test suite. A bundled version is | ||
+ | included as ''runtests.py'' in the ''python/'' subdirectory, which is | ||
+ | used to run the unit tests only when a ''make check'' command is | ||
+ | issued in the ''python/'' subdirectory. | ||
+ | |||
+ | Some of the test requirements aren't available on the Debian systems | ||
+ | we use for development, so we often test inside a Python | ||
+ | //virtualenv//. A suitable virtualenv for testing (on Debian Lenny) | ||
+ | can be created thus: | ||
+ | |||
+ | <code bash> | ||
+ | virtualenv .env | ||
+ | . .env/bin/activate | ||
+ | easy_install pip | ||
+ | pip install -r tests/requirements.txt | ||
+ | </code> | ||
+ | |||
+ | There's also a script to create a test environment, complete with | ||
+ | database initialization. This is used by our CI server. The | ||
+ | following will configure and build NAV automatically, and install it | ||
+ | into a directory called ''workspace/build''. It will also create a | ||
+ | suitable virtualenv in ''workspace/.env'', which you can activate | ||
+ | before running tests: | ||
+ | |||
+ | <code bash> | ||
+ | export PGDATABASE=testdb | ||
+ | export PGUSER=testuser | ||
+ | tests/bootstrap-test-environment.sh workspace | ||
+ | </code> | ||
+ | |||
+ | |||
+ | ===== Javascript testing ===== | ||
+ | |||
+ | Testing of javascript is in its infancy in NAV. We are currently using [[http://busterjs.org/|buster.js]] as testing toolkit. | ||
+ | |||
+ | To install buster.js install [[http://nodejs.org/|node]] and then: | ||
+ | <code> | ||
+ | npm install -g buster | ||
+ | </code> | ||
+ | |||
+ | As we use [[http://requirejs.org/|require.js]] you need the AMD module of buster aswell. Install it in the /media/js directory: | ||
+ | <code> | ||
+ | npm install buster-amd | ||
+ | </code> | ||
+ | |||
+ | To run the tests you need to | ||
+ | - Start a buster server by typing ''buster-server'' | ||
+ | - Capture browsers by pointing browsers to the buster-server (default localhost:1111) | ||
+ | - Go to /media/js | ||
+ | - Run the tests by typing ''buster-test'' | ||
+ | |||
+ | All tests are located under ''media/js/tests/''. Create new tests there. For syntax, assertions and related stuff take a look at the tests already there and [[http://busterjs.org/docs/|the buster docs]]. | ||
+ | ===== Jenkins ===== | ||
+ | |||
+ | We use //Jenkins// (formerly //Hudson//) for Continuous Integration testing of | ||
+ | NAV. All the automated tests are run each time new changesets are pushed to | ||
+ | the NAV repositories. Jenkins also runs pylint to create stats on code | ||
+ | quality. | ||
+ | |||
+ | Our Jenkins installation is available on http://nav.uninett.no/jenkins . | ||
+ | |||
+ | ===== Tips and tricks ===== | ||
+ | |||
+ | ===== Make fixtures for integration testing ===== | ||
+ | |||
+ | <code> | ||
+ | from django.core import serializers | ||
+ | from nav.models.manage import Netbox | ||
+ | |||
+ | fixtures = serializers.serialize("xml", Netbox.objects.all()[:2]) | ||
+ | </code> | ||
+ | |||
+ | Fixtures can so be used in your integration tests by extending | ||
+ | the test case DjangoTransactionTestCase in nav.tests.cases | ||
+ | |||
+ | See nav.tests.integration.l2trace_test for an example on applying | ||
+ | fixtures for your particular test case. | ||
+ | |||
+ | Also keep in mind you have to make sure you have the model | ||
+ | dependency in correct order when importing. | ||
+ | Example: Netbox contains a location to a Room where it is located, | ||
+ | you have to make sure Room's are imported first before importing | ||
+ | Netbox's | ||
+ | |||
+ | See https://docs.djangoproject.com/en/dev/topics/serialization/ | ||
+ | |||
+ | TODO: Be able to use [[https://docs.djangoproject.com/en/dev/ref/django-admin/#dumpdata-appname-appname-appname-model|django-admin's management command: dumpdata]] | ||
+ | to create fixtures. | ||
+ | |||
+ | |||
====== Submitting patches ====== | ====== Submitting patches ====== | ||
Line 286: | Line 450: | ||
addressed to the //nav-dev// mailing list. Please **do not | addressed to the //nav-dev// mailing list. Please **do not | ||
patchbomb** the mailing list with multiple emails. | patchbomb** the mailing list with multiple emails. | ||
- |