The history of NAV

The development of NAV started at NTNU in 1999. Prior to that NTNU, like so many others, had acquired a traditional network management system (HP Openview) that hopefully would fulfil our needs. After thorough investigation and adjustment of the system we found however that it lacked functionality in many areas. We decided to implement a basic system ourself.

The name NAV came about later, in the initial stages we called our system NettInfo. The first tasks we solved were:

• A rudimentary collection system that gathered router port data using SNMP.

We introduced MRTG at the time and found it tedious to manually update the MRTG config file, thus we made:

• An automated system for generating the MRTG configs based on the actual router ports in use.

At the time we also developed a separate system, called Falkeblikk:

• A basic status monitoring that used ping to verify availability of routers, switches and servers. Detected outages where reported using SMS (or email). (Falkeblikk was in 2001 replaced by Live and in 2002 replaced by pping).

Further we expanded to cover:

• An ARP data collector and database giving us historical information on IP and MAC address usage.

• Web based reports with data on routers, router ports, switches and switch ports.

• The first version of the network map showing traffic load on a manually laid out network graph.

Project participants in 1999: Vidar Faltinsen (project leader), Trygve Lunheim, Tor-Arne Kvaløy, Eric Sandnes, Stig Venås, Gro-Anita Vindheim, Knut-Helge Vindheim.

The development at the time was focused around perl and php. In 2000 we introduced java in an upgraded version of the traffic map (vlanplot). We also introduced mySQL for our database. Further:

• We evolved our collection system around an important design principle; we would use seed files and SNMP to build our model of the running network. In other words; we would not put resources into doing auto discovery. Our initial understanding of topology required port naming standards on the configured routers and switches in the network.

• We improved our web front end, introducing a basic authentication and authorization scheme supporting three levels of access (public, restricted and secret).

• The initial event and alert engine, named Trapdetect, was implemented. Trapdetect incorporated an snmp trap demon. Trapdetect parsed incoming traps and sent alarms to users using email or SMS. (Trapdetect is in NAV v3 replaced by event engine, alert engine and alert profiles).

• We moved from MRTG to RRD and Cricket. We automated the process of making the Cricket configuration tree.

Project participants in 2000: John Magne Bredal, Kristian Eide, Vidar Faltinsen (project leader), Trygve Lunheim, Gro-Anita Vindheim, Knut-Helge Vindheim.

In early 2001 UNINETT gained interest in our system. They asked whether it could be used on other universities and colleges as well. At the time the system was tightly coupled with the NTNU network design and the local naming standards used for equipments and router/ switch ports.

We decided to take the challenge and with funding from UNINETT the process of making a more general network management system started. This was when our name came about; Network Administration Visualized or NAV. The system running at NTNU at the time was referred to as NAV version 1 (or NTNU NAV).

The 2001 development project (called NAVMe) redesigned the NAV database from scratch introducing many improvements. We also moved from mySQL to postgreSQL with the advantages that imply (cascading deletes etc). The collection system was revitalized and we made a general report generator.

A crucial result was the general topology detector. We no longer based topology detection on port naming standards, at least not on the switch layer. We introduced a general CAM collector gathering mac address to switch port data. By complementing this with CDP data we were able to understand the physical network topology on layer 2. Further we gathered vlan information including trunk port data and spanning tree states. We also derived the actual scope of the various vlans based on the actual live status in the running network.

We adopted CVS as our code revision system. (CVS was replaced by subversion in 2003).

The result of project NAVMe was NAV v 2.0. NAV v2.0 was successfully installed at NTNU and at the University of Tromsø in late 2001.

In summary NAV version 2.0 incorporated:

• A general collection system building a model of the running network

• A status monitor (Live) reporting network outage (using topology information to report on root cause)

• A general report generator introducing an easy to expand set of reports (equipment types, software version, IP address usage, vlan scope, switch port state etc).

• An automated Cricket config builder allowing “maintenance free” gathering of network statistics, including octet, packet and error counters.

• A graphical drill-down display of the network topology also visualizing traffic and CPU load.

• A machine tracker with historical data on the whereabouts of machines (IP and mac adresses) in the network (pinpointed down to the connected switch port).

• A general RRD threshold monitor sending alarms on rising and falling thresholds (using hysteresis).

• An event and alert system (Trapdetect) with the ability to interpret internal and external (SNMP trap) events and sending alerts using SMS or email to NAV users. NAV users could configure their alert profile using a web based front end.

Project participants in 2001: John Magne Bredal, Kristian Eide, Vidar Faltinsen (project leader), Sigurd Gartmann, Gro-Anita Vindheim, Knut-Helge Vindheim.

With continued contribution from NTNU and UNINETT we kept on with our work in two follow-up projects named NAVMore and NAVRun. The former focused on further development of NAV, while the latter introduced an installation system making NAV easily available to UNINETT members; i.e. Norwegian universities and colleges.

The development was now done in two parallel tracks; version 2.1 and version 3.

Version 2.1 introduced improvements to the existing version 2.0 and was geared for the 2002 distribution of NAV to UNINETT members. New functionality in NAV version 2.1 included:

• Improvement in the topology detection, supporting a larger set of equipment types.

• A Cisco syslog parser and database.

• A tabular display of the IP address scope usage.

By the end of 2002 all four Norwegian universities and a total of six colleges where running NAV v2.1. Complementing our documentation pages and our mailing lists we introduced a web based bug reporter system (using php bugtracker). We further established a scheme for delivering patches to our system. At the time distribution was done through ftp with a closed user group.

In the summer of 2002 we radically expanded the development group. The main effort was on the new NAV version 3. Unfortunately development of NAV version 3 has been more time demanding than expected. In fact the version 3 results produced in 2002 have not been deployed until the beta release of NAV v3 in early 2004.

Nevertheless, the NAV v3 2002 results included:

• A sorted, tabular display of RRD data.

• Cricket/RRD collection of server statistics (i.e. disk usage, cpu load, number of users etc).

• A new status monitor, pping, working in parallel (thread based, implemented in python).

• A service monitor detecting outages of services. Initially supporting ssh, http, imap, pop, smtp, smb, rpc, dns, dc. The monitor is plugin based, allowing for easily expansion of new services to monitor.

• A general event system, the event engine, replaces trapDetect. Event engine parses an event queue and has built in mechanisms for correlating events and reporting based on network topology. With event engine SNMP is no longer the only input possibility for events. Further, the task of processing alerts to users is separated from the more general work of coping with events. The event engine also reduces the complexity needed in the basic monitors (status, threshold, service etc), as they no longer need to worry about network topology issues.

• A new alert engine and a new alert profile system focused on delivering alarms to NAV users based on their configured profile. Alert engine and alert profile is very general. Where trapdetect had many constraints as to how the NAV user could adjust his profile, alert profile allows the user to configure almost anything. Further, it is easy to support more alarm types in the future. Initially support for email and SMS was implemented.

Project participants in 2002: John Magne Bredal, Kristian Eide, Vidar Faltinsen (project leader), Sigurd Gartmann, Erik Gorset, Steinar Hamre, Magnus Thanem Nordseth, Daniel Sandvold, Andreas Åkre Solberg, Stian Søiland, Knut-Helge Vindheim, Morten Vold, Arne Øslebø.

In 2003 focus has been on finalizing NAV version 3.0. Once again NTNU and UNINETT are supporting development, and a large group of students and staff are involved in development. This years project was called tigaNAV. Some important decisions are made:

• Adopt English as our language in every aspect of the system (no more Norwegian). This would include the database (with names on tables and fields), the code (with comments, and ideally the names of all variables), the web interface (of course) and all documentation, including the project report.

• Adopt python as the preferred programming language for the user interface. Introduce a templating mechanism, thus abstracting the look and feel from the content programming. Cheetah was chosen as our templating engine. In effect we wanted a more consistent (and more professional) user interface.

• Adopt a general and powerful authentication and authorization mechanism. Privileges may be granted to account groups based on a “who, what, where” principle.

• Replace CVS with subversion. Subversion allows atomic commits and history is kept in cases where files or directories are renamed or moved.

• The monolithic installation approach used for NAV v2.1 is left. The source code repository is restructured and a build system is established. Installation layout is also restructured adopting a typical UNIX like directory structure.

Further many significant changes is done in the underlying collection system. One important goal is to improve NAV's ability to support new equipment types:

• An OID database is introduced containing information on all SNMP OIDs NAV used in the collection process. The new collection engine, getDeviceData, uses the OID database. When new equipment is added to NAV it is tested against all the OIDs and classified accordingly.

The new collection system (getDeviceData) uses a plug in-based architecture, working in threads. We allow for fine-grained control of collection intervals. In effect the NAV database is kept more up-to-date with changes in the managed network (in version 2 updates were done on a nightly basis).

Finally there are important functional improvements:

• A more general operational status page with status on all operational events (eventually).

• An integrated message system to inform NAV users of scheduled outage, special faults and other operational events.

• The IP device center presenting all information on a device in one web page with links to reports, statistics, switch port data etc.

• The network explorer introducing a graphical display of the network layout on a per vlan basis.

• An RRD browser with the ability to gather different statistics on the same page or even in the same graph.

• Device management with the ability to track milestone events of physical devices from order and arrival through the operational stages.

Also worth mentioning; NAV has so far been presented on numerous occasions/ conferences in Norway. In 2003 NAV is for the first time presented internationally:

• For the Cisco Works team (with the Cisco Works product manager and director of engineering) at the Cisco headquarters in San Jose, California. March 2003.

• On the 21st NORDUnet Networking Conference in Reykjavik, Iceland. August 2003.

Project participants in 2003: John Magne Bredal, Kristian Eide, Vidar Faltinsen (project leader), Sigurd Gartmann, Bjørn Ove Grøtan, Hans Jørgen Hoel, Erlend Mjåvatten, Magnus Thanem Nordseth, Andreas Åkre Solberg, Magnar Sveen, Stian Søiland, Gro-Anita Vindheim, Morten Vold, Arne Øslebø.

With most of the NAV version 3 development projects left behind, the focus in 2004 was to finish off unfinished work, fixing bugs and testing the system thoroughly. In terms of new functionality, a conservative model was followed. Our focus was on completing NAV3 and thus permanently leaving NAV2 behind.

Many hours were put into further improving the collection system (getDeviceData). We also made important enhancements to event engine, the traffic map, network explorer and the message system.

An important decision of releasing NAV under GPL is made. In the process many arguments are put forward, the most important being the new opening for distributing of code. It is our hope that a more wide spread deployment of NAV will in return give interesting code contributions back to the project.

The process of preparing the code for general availability was on the agenda in the second quarter in 2004. In parallel the code was beta tested on an increasingly set of Norwegian universities and colleges.

Documentation was improved and translated to English, and a new wiki web site (based on MoinMoin) was established, also allowing for a more distributed contribution of documentation.

A total of 8 NAV 3.0 beta releases was released in 2004, beta4 (May 11) being the first GPL release and beta7 (July 27) being the first with a changelog (which has been updated since).

Project participants in 2004: John Magne Bredal, Kristian Eide, Vidar Faltinsen, Sigurd Gartmann, Hans Jørgen Hoel, Magnus Thanem Nordseth, Andreas Åkre Solberg, Magnar Sveen, Stian Søiland, Gro-Anita Vindheim, Morten Vold (project leader), Arne Øslebø.

2005 was a relatively slow year for NAV. NTNU and UNINETT did not have so much resources to put into the project. At the start of the year we introduced beta9, followed by beta10 in February. Focus was on bug fixing the most vital components. In June we finally announced our first release candidate (rc1), followed by a second in July.

NAV 3.0.0 was finally released on September 16 2005 with the following highlights:

• Completely new web interface - new look
• New authentication and authorization system
• New Java based snmp collector: getDeviceData
• New database design
• New event- and alarm system (event engine, alert engine, alert profiles)
• Module monitor
• New message system
• New tool Network discovery, improved vlanplot (the network map)

For further details on the improvements made in the 2005 releases, we refer to the 3.0 change log with its history.

Project participants in 2005: John Magne Bredal, Kristian Eide, Vidar Faltinsen, Gro-Anita Vindheim, Morten Vold (project leader).

2006 has been a definite upswing for NAV. Project development is now supported by the UNINETT GigaCampus programme with an overall goal of getting NAV up and running at most major universities and university colleges in Norway within 2008. Morten Brekkevold (previously Vold) is devoted full time to NAV, now being employed by UNINETT. Vidar Faltinsen made a comeback as project leader. John Magne Bredal at NTNU devoted 40% of his time working with NAV. We also recruited a student, Stein Magnus Jodal, to work on NAV as a summer intern, and we kept him on for the fall.

The Universities of Tromsø and Bergen began sponsoring a FreeBSD port of NAV, while Tromsø offered to contribute their own code to NAV. The Debian developer Morten Werner Olsen at The University of Oslo was commisioned by UNINETT to maintain a Debian NAV package. A SuSE package was maintained by NTNU and Centos RPMs were contributed by Alexander Krapivin at MSU, Moscow, Russia.

A strategy of targeting new feature releases twice a year was established (with bugfix releases in between). NAV 3.1 was released in June and 3.2, with an original schedule of October, was released in February 2007.

NAV gained the capability to actively block switch ports through the addition of the NTNU-contributed Arnold tool. Adding the ability to collect high speed traffic counters (64 bits) made it possible to reduce the Cricket collector frequency for gigabit switch ports from every 1 minute to every 5 minutes, thus removing the need to have a separate configuration tree for gigabit ports. The IP Device Center (later renamed to IP Device Info) was introduced.

We established a project page for NAV at SourceForge. This was used for bug tracking and hosting source code releases. A month after the 3.1 release, we had seen approximately 160 downloads from around the world (at the same time, the 3.0 version had been downloaded 674 times).

Project participants in 2006: John Magne Bredal, Vidar Faltinsen (project leader), Stein Magnus Jodal, Morten Brekkevold.

The NAV year of 2007 opened with the belated 3.2 release in February. The 3.2 release also marked our first step towards realizing a new long-term goal: Reduce the number of dependencies and programming languages utilized in NAV.

The 3.2 release replaced the messages system with separate maintenance and message systems, rewritten from scratch, after many usability complaints from users. The Perl SMS daemon was rewritten in Python, adding support for SMS dispatcher plugins. The Ranked Statisdtics feature was introduced. Much of the behind-the-scenes HTML and CSS was cleaned up and restructured. The release also brought numerous small improvements to NAV overall.

October 1st brought the NAV 3.3 feature release. This release added the long-awaited plugin based SNMP trap daemon, which enabled NAV to translate traps into NAV events and alerts. Browsing and searching of device history was greatly improved. The web interface received a major facelift. Switch/router interface names replaced port numbers as identifiers in most parts of the web interface. A router port view was added to IP Device Center (Info).

The 2007 development project saw Morten Brekkevold take over as the project leader. Kristian Klette was taken on as a new student intern, and started working on a Traffic Map replacement, based on the Prefuse visualization library.

Project participants in 2007: John Magne Bredal, Vidar Faltinsen, Stein Magnus Jodal, Kristian Klette, Jostein Gogstad, Morten Brekkevold (project leader).

In May 2008 NAV 3.4 was released. IPv6 support was added for the machine tracker, subnet collection and subnet matrix. The LDAP authentication was improved.

In the summer and autumn there was ongoing development for v3.5.

In January NAV 3.5 was released:

• A new and improved IP Device Info now replaced IP Device Center.
• Alert Profiles and Alert Engine were rewritten in Python.
• vlanplot was replaced by netmap.
• Network explorer was rewritten in Python.
• A new Radius tool was introduced that was able to track 802.1X authenticated users on the network
• Report was improved and local reports were allowed
• HP switch switch stack support was report do to many problems

NAV 3.6 was a major release for NAV as it introduced the forth generation SNMP collection system - ipdevpoll. ipdevpoll would thus replace getDeviceData. The firs beta version of 3.6 came in May and the final release (after 6 beta version) was launched in October. NAV 3.6 included:

• New SNMP collector: ipdevpoll
• Major changes to the dataves with a new common interface table (leaving the separate router interface and switxh port tables behind)
• New backend daemon mailin: For receiving external alarms as email, parsing them and posting events on the event queue
• Enhancements to report system: CSV export, monthly availability report and more
• Several subsystems rewritten: machine tracker, dev history, makecricketconfig (from perl to python)
• Support “ignore resolved alerts” in Alert Profiles

NAV 3.7 came just a month after 3.6, in November, introducing more results from summer students:

• Geomap - a geographical map using OpenStreetMap data.
• A sudo (mechaism allowing NAV administrators to manage user’s alert profiles and other setup
• Introducing AD LDAP support

In February NAV 3.8 was out with:

• New tool portAdmin that allows for configuring vlan value and port descriptions for switch ports.
• The seedDB tool was rewritten
• A report for inactive devices was included

In May 3.9 came with:

• Threshold manager: New GUI to set threshold alarms - finally in place
• New tool Mac Watch: Generate alarm when a mac address appears on the network
• Improved the physical topology detection (“topology” replaces “networkdiscovery topology”)
• Support for Extreme Switches
• pynetsmp/NetSNMP support for ipdevpoll - significant performance impact

In December 3.10 came with:

• Port link status monitoring with ipdevpoll - limit to topology ports if you like
• Cricket statistics for environmental sensors in network equipment and Weathergoose
• Generate alarm when a redundant power or fan in a switch fails (Cisco and HP)
• Manage alarms from UPS: supports UPS-MIB (RFC 1628), APC, Eaton and MGE
• Improved vlan topology detection (“topology” now entirely replaces “networkdiscovery”)
• Parallel DNS lookups increases performance
• The status monitor (pping) now supports IPv6
• Alerts when SNMP agent stops responding

In November 2011 a NAV reference committee was established. The reference committee consists of NAV power users from dominant Norwegian universities and university colleges. The reference committee will give advice to future developments in NAV and help UNINETT in making our priorities. The committee has members from these institutions: NTNU, UiT, UiO, UiA, HiVolda.

In March NAV 3.11 was released with the following improvements:

• LLDP support for improved topology detection.
• getBoksMacs replaced by ipdevpoll plugins in `topo` job.
• Q-BRIDGE-MIB now also used when collecting switch forwarding tables,

meaning improved machine tracking data for switch vendors like HP, Juniper

  and Alcatel.
* The Machine Tracker's IP search form has consolidated the `To` and `From`
  IP address search fields into a single `IP Range` search field with
  extended search syntax for ranges of IP addresses.  See LP#643544 for
  details.
* ipdevpoll can now be made to run single job configurations from
  ipdevpoll.conf, to run in the foreground, and to log to stderr instead of
  its log file.
* ipdevpoll will reset its log levels from logging.conf upon receiving a
  SIGHUP signal.
* ipdevpoll will log the currently active polling jobs upon receiving a
  SIGUSR1 signal.

In June 2012 the number of full time NAV-developers doubled!

In September NAV 3.12 was out with:

• Rewrite of new Netmap using open standards as HTML5, Javascript (D3JS) and SVG (Scalable vector graphics)
• A cleaner and more responsive Ip Device Info
• An improved navbar search
• Room information including devices and a powerful netbox interface overview
• IPv6 traffic counters
• An improved device poller
• Greater use of django in favor of mod_python