backendprocesses
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
backendprocesses [2008/04/03 10:16] – bredal | backendprocesses [2015/09/24 12:48] (current) – warn of obsolesence morten | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Back-end processes in NAV ====== | ====== Back-end processes in NAV ====== | ||
- | NAV has a number of back-end processes. This document gives an overview, listing key information | + | <note warning> |
- | detailed description | + | |
- | + | ||
- | + | ||
- | + | ||
- | The following figure complements this document (the NAV 3.3 snmptrapd is not included in the figure): | + | |
- | + | ||
- | {{architecture1.png? | + | |
- | + | ||
- | + | ||
+ | NAV has a number of back-end processes. This page attempts to give an overview of them. | ||
+ | {{: | ||
===== nav list / nav status ===== | ===== nav list / nav status ===== | ||
Line 24: | Line 16: | ||
* [[backendprocesses# | * [[backendprocesses# | ||
* [[# | * [[# | ||
- | * [[#getdevicedata|getDeviceData]] | + | * [[#ipdevpoll|ipdevpoll]] |
- | * [[# | + | |
* [[# | * [[# | ||
* [[# | * [[# | ||
* [[# | * [[# | ||
- | * [[# | ||
* [[# | * [[# | ||
+ | * [[# | ||
* [[# | * [[# | ||
* [[# | * [[# | ||
- | * [[# | ||
* [[# | * [[# | ||
+ | * [[# | ||
+ | * [[# | ||
====== Building the network model ====== | ====== Building the network model ====== | ||
- | ===== getDeviceData | + | ===== ipdevpoll |
- | + | ||
- | + | ||
==== Key information ==== | ==== Key information ==== | ||
- | ^ Process name | + | ^ Process name |
- | ^ Alias | gDD / the snmp data collector | + | |
^ Polls network | ^ Polls network | ||
^ Brief description | ^ Brief description | ||
- | ^ Depends upon | Seed data must be filled in the netbox table, | + | ^ Depends upon | Seed data must be filled in the netbox table, |
- | ^ Updates tables | + | ^ Updates tables |
- | ^ Run mode | Daemon process. Thread based. | + | ^ Run mode | Daemon process | |
- | ^ Default scheduling | + | ^ Default scheduling |
- | ^ Config file | getDeviceData.conf | | + | ^ Config file | ipdevpoll.conf | |
- | ^ Log files | getDeviceData.log og getDeviceData/ | + | ^ Log files | ipdevpoll.log | |
- | ^ Programming language | Java | | + | ^ Programming language | Python |
- | ^ Lines of code | Approx 8200 | | + | ^ Further doc | | |
- | ^ Further doc | [[http:// | + | |
Line 65: | Line 52: | ||
==== Details ==== | ==== Details ==== | ||
- | * Initial OID classification | + | * jobs and plugins |
- | + | * inventory job \\ Polls for inventory information every 6 hours (by default). | |
- | * Plugin-based architecture | + | * profiler job \\ Runs every 5 minutes, profiling devices if deemed necessary. |
- | * Device plugins collects data with SNMP. Each device plugin is geared towards a particular type of equipment, supporting a particular subset of OIDs. See further doc for details. | + | * logging job \\ Runs every 30 minutes |
- | * Data plugins updates NAVdb with data fed from the device plugins. A particular data plugin is responsible for a particular table (or set of tables) in the database. See further doc for details. | + | |
- | + | ||
- | * Module monitor | + | |
- | + | ||
- | + | ||
- | ===== iptrace ===== | + | |
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | ==== Key information ==== | + | |
- | + | ||
- | ^ Process name | iptrace | | + | |
- | ^ Alias | IP-to-mac collector / arplogger| | + | |
- | ^ Polls network | + | |
- | ^ Brief description | + | |
- | ^ Depends upon | The routers (GW / GSW) must be in the netbox table. To assign prefixes to arp entries, [[# | + | |
- | ^ Updates tables | + | |
- | ^ Run mode | cron | | + | |
- | ^ Default scheduling | + | |
- | ^ Config file | getBoksMacs.conf | | + | |
- | ^ Log file | getBoksMacs.log | | + | |
- | ^ Programming language | Perl| | + | |
- | ^ Lines of code | Approx 130 lines| | + | |
- | ^ Further doc | [[http:// | + | |
- | + | ||
- | + | ||
- | + | ||
- | ==== Details ==== | + | |
- | + | ||
- | * iptrace understands proxy arp and will not store arp entries that are " | + | |
- | * iptrace ignores routers that are known to be down (fixed in NAV 3.1). | + | |
- | * The command line tool [[commandlinetools# | + | |
Line 114: | Line 67: | ||
^ Polls network | ^ Polls network | ||
^ Brief description | ^ Brief description | ||
- | ^ Depends upon | [[#getDeviceData|getDeviceData]] must have created the swport tables for the switches. | | + | ^ Depends upon | [[#ipdevpoll|ipdevpoll]] must have created the swport tables for the switches. | |
^ Updates tables | ^ Updates tables | ||
^ Run mode | cron | | ^ Run mode | cron | | ||
Line 121: | Line 74: | ||
^ Log file | getBoksMacs.log | | ^ Log file | getBoksMacs.log | | ||
^ Programming language | Java | | ^ Programming language | Java | | ||
- | ^ Lines of code | Approx 1400 | | + | ^ Further doc | [[http://nav.uninett.no/ |
- | ^ Further doc | [[http://metanav.uninett.no/ | + | |
Line 155: | Line 107: | ||
One notable improvement is the addition of the interface field in the swport table. It is used for matching the CDP remote interface, and makes this matching much more reliable. Also, both the cam and the swp_netbox tables now use netboxid and ifindex to uniquely identify a swport port instead of the old netboxid, module, port-triple. This has significantly simplified swport port matching, and especially since the old module field of swport was a shortened version of what is today the interface field, reliability has increased as well. | One notable improvement is the addition of the interface field in the swport table. It is used for matching the CDP remote interface, and makes this matching much more reliable. Also, both the cam and the swp_netbox tables now use netboxid and ifindex to uniquely identify a swport port instead of the old netboxid, module, port-triple. This has significantly simplified swport port matching, and especially since the old module field of swport was a shortened version of what is today the interface field, reliability has increased as well. | ||
- | - | ||
- | |||
- | ===== networkDiscovery_topology ===== | ||
- | |||
- | |||
- | |||
+ | ===== topology ===== | ||
==== Key information ==== | ==== Key information ==== | ||
- | ^ Process name | + | ^ Process name |
- | ^ Alias | Physical Topology Builder | | + | ^ Alias | Physical |
^ Polls network | ^ Polls network | ||
- | ^ Brief description | + | ^ Brief description |
- | ^ Depends upon | mactrace fills data in swp_netbox representing the candidate | + | ^ Depends upon | mactrace fills data in '' |
^ Updates tables | ^ Updates tables | ||
^ Run mode | cron | | ^ Run mode | cron | | ||
^ Default scheduling | ^ Default scheduling | ||
^ Config file | None | | ^ Config file | None | | ||
- | ^ Log file | + | ^ Log file |
- | ^ Programming language | Java | | + | ^ Programming language | Python |
- | ^ Lines of code | Approx 1500 (shared with vlan topology builder) | | + | |
- | ^ Further doc | [[http:// | + | |
==== Details ==== | ==== Details ==== | ||
- | FIXME This is cut and paste from the tigaNAV report. Consider a rewrite. | + | === Physical topology === |
+ | The topology discovery system builds NAV's view of the network topology based | ||
+ | on cues from information collected previously via SNMP. | ||
+ | The information cues come from routers' | ||
+ | Discovery caches, interface physical (MAC) addresses, switch forwarding tables | ||
+ | and CDP (Cisco Discovery Protocol). | ||
+ | pre-parsed these cues and created a list of neighbor candidates for each port | ||
+ | in the network. | ||
- | The network topology discovery system automatically discovers the physical topology | + | The physical topology |
- | + | of neighbor candidates | |
- | * We start with a candidate list for each swport port. These are the switches located behind a switch port and the goal of the algorithm is to pick the one to which it is connected directly. Some of the candidate lists, those of the switches one level up from the edge, will contain only one candidate. We can thus pick this as the switch directly connected and proceed to remove said switches from all other lists. After this removal there will be more candidate lists with only one candidate, and we can apply the same procedure again. | + | |
- | * If we have the complete information about the network we could now simply iterate until all candidate lists were empty; however, to deal with missing information we sometimes have to make an educated guess of which is the directly connected switch. The network topology discover system makes the guess by looking at how far each candidate is from the router and how many switches are connected below them, and then try to pick the one which most closely matches the current switch. | + | |
- | + | ||
- | In practice the use of CDP makes this process very reliable for the devices supporting it, and this makes it easier to correctly determine the remaining topology even in the case of missing information. | + | |
- | + | ||
- | ===== networkDiscovery_vlan ===== | + | |
- | + | ||
- | + | ||
- | ==== Key information ==== | + | |
- | ^ Process name | networkDiscovery.sh vlan| | + | |
- | ^ Alias | Vlan Topology Builder | | + | |
- | ^ Polls network | + | |
- | ^ Brief description | + | |
- | ^ Depends upon | The physical topology need to be in place, this process therefore supersedes the physical topology builder.| | + | |
- | ^ Updates tables | + | |
- | ^ Run mode | cron | | + | |
- | ^ Default scheduling | + | |
- | ^ Config file | None | | + | |
- | ^ Log file | networkDiscovery/ | + | |
- | ^ Programming language | Java | | + | |
- | ^ Lines of code | See the physical topology builder above | | + | |
- | ^ Further doc | [[http:// | + | |
- | + | ||
- | + | ||
- | ==== Details ==== | + | |
- | FIXME This is cut and paste from the tigaNAV report. Consider a rewrite. | + | In practice the use of CDP makes this process very reliable for the devices |
+ | supporting it, and this makes it easier to correctly determine | ||
+ | topology even in the case of missing information. CDP is, however, not | ||
+ | trusted more than switch forwarding tables, as CDP packets may pass unaltered | ||
+ | through switches that don't support CDP, causing CDP data to be inaccurate. | ||
- | After the physical | + | === VLAN topology |
- | The vlan discovery system uses a simple top-down depth-first graph traversal algorithm to discover which VLANs are actually running on the different trunks and in which direction. Direction is here defined relative to the router port, which is the top of the tree, currently owning the lowest gateway IP or the virtual IP in the case of HSRP. In addition, since NAV v3 now fully supports the reuse of VLAN numbers, the vlan discovery system will also make the connection from VLAN number to actual vlan as defined in the vlan table for all non-trunk ports it encounters. | + | After the physical topology model of the network has been built, the logical |
+ | topology | ||
+ | trunking, which can transport several independent VLANs over a single physical | ||
+ | link, the logical topology can be non-trivial and indeed, in practice | ||
+ | usually is. | ||
- | A special case are //closed// VLANs which do not have a gateway IP; the vlan discovery system | + | The vlan discovery system |
+ | algorithm to discover which VLANs are actually running on the different trunks | ||
+ | and in which direction. Direction is here defined relative to the router port, | ||
+ | which is the top of the tree, currently owning the lowest gateway IP or the | ||
+ | virtual IP in the case of HSRP. | ||
+ | parts of the network is supported. | ||
- | The implementation of this subsystem is again complicated by factors such as the need for checking at both ends of a trunk if the VLAN is allowed to traverse it, the fact that VLAN numbers on each end of non-trunk links need not match (the number closer to the top of the tree should then be given precedence and the lower VLAN numbers rewritten to match), that both trunks and non-trunks can be blocked (again at either end) by the spanning tree protocol and of course that it needs to be highly efficient and scalable in the case of large networks with thousands of switches and tens of thousands of switch ports. | + | The VLAN topology detector does not currently support mapping unrouted VLANs. |
====== Monitoring the network ====== | ====== Monitoring the network ====== | ||
Line 241: | Line 180: | ||
^ Log file | pping.log | ^ Log file | pping.log | ||
^ Programming language | Python | | ^ Programming language | Python | | ||
- | ^ Lines of code | Approx 4200, shared with servicemon | | + | ^ Further doc | See below, based on and translated from [[http://nav.uninett.no/ |
- | ^ Further doc | See below, based on and translated from [[http://metanav.uninett.no/ | + | |
Line 325: | Line 263: | ||
^ Log file | servicemon.log | ^ Log file | servicemon.log | ||
^ Programming language | Python | | ^ Programming language | Python | | ||
- | ^ Lines of code | See pping above, shared code base | | + | ^ Further doc | See the [[servicemon]] page and/ |
- | ^ Further doc | [[http://metanav.uninett.no/ | + | |
==== Details ==== | ==== Details ==== | ||
Line 348: | Line 285: | ||
^ Log file | thresholdMon.log | ^ Log file | thresholdMon.log | ||
^ Programming language | Python | | ^ Programming language | Python | | ||
- | ^ Lines of code | Approx 400 | | ||
^ Further doc | See [[ThresholdMonitor]] | | ^ Further doc | See [[ThresholdMonitor]] | | ||
Line 355: | Line 291: | ||
* See [[ThresholdMonitor]] | * See [[ThresholdMonitor]] | ||
- | |||
- | ===== moduleMon ===== | ||
- | |||
- | |||
- | ==== Key information ==== | ||
- | ^ Process name | getDeviceData data plugin moduleMon | | ||
- | ^ Alias | The module monitor | | ||
- | ^ Polls network | ||
- | ^ Brief description | ||
- | ^ Depends upon | The switch or router to be processed by gDD with apropriate data in module and gwport/ | ||
- | ^ Updates tables | ||
- | ^ Run mode | daemon, a part of gDD. | | ||
- | ^ Default scheduling | ||
- | ^ Config file | see gDD | | ||
- | ^ Log file | see gDD | ||
- | ^ Programming language | Java | | ||
- | ^ Lines of code | Part of gDD, see gDD. | | ||
- | ^ Further doc | Not much. | | ||
Line 392: | Line 310: | ||
^ Log file | eventEngine.log | ^ Log file | eventEngine.log | ||
^ Programming language | Java | | ^ Programming language | Java | | ||
- | ^ Lines of code | Approx 3000 lines | | + | ^ Further doc | [[http://nav.uninett.no/ |
- | ^ Further doc | [[http://metanav.uninett.no/ | + | |
==== Details ==== | ==== Details ==== | ||
Line 416: | Line 333: | ||
^ Log file | maintengine.log | ^ Log file | maintengine.log | ||
^ Programming language | Python | | ^ Programming language | Python | | ||
- | ^ Lines of code | Approx 300 | | + | ^ Further doc | Old doc: [[http://nav.uninett.no/ |
- | ^ Further doc | Old doc: [[http://metanav.uninett.no/ | + | |
==== Details ==== | ==== Details ==== | ||
Line 437: | Line 353: | ||
^ Config file | alertengine.cfg | | ^ Config file | alertengine.cfg | | ||
^ Log file | alertengine.log og alertengine.err.log | ^ Log file | alertengine.log og alertengine.err.log | ||
- | ^ Programming language | perl | | + | ^ Programming language | Python |
- | ^ Lines of code | Approx 1900 | | + | ^ Further doc | [[http://nav.uninett.no/ |
- | ^ Further doc | [[http://metanav.uninett.no/ | + | |
==== Details ==== | ==== Details ==== | ||
Line 462: | Line 377: | ||
^ Log file | smsd.log | | ^ Log file | smsd.log | | ||
^ Programming language | Python (Perl in 3.1) | | ^ Programming language | Python (Perl in 3.1) | | ||
- | ^ Lines of code | Approx 1200 | | ||
^ Further doc | subsystem/ | ^ Further doc | subsystem/ | ||
Line 502: | Line 416: | ||
- | ===== The snmptrapd ===== | + | ===== snmptrapd ===== |
Line 520: | Line 434: | ||
^ Log file | snmptrapd.log and snmptraps.log | ^ Log file | snmptrapd.log and snmptraps.log | ||
^ Programming language | Python | ^ Programming language | Python | ||
- | ^ Lines of code | Approx 200 + traphandlers | | ||
^ Further doc | - | | ^ Further doc | - | | ||
Line 537: | Line 450: | ||
^ Polls network | ^ Polls network | ||
^ Brief description | ^ Brief description | ||
- | ^ Depends upon | That gDD has filled the gwport, swport tables (and more...) | | + | ^ Depends upon | That ipdevpoll |
^ Updates tables | ^ Updates tables | ||
^ Run mode | cron | | ^ Run mode | cron | | ||
Line 543: | Line 456: | ||
^ Config file | None | | ^ Config file | None | | ||
^ Log file | cricket-changelog | ^ Log file | cricket-changelog | ||
- | ^ Programming language | perl | | + | ^ Programming language | python |
- | ^ Lines of code | Approx 1600 | | + | |
^ Further doc | [[howtoconfigurecricket|How to configure Cricket addons in NAV v3]] | | ^ Further doc | [[howtoconfigurecricket|How to configure Cricket addons in NAV v3]] | | ||
Line 565: | Line 477: | ||
^ Log file | cricket/ | ^ Log file | cricket/ | ||
^ Programming language | not relevant | | ^ Programming language | not relevant | | ||
- | ^ Lines of code | not relevant | | ||
^ Further doc | not relevant | | ^ Further doc | not relevant | | ||
Line 584: | Line 495: | ||
^ Log file | ? | ^ Log file | ? | ||
^ Programming language | Perl | | ^ Programming language | Perl | | ||
- | ^ Lines of code | Approx 200 | | ||
^ Further doc | - | | ^ Further doc | - | | ||
Line 610: | Line 520: | ||
^ Log file | None | ^ Log file | None | ||
^ Programming language | Python | | ^ Programming language | Python | | ||
- | ^ Lines of code | Approx 350 | | + | ^ Further doc | [[http://nav.uninett.no/ |
- | ^ Further doc | [[http://metanav.uninett.no/ | + | |
==== Details ==== | ==== Details ==== | ||
Line 632: | Line 541: | ||
^ Default scheduling | ^ Default scheduling | ||
^ Programming language | | | ^ Programming language | | | ||
- | ^ Lines of code | | | ||
^ Further doc | [[Arnold|Arnold]] | | ^ Further doc | [[Arnold|Arnold]] | | ||
backendprocesses.1207217787.txt.gz · Last modified: by bredal