NAV Wiki

User Tools

Site Tools

Trace: • backendprocesses
backendprocesses

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
backendprocesses [2011/04/03 18:23] – [Key information] faltinbackendprocesses [2015/09/24 12:48] (current) – warn of obsolesence morten
Line 1: Line 1:
 ====== Back-end processes in NAV ====== ====== Back-end processes in NAV ======
 +
 +<note warning>This page has not been updated for several years, and may be quite outdated for NAV 4</note>
  
 NAV has a number of back-end processes. This page attempts to give an overview of them. NAV has a number of back-end processes. This page attempts to give an overview of them.
  
-The following figure used to complements this document, but is getting a bit outdated (it hasn't been updated since NAV 3.2). +{{:backend-processes-3.10.png?800|The NAV processes}}
-Note: getDeviceData is now replaced with ipdevpoll. +
- +
-{{architecture1.png?800|The NAV processes}} +
- +
- +
- +
- +
  
 ===== nav list / nav status ===== ===== nav list / nav status =====
Line 26: Line 20:
    * [[#mactrace|mactrace]]    * [[#mactrace|mactrace]]
    * [[#maintengine|maintengine]]    * [[#maintengine|maintengine]]
-   * [[#networkdiscovery_topology|networkDiscovery]] (physical and [[#networkdiscovery_vlan|vlan discovery]]) 
    * [[#pping|pping]]    * [[#pping|pping]]
 +   * [[#psuwatch|psuwatch]]
    * [[#servicemon|servicemon]]    * [[#servicemon|servicemon]]
    * [[#smsd|smsd]]    * [[#smsd|smsd]]
-   * [[#thresholdmon|thresholdMon]] 
    * [[#snmptrapd|snmptrapd]]    * [[#snmptrapd|snmptrapd]]
 +   * [[#thresholdmon|thresholdMon]]
 +   * [[#topology|topology]]
  
  
Line 79: Line 74:
 ^ Log file             | getBoksMacs.log | ^ Log file             | getBoksMacs.log |
 ^ Programming language | Java | ^ Programming language | Java |
-^ Further doc          | [[http://metanav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 2.1]] (Norwegian), [[http://metanav.uninett.no/static/reports/tigaNAV.pdf|tigaNAV report ch 5.4.5 and ch 5.5.3]] |+^ Further doc          | [[http://nav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 2.1]] (Norwegian), [[http://nav.uninett.no/static/reports/tigaNAV.pdf|tigaNAV report ch 5.4.5 and ch 5.5.3]] |
  
  
Line 112: Line 107:
  
 One notable improvement is the addition of the interface field in the swport table. It is used for matching the CDP remote interface, and makes this matching much more reliable. Also, both the cam and the swp_netbox tables now use netboxid and ifindex to uniquely identify a swport port instead of the old netboxid, module, port-triple. This has significantly simplified swport port matching, and especially since the old module field of swport was a shortened version of what is today the interface field, reliability has increased as well. One notable improvement is the addition of the interface field in the swport table. It is used for matching the CDP remote interface, and makes this matching much more reliable. Also, both the cam and the swp_netbox tables now use netboxid and ifindex to uniquely identify a swport port instead of the old netboxid, module, port-triple. This has significantly simplified swport port matching, and especially since the old module field of swport was a shortened version of what is today the interface field, reliability has increased as well.
-- 
- 
-===== networkDiscovery_topology ===== 
- 
- 
- 
  
 +===== topology =====
 ==== Key information ==== ==== Key information ====
  
-^ Process name         | networkDiscovery.sh topology| +^ Process name         | navtopology | 
-^ Alias                | Physical Topology Builder |+^ Alias                | Physical and VLAN Topology Builder |
 ^ Polls network        | No | ^ Polls network        | No |
-^ Brief description    | Builds the physical topology of the network; i.e. which netbox is connected to which netbox. | +^ Brief description    | Builds NAV's model of the physical network topology as well as the VLAN sub-topologies | 
-^ Depends upon         | mactrace fills data in swp_netbox representing the candidate list of physical neighborship. This is the data that the physical topology builder uses.|+^ Depends upon         | mactrace fills data in ''swp_netbox'', representing the list of physical neighbor candidates. This is the data that the physical topology builder uses. |
 ^ Updates tables       | Sets the to_netboxid and to_swportid fields in the swport and gwport tables. | ^ Updates tables       | Sets the to_netboxid and to_swportid fields in the swport and gwport tables. |
 ^ Run mode             | cron | ^ Run mode             | cron |
 ^ Default scheduling   | every hour (35 * * * *) | ^ Default scheduling   | every hour (35 * * * *) |
 ^ Config file          | None |  ^ Config file          | None | 
-^ Log file             | networkDiscovery/networkDiscovery-topology.html og networkDiscovery/networkDiscovery-stderr.log   |  +^ Log file             | navtopology.log |  
-^ Programming language | Java | +^ Programming language | Python |
-^ Further doc          | [[http://metanav.uninett.no/static/reports/tigaNAV.pdf|tigaNAV report ch 5.5.4]] | +
  
 ==== Details ==== ==== Details ====
  
-FIXME This is cut and paste from the tigaNAV report. Consider a rewrite.+=== Physical topology ===
  
 +The topology discovery system builds NAV's view of the network topology based
 +on cues from information collected previously via SNMP.  
  
 +The information cues come from routers' IPv4 ARP caches and IPv6 Neighbor
 +Discovery caches, interface physical (MAC) addresses, switch forwarding tables
 +and CDP (Cisco Discovery Protocol).  The mactrace process has already
 +pre-parsed these cues and created a list of neighbor candidates for each port
 +in the network.
  
-The network topology discovery system automatically discovers the physical topology of the network monitored by NAV based on the data in the swp_netbox table collected by the cam logger. No major updates have been necessary except for adjustment to the new structure of the NAVdb; the basic algorithm remains the same. While the implementation of said algorithm is somewhat complicated as to gracefully handle missing data, the following is a simplified description:+The physical topology detection algorithm is responsible for reducing the list 
 +of neighbor candidates of each port to just one single device.
  
-   * We start with a candidate list for each swport port. These are the switches located behind a switch port and the goal of the algorithm is to pick the one to which it is connected directly. Some of the candidate lists, those of the switches one level up from the edge, will contain only one candidate. We can thus pick this as the switch directly connected and proceed to remove said switches from all other lists. After this removal there will be more candidate lists with only one candidate, and we can apply the same procedure again. +In practice the use of CDP makes this process very reliable for the devices 
-   * If we have the complete information about the network we could now simply iterate until all candidate lists were empty; however, to deal with missing information we sometimes have to make an educated guess of which is the directly connected switch. The network topology discover system makes the guess by looking at how far each candidate is from the router and how many switches are connected below them, and then try to pick the one which most closely matches the current switch.+supporting it, and this makes it easier to correctly determine the remaining 
 +topology even in the case of missing information.  CDP is, however, not 
 +trusted more than switch forwarding tables, as CDP packets may pass unaltered 
 +through switches that don't support CDP, causing CDP data to be inaccurate.
  
-In practice the use of CDP makes this process very reliable for the devices supporting it, and this makes it easier to correctly determine the remaining topology even in the case of missing information.+=== VLAN topology ===
  
-===== networkDiscovery_vlan =====+After the physical topology model of the network has been built, the logical 
 +topology of the VLANs still remains.  Since modern switches support 802.1Q 
 +trunking, which can transport several independent VLANs over a single physical 
 +link, the logical topology can be non-trivial and indeed, in practice it 
 +usually is.
  
 +The vlan discovery system uses a simple top-down depth-first graph traversal
 +algorithm to discover which VLANs are actually running on the different trunks
 +and in which direction. Direction is here defined relative to the router port,
 +which is the top of the tree, currently owning the lowest gateway IP or the
 +virtual IP in the case of HSRP.  Re-use of VLAN numbers in physicallyq disjoint
 +parts of the network is supported.
  
-==== Key information ==== +The VLAN topology detector does not currently support mapping unrouted VLANs.
-^ Process name         | networkDiscovery.sh vlan| +
-^ Alias                | Vlan Topology Builder | +
-^ Polls network        | No  | +
-^ Brief description    | Builds the per vlan topology on the swithed network with interconnected trunks. The algorithm is a top-down depth-first traversel starting at the primary router port for the vlan. | +
-^ Depends upon         | The physical topology need to be in place, this process therefore supersedes the physical topology builder.| +
-^ Updates tables       | swportvlan | +
-^ Run mode             | cron | +
-^ Default scheduling   | every hour (38 * * * *) | +
-^ Config file          | None |  +
-^ Log file             | networkDiscovery/networkDiscovery-vlan.html og networkDiscovery/networkDiscovery-stderr.log   |  +
-^ Programming language | Java | +
-^ Further doc          | [[http://metanav.uninett.no/static/reports/tigaNAV.pdf|tigaNAV report ch 5.5.5]] | +
- +
- +
-==== Details ==== +
- +
-FIXME This is cut and paste from the tigaNAV report. Consider a rewrite. +
- +
-After the physical topology of the network has been mapped by the network topology discover system it still remains to explore the logical topology, or the VLANs. Since modern switches support trunking, which can transport several independent VLANs over a single physical link, the logical topology can be non-trivial and indeed, in practice it usually is. +
- +
-The vlan discovery system uses a simple top-down depth-first graph traversal algorithm to discover which VLANs are actually running on the different trunks and in which direction. Direction is here defined relative to the router port, which is the top of the tree, currently owning the lowest gateway IP or the virtual IP in the case of HSRP. In addition, since NAV v3 now fully supports the reuse of VLAN numbers, the vlan discovery system will also make the connection from VLAN number to actual vlan as defined in the vlan table for all non-trunk ports it encounters. +
- +
-A special case are //closed// VLANs which do not have a gateway IP; the vlan discovery system will still traverse these VLANs without setting any direction and also creating a new VLAN record in the vlan table. The NAV administrator can fill inn descriptive information afterward if desired. +
- +
-The implementation of this subsystem is again complicated by factors such as the need for checking at both ends of a trunk if the VLAN is allowed to traverse it, the fact that VLAN numbers on each end of non-trunk links need not match (the number closer to the top of the tree should then be given precedence and the lower VLAN numbers rewritten to match), that both trunks and non-trunks can be blocked (again at either end) by the spanning tree protocol and of course that it needs to be highly efficient and scalable in the case of large networks with thousands of switches and tens of thousands of switch ports.+
  
 ====== Monitoring the network ====== ====== Monitoring the network ======
Line 196: Line 180:
 ^ Log file             | pping.log   |  ^ Log file             | pping.log   | 
 ^ Programming language | Python | ^ Programming language | Python |
-^ Further doc          | See below, based on and translated from [[http://metanav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 3.4]] (Norwegian) |+^ Further doc          | See below, based on and translated from [[http://nav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 3.4]] (Norwegian) |
  
  
Line 279: Line 263:
 ^ Log file             | servicemon.log   |  ^ Log file             | servicemon.log   | 
 ^ Programming language | Python | ^ Programming language | Python |
-^ Further doc          | See the [[servicemon]] page and/or [[http://metanav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 3.5]] (Norwegian) |+^ Further doc          | See the [[servicemon]] page and/or [[http://nav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 3.5]] (Norwegian) |
  
 ==== Details ==== ==== Details ====
Line 326: Line 310:
 ^ Log file             | eventEngine.log   |  ^ Log file             | eventEngine.log   | 
 ^ Programming language | Java | ^ Programming language | Java |
-^ Further doc          | [[http://metanav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 3.6]] (Norwegian). Updates in [[http://metanav.uninett.no/static/reports/tigaNAV.pdf|tigaNAV report ch 4.3.1]]. |+^ Further doc          | [[http://nav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 3.6]] (Norwegian). Updates in [[http://nav.uninett.no/static/reports/tigaNAV.pdf|tigaNAV report ch 4.3.1]]. |
  
 ==== Details ==== ==== Details ====
Line 349: Line 333:
 ^ Log file             | maintengine.log   |  ^ Log file             | maintengine.log   | 
 ^ Programming language | Python | ^ Programming language | Python |
-^ Further doc          | Old doc: [[http://metanav.uninett.no/static/reports/tigaNAV.pdf|tigaNAV report ch 8]]. The maintenance system was rewritten for NAV 3.1. See [[devel:tasklist2006#t3rewrite_the_message_and_maintenance_tool|here]] for more. |+^ Further doc          | Old doc: [[http://nav.uninett.no/static/reports/tigaNAV.pdf|tigaNAV report ch 8]]. The maintenance system was rewritten for NAV 3.1. See [[devel:tasklist2006#t3rewrite_the_message_and_maintenance_tool|here]] for more. |
  
 ==== Details ==== ==== Details ====
Line 369: Line 353:
 ^ Config file          | alertengine.cfg |  ^ Config file          | alertengine.cfg | 
 ^ Log file             | alertengine.log og alertengine.err.log   |  ^ Log file             | alertengine.log og alertengine.err.log   | 
-^ Programming language | perl | +^ Programming language | Python | 
-^ Further doc          | [[http://metanav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 3.7 and 3.8]] (Norwegian). |+^ Further doc          | [[http://nav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 3.7 and 3.8]] (Norwegian). |
  
 ==== Details ==== ==== Details ====
Line 432: Line 416:
  
  
-===== The snmptrapd =====+===== snmptrapd =====
  
  
Line 466: Line 450:
 ^ Polls network        | No | ^ Polls network        | No |
 ^ Brief description    | | ^ Brief description    | |
-^ Depends upon         | That gDD has filled the gwport, swport tables (and more...) |+^ Depends upon         | That ipdevpoll has filled the gwport, swport tables (and more...) |
 ^ Updates tables       | The RRD database (rrd_file and rrd_datasource) | ^ Updates tables       | The RRD database (rrd_file and rrd_datasource) |
 ^ Run mode             | cron | ^ Run mode             | cron |
Line 536: Line 520:
 ^ Log file             | None   |  ^ Log file             | None   | 
 ^ Programming language | Python | ^ Programming language | Python |
-^ Further doc          | [[http://metanav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 2.4]] (Norwegian). |+^ Further doc          | [[http://nav.uninett.no/static/reports/NAVMore.pdf|NAVMore report ch 2.4]] (Norwegian). |
  
 ==== Details ==== ==== Details ====
backendprocesses.1301855008.txt.gz · Last modified: by faltin
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki