NAV Wiki

User Tools

Site Tools

Trace:
backendprocesses

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
backendprocesses [2007/10/10 16:53]
faltin 		backendprocesses [2011/04/08 14:06]
faltin [Key information]
Line 1: Line 1:
 ====== Back-end processes in NAV ====== ====== Back-end processes in NAV ======
  
-NAV has a number of back-end processes. This document gives an overview, listing key information and  +NAV has a number of back-end processes. This page attempts to give an overview ​of them.
-detailed description for each process. We also give references to documentation found elsewhere on metaNAV.+
  
- +The figure ​below complements this document. Unfortunately ​the figure is getting a bit outdated (it hasn't been updated since NAV 3.2): 
- +  * getDeviceData ​is now replaced with ipdevpoll 
-The following ​figure complements this document ​(the NAV 3.3 snmptrapd ​is not included in the figure):+  * iptrace does not exist anymore (the job is done by ipdevpoll) 
 +  * snmptrapd is a new daemon process
  
 {{architecture1.png?​800|The NAV processes}} {{architecture1.png?​800|The NAV processes}}
Line 24: Line 24:
    * [[backendprocesses#​collecting_statistics|cricket]] (includes makecricketConfig,​ Cricket collector and cleanrrds)    * [[backendprocesses#​collecting_statistics|cricket]] (includes makecricketConfig,​ Cricket collector and cleanrrds)
    * [[#​eventengine|eventengine]]    * [[#​eventengine|eventengine]]
-   * [[#getdevicedata|getDeviceData]] +   * [[#ipdevpoll|ipdevpoll]]
-   * [[#​iptrace|iptrace]]+
    * [[#​logengine|logengine]]    * [[#​logengine|logengine]]
    * [[#​mactrace|mactrace]]    * [[#​mactrace|mactrace]]
Line 39: Line 38:
 ====== Building the network model ====== ====== Building the network model ======
  
-===== getDeviceData ​===== +===== ipdevpoll ​=====
- +
- +
  
 ==== Key information ==== ==== Key information ====
  
-^ Process name         ​| ​getDeviceData ​ | +^ Process name         ​| ​ipdevpoll ​|
-^ Alias                | gDD / the snmp data collector  ​|+
 ^ Polls network ​       | Yes            | ^ Polls network ​       | Yes            |
 ^ Brief description ​   | Collects SNMP data from equipment in the netbox table and stores data regarding the equipment in a number of tables. Does not build topology. | ^ Brief description ​   | Collects SNMP data from equipment in the netbox table and stores data regarding the equipment in a number of tables. Does not build topology. |
-^ Depends upon         | Seed data must be filled in the netbox table, ​either by the [[seedessentials|Edit Database tool]] ​or by the autodiscovery contrib || +^ Depends upon         | Seed data must be filled in the netbox table, ​using the [[seedessentials|Seed Database tool]] | 
-^ Updates tables ​      | netbox, netboxsnmpoid,​ netboxinfo, device, module, gwport, gwportprefix,​ prefix, vlan, swport, swportallowedvlan, netbox_vtpvlan ​| +^ Updates tables ​      | netbox, netboxsnmpoid,​ netboxinfo, device, module, gwportprefix,​ prefix, vlan, interface, swportallowedvlan | 
-^ Run mode             | Daemon process. Thread based. ​| +^ Run mode             | Daemon process | 
-^ Default scheduling ​  ​| ​Initial data collection for new netboxes ​is done every 5 minutes. Update polls on existing netboxes ​is done every 6 hrs. Collection of certain OIDs for the netbox may deviate from this interval; i.e. the moduleMon OID is polled every hour. | +^ Default scheduling ​  ​| ​Polling ​is organized into jobs in ''​ipdevpoll.conf'',​ so is job scheduling. | 
-^ Config file | getDeviceData.conf | +^ Config file | ipdevpoll.conf | 
-^ Log files | getDeviceData.log og getDeviceData/​getDeviceData-stderr.log |  +^ Log files | ipdevpoll.log |  
-^ Programming language | Java | +^ Programming language | Python ​| 
-^ Lines of code        | Approx 8200 | +^ Further doc          | |
-^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​tigaNAV.pdf|tigaNAV report chapter 5]]|+
  
  
Line 65: Line 59:
 ==== Details ==== ==== Details ====
  
-  * Initial OID classification ​\\ When gDD detects a new box that has a valid snmp read community (regardless of category), he will start initial OID classifiation. This is done by testing the netbox against all OIDs in the snmpoid table and in turn populating the netboxsnmpoid table. Testing is done based on attributes in snmpoid table, see reference to further doc for details. Frequency will be set based on the snmpoid.defaultfreq. +  * jobs and plugins ​\\ All ipdevpoll'​s work is done by plugins.  ​Plugins are organized into jobs, and jobs are scheduled ​for each active IP device individually. 
- +  * inventory job \\ Polls for inventory information every 6 hours (by default).  ​Inventory information includes interfaces, serial numbers, modules, VLANs and prefixes. 
-  * Plugin-based architecture ​\\ gDD has a plugin based architecture. Plugins fall into two types; device plugins and data plugins:  +  * profiler job \\ Runs every 5 minutes, profiling devices if deemed necessary.  ​NAV ​has an internal list of SNMP OIDs that are tested for compatibility with each device.  ​This ​is used to create ​a sort of profile that says what the device supports ​- the profile is typically used to produce a Cricket configuration that will collect statistics ​from proprietary OIDs. 
-     * Device plugins collects data with SNMP. Each device plugin is geared towards a particular type of equipment, supporting a particular subset of OIDs. See further doc for details. ​  +  * logging job \\ Runs every 30 minutes ​and collects ​log-like information from devices.  ​At the time being, only the arp plugin runs, collecting ARP caches from routers.  ​ARP data is logged ​to a table, ​and aids in topology detection and client machine tracking.
-     * Data plugins updates NAVdb with data fed from the device plugins. A particular data plugin is responsible for a particular table (or set of tables) in the database. See further doc for details. +
- +
-  * Module monitor ​\\ The module monitor is a data plugin within gDD. It has the dedicated function ​of detecting outage of modules in operating netboxes. When a module ​is detected down a moduleDown event is posted on the event queue (eventq). +
- +
- +
-===== iptrace ===== +
- +
- +
- +
- +
-==== Key information ==== +
- +
-^ Process name         | iptrace | +
-^ Alias                | IP-to-mac collector / arplogger| +
-^ Polls network ​       | Yes | +
-^ Brief description ​   | Collects arp data from routers and stores this information in the arp table. | +
-^ Depends upon         | The routers (GW / GSW) must be in the netbox table. To assign prefixes to arp entries, [[#​getDeviceData|getDeviceData]] must have done router data collection. | +
-^ Updates tables ​      | arp | +
-^ Run mode             | cron | +
-^ Default scheduling ​  ​| ​every 30 minutes ​(0,30 * * * *). No threads | +
-^ Config file          | pping.conf | +
-^ Log file             | pping.log | +
-^ Programming language | Perl| +
-^ Lines of code        | Approx 130 lines| +
-^ Further doc          | [[http://​metanav.uninett.no/​static/​reports//​NAVMe.pdf|NAVMe report ch 4.5.8]] (Norwegian) | +
- +
- +
- +
-==== Details ==== +
- +
-  * iptrace understands proxy arp and will not store arp entries that are "​false"​. +
-  * iptrace ignores routers that are known to be down (fixed in NAV 3.1). +
-  * The command line tool [[commandlinetools#​navclean.py|navclean.py]] offers ​a means of deleting old arp (and cam) entries.+
  
  
Line 113: Line 74:
 ^ Polls network ​       | Yes | ^ Polls network ​       | Yes |
 ^ Brief description ​   | Collects mac addresses behind switch table data for all switches (cat GSW, SW, EDGE). The process also checks for spanning tree blocked ports. | ^ Brief description ​   | Collects mac addresses behind switch table data for all switches (cat GSW, SW, EDGE). The process also checks for spanning tree blocked ports. |
-^ Depends upon         | [[#getDeviceData|getDeviceData]] must have created the swport tables for the switches. |+^ Depends upon         | [[#ipdevpoll|ipdevpoll]] must have created the swport tables for the switches. |
 ^ Updates tables ​      | cam (mac adresses), netboxinfo (CDP neighbors), swp_netbox (the candidate list for the physical topology builder), swportblocked (switch ports that are blocked by spannning for a given vlan). | ^ Updates tables ​      | cam (mac adresses), netboxinfo (CDP neighbors), swp_netbox (the candidate list for the physical topology builder), swportblocked (switch ports that are blocked by spannning for a given vlan). |
 ^ Run mode             | cron | ^ Run mode             | cron |
Line 120: Line 81:
 ^ Log file             | getBoksMacs.log | ^ Log file             | getBoksMacs.log |
 ^ Programming language | Java | ^ Programming language | Java |
-^ Lines of code        | Approx 1400 | 
 ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 2.1]] (Norwegian),​ [[http://​metanav.uninett.no/​static/​reports/​tigaNAV.pdf|tigaNAV report ch 5.4.5 and ch 5.5.3]] | ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 2.1]] (Norwegian),​ [[http://​metanav.uninett.no/​static/​reports/​tigaNAV.pdf|tigaNAV report ch 5.4.5 and ch 5.5.3]] |
  
Line 174: Line 134:
 ^ Log file             | networkDiscovery/​networkDiscovery-topology.html og networkDiscovery/​networkDiscovery-stderr.log ​  ​| ​ ^ Log file             | networkDiscovery/​networkDiscovery-topology.html og networkDiscovery/​networkDiscovery-stderr.log ​  ​| ​
 ^ Programming language | Java | ^ Programming language | Java |
-^ Lines of code        | Approx 1500 (shared with vlan topology builder) | 
 ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​tigaNAV.pdf|tigaNAV report ch 5.5.4]] | ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​tigaNAV.pdf|tigaNAV report ch 5.5.4]] |
  
Line 206: Line 165:
 ^ Log file             | networkDiscovery/​networkDiscovery-vlan.html og networkDiscovery/​networkDiscovery-stderr.log ​  ​| ​ ^ Log file             | networkDiscovery/​networkDiscovery-vlan.html og networkDiscovery/​networkDiscovery-stderr.log ​  ​| ​
 ^ Programming language | Java | ^ Programming language | Java |
-^ Lines of code        | See the physical topology builder above | 
 ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​tigaNAV.pdf|tigaNAV report ch 5.5.5]] | ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​tigaNAV.pdf|tigaNAV report ch 5.5.5]] |
  
Line 240: Line 198:
 ^ Log file             | pping.log ​  ​| ​ ^ Log file             | pping.log ​  ​| ​
 ^ Programming language | Python | ^ Programming language | Python |
-^ Lines of code        | Approx 4200, shared with servicemon | 
 ^ Further doc          | See below, based on and translated from [[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 3.4]] (Norwegian) | ^ Further doc          | See below, based on and translated from [[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 3.4]] (Norwegian) |
  
Line 324: Line 281:
 ^ Log file             | servicemon.log ​  ​| ​ ^ Log file             | servicemon.log ​  ​| ​
 ^ Programming language | Python | ^ Programming language | Python |
-^ Lines of code        | See pping above, shared code base | +^ Further doc          | See the [[servicemon]] page and/​or ​[[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 3.5]] (Norwegian) |
-^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 3.5]] (Norwegian) |+
  
 ==== Details ==== ==== Details ====
Line 347: Line 303:
 ^ Log file             | thresholdMon.log ​  ​| ​ ^ Log file             | thresholdMon.log ​  ​| ​
 ^ Programming language | Python | ^ Programming language | Python |
-^ Lines of code        | Approx 400 | 
 ^ Further doc          | See [[ThresholdMonitor]] | ^ Further doc          | See [[ThresholdMonitor]] |
  
Line 354: Line 309:
  
   * See [[ThresholdMonitor]]   * See [[ThresholdMonitor]]
- 
-===== moduleMon ===== 
- 
- 
-==== Key information ==== 
-^ Process name         | getDeviceData data plugin moduleMon | 
-^ Alias                | The module monitor | 
-^ Polls network ​       | Yes | 
-^ Brief description ​   | A plugin to gDD. A dedicated OID is polled. If this is a HP switch, a specific HP OID is used (oidkey hpStackStatsMemberOperStatus),​ similarly for 3Com (oidkey 3cIfMauType). For other equipment the genereric moduleMon OID is used. For 3com and HP the OID actually tells us if a module is down or not. For the generic test we (in lack of something better) check if an arbitrary ifindex on the module in question responds. If the module has no ports, no check is done.  | 
-^ Depends upon         | The switch or router to be processed by gDD with apropriate data in module and gwport/​swport. | 
-^ Updates tables ​      | posts moduleMon events on the eventq. Sets in addition the boolean module.up value. | 
-^ Run mode             | daemon, a part of gDD. | 
-^ Default scheduling ​  | Depends on the defaultfreq of the moduleMon OID (equivalently for the HP and 3com OIDs) Defaults to 1 hour. | 
-^ Config file          | see gDD |  
-^ Log file             | see gDD   ​| ​ 
-^ Programming language | Java | 
-^ Lines of code        | Part of gDD, see gDD. | 
-^ Further doc          | Not much. | 
  
  
Line 391: Line 328:
 ^ Log file             | eventEngine.log ​  ​| ​ ^ Log file             | eventEngine.log ​  ​| ​
 ^ Programming language | Java | ^ Programming language | Java |
-^ Lines of code        | Approx 3000 lines | 
 ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 3.6]] (Norwegian). Updates in [[http://​metanav.uninett.no/​static/​reports/​tigaNAV.pdf|tigaNAV report ch 4.3.1]]. | ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 3.6]] (Norwegian). Updates in [[http://​metanav.uninett.no/​static/​reports/​tigaNAV.pdf|tigaNAV report ch 4.3.1]]. |
  
Line 415: Line 351:
 ^ Log file             | maintengine.log ​  ​| ​ ^ Log file             | maintengine.log ​  ​| ​
 ^ Programming language | Python | ^ Programming language | Python |
-^ Lines of code        | Approx 300 | 
 ^ Further doc          | Old doc: [[http://​metanav.uninett.no/​static/​reports/​tigaNAV.pdf|tigaNAV report ch 8]]. The maintenance system was rewritten for NAV 3.1. See [[devel:​tasklist2006#​t3rewrite_the_message_and_maintenance_tool|here]] for more. | ^ Further doc          | Old doc: [[http://​metanav.uninett.no/​static/​reports/​tigaNAV.pdf|tigaNAV report ch 8]]. The maintenance system was rewritten for NAV 3.1. See [[devel:​tasklist2006#​t3rewrite_the_message_and_maintenance_tool|here]] for more. |
  
Line 436: Line 371:
 ^ Config file          | alertengine.cfg |  ^ Config file          | alertengine.cfg | 
 ^ Log file             | alertengine.log og alertengine.err.log ​  ​| ​ ^ Log file             | alertengine.log og alertengine.err.log ​  ​| ​
-^ Programming language | perl | +^ Programming language | Python ​|
-^ Lines of code        | Approx 1900 |+
 ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 3.7 and 3.8]] (Norwegian). | ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 3.7 and 3.8]] (Norwegian). |
  
Line 445: Line 379:
  
 ===== smsd ===== ===== smsd =====
 +
 +
  
  
Line 451: Line 387:
 ^ Alias                | The SMS daemon | ^ Alias                | The SMS daemon |
 ^ Polls network ​       | No | ^ Polls network ​       | No |
-^ Brief description ​   | Checks the sms queue for new messages, formats the messages into one SMS and dispatches it via one or more dispatchers with a general interface. Support for multiple dispatchers are handled by a dispatcher handler layer. | +^ Brief description ​   | Checks the navprofiles.smsq table for new messages, formats the messages into one SMS and dispatches it via one or more dispatchers with a general interface. Support for multiple dispatchers are handled by a dispatcher handler layer. | 
-^ Depends upon         | alertEngine fills the smsq |+^ Depends upon         | alertEngine fills the navprofiles.smsq table |
 ^ Updates tables ​      | Updates the sent and timesent values of navprofiles.smsq | ^ Updates tables ​      | Updates the sent and timesent values of navprofiles.smsq |
 ^ Run mode             | Daemon process | ^ Run mode             | Daemon process |
 ^ Default scheduling ​  | Polls the sms queue every x minutes | ^ Default scheduling ​  | Polls the sms queue every x minutes |
 ^ Config file          | smsd.conf |  ^ Config file          | smsd.conf | 
-^ Log file             | smsd.log ​  ​|  +^ Log file             | smsd.log |  
-^ Programming language | Python ​ in NAV 3.2 (perl in 3.1) | +^ Programming language | Python (Perl in 3.1) | 
-^ Lines of code        ​| In NAV 3.2: approx 1200 | +^ Further doc          ​| subsystem/​smsd/​README in the NAV sources describes the available dispatchers and more | 
-^ Further doc          ​| - |+ 
 + 
 + 
 + 
 +==== Details ==== 
 + 
 + 
 +=== Usage === 
 + 
 +As described when given the ''​-''''​–help''​ argument: 
 + 
 +  Usage: smsd [-h] [-c] [-d sec] [-t phone no.] 
 +   
 +    -h, --help ​           Show this help text 
 +    -c, --cancel ​         Cancel (mark as ignored) all unsent messages 
 +    -d, --delay ​          Set delay (in seconds) between queue checks 
 +    -t, --test ​           Send a test message to <phone no.> 
 + 
 +Especially note the ''​-''''​-test''​ option, which is useful for debugging when experiencing problems with smsd. 
 + 
 + 
 +=== Configuration === 
 + 
 +The configuration file smsd.conf lets you configure the following: 
 + 
 +^ parameter ​ ^ description ​                                    ^ default ​  ^ 
 +| username ​  | System user the process should try to run as    | navcron ​  | 
 +| delay      | Delay in seconds between queue runs             | 30        | 
 +| autocancel | Automatically cancel all messages older than '​autocancel',​ 0 to disable. Format like the PostgreSQL interval type, e.g. '1 day 12 hours'​. | 0 | 
 +| loglevel ​  | Filter level for log messages. Valid options are DEBUG, INFO, WARNING, ERROR, CRITICAL | INFO | 
 +| mailwarnlevel | Filter level for log messages sent by mail.  | ERROR     | 
 +| mailserver | Mail server to send log messages via.           | localhost | 
 +| dispatcherretry | Time, in seconds, before a dispatcher is retried after a failure | 300 | 
 +| dispatcherN | Dispatchers in prioritized order. Cheapest first, safest last. N should be 1,2,3,... | dispatcher1 defaults to GammuDispatcher | 
 + 
 +In addition, some dispatchers need extra configuration as described in comments in the config file.
  
  
-===== The snmptrapd =====+===== snmptrapd =====
  
  
Line 481: Line 452:
 ^ Log file             | snmptrapd.log and snmptraps.log ​ |  ^ Log file             | snmptrapd.log and snmptraps.log ​ | 
 ^ Programming language | Python ​  | ^ Programming language | Python ​  |
-^ Lines of code        | Approx 200 + traphandlers | 
 ^ Further doc          | - | ^ Further doc          | - |
  
Line 498: Line 468:
 ^ Polls network ​       | No | ^ Polls network ​       | No |
 ^ Brief description ​   | | ^ Brief description ​   | |
-^ Depends upon         | That gDD has filled the gwport, swport tables (and more...) |+^ Depends upon         | That ipdevpoll ​has filled the gwport, swport tables (and more...) |
 ^ Updates tables ​      | The RRD database (rrd_file and rrd_datasource) | ^ Updates tables ​      | The RRD database (rrd_file and rrd_datasource) |
 ^ Run mode             | cron | ^ Run mode             | cron |
Line 504: Line 474:
 ^ Config file          | None |  ^ Config file          | None | 
 ^ Log file             | cricket-changelog ​  ​| ​ ^ Log file             | cricket-changelog ​  ​| ​
-^ Programming language | perl | +^ Programming language | python ​|
-^ Lines of code        | Approx 1600 |+
 ^ Further doc          | [[howtoconfigurecricket|How to configure Cricket addons in NAV v3]] | ^ Further doc          | [[howtoconfigurecricket|How to configure Cricket addons in NAV v3]] |
  
Line 526: Line 495:
 ^ Log file             | cricket/​giga.log og cricket/​normal.log ​  ​| ​ ^ Log file             | cricket/​giga.log og cricket/​normal.log ​  ​| ​
 ^ Programming language | not relevant | ^ Programming language | not relevant |
-^ Lines of code        | not relevant | 
 ^ Further doc          | not relevant | ^ Further doc          | not relevant |
  
Line 545: Line 513:
 ^ Log file             | ?   ​| ​ ^ Log file             | ?   ​| ​
 ^ Programming language | Perl | ^ Programming language | Perl |
-^ Lines of code        | Approx 200 | 
 ^ Further doc          | - | ^ Further doc          | - |
  
Line 571: Line 538:
 ^ Log file             | None   ​| ​ ^ Log file             | None   ​| ​
 ^ Programming language | Python | ^ Programming language | Python |
-^ Lines of code        | Approx 350 | 
 ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 2.4]] (Norwegian). | ^ Further doc          | [[http://​metanav.uninett.no/​static/​reports/​NAVMore.pdf|NAVMore report ch 2.4]] (Norwegian). |
  
Line 593: Line 559:
 ^ Default scheduling ​  | | ^ Default scheduling ​  | |
 ^ Programming language | | ^ Programming language | |
-^ Lines of code        | | 
 ^ Further doc          | [[Arnold|Arnold]] | ^ Further doc          | [[Arnold|Arnold]] |
  
backendprocesses.txt · Last modified: 2015/09/24 12:48 by morten

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki