User Tools

Site Tools


devel:blueprints:ipdevpoll

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
devel:blueprints:ipdevpoll [2009/07/02 14:08]
klette
devel:blueprints:ipdevpoll [2012/05/07 11:26] (current)
morten fix url
Line 1: Line 1:
 ====== ipdevpoll design specification ====== ====== ipdevpoll design specification ======
 +
 +**This page is currently being updated to match whats being developed. Dont trust this stuff just yet :-)**
 +
 ===== Introduction ===== ===== Introduction =====
 //​ipdevpoll//​ is the planned replacement for getDeviceData,​ becoming the third generation SNMP polling framework for NAV.  This page will lay out the various design ideas and specifications for the new program. //​ipdevpoll//​ is the planned replacement for getDeviceData,​ becoming the third generation SNMP polling framework for NAV.  This page will lay out the various design ideas and specifications for the new program.
Line 7: Line 10:
   * Asynchronous SNMP polling, built with [[http://​twistedmatrix.com/​trac/​|Twisted]] and [[http://​twistedsnmp.sourceforge.net/​|TwistedSNMP]]   * Asynchronous SNMP polling, built with [[http://​twistedmatrix.com/​trac/​|Twisted]] and [[http://​twistedsnmp.sourceforge.net/​|TwistedSNMP]]
   * Plugin based architecture   * Plugin based architecture
-  * Data persistence based on Twisted aDBI? (FIXME) 
  
 ==== Core application ==== ==== Core application ====
  
-The gDD core application has the following basic tasks:+The core application has the following basic tasks:
  
   * Schedule executing of jobs   * Schedule executing of jobs
-  * Execute ​plugins ​contained in job in correct order +  * Execute ​plug-ins ​contained in job in correct order 
-  * Handle rescheduling when a plugin signals that previous state has been invalidated +  * Provide data persistence to plug-ins
-  * Provide data persistence to plugins+
  
-Scheduling of jobs instead of individual plugins has been chosen to simplify the application due to the high level of dependency between some of the plugins. Each job defined in the the gDD configuration defines ​set of plugins that are to be run at a given interval. The scheduling ​of these intervals ​is handled ​using Twisted'​s built in scheduling mechanisms.+Jobs are group of plug-ins ​run in a specified order, and scheduled ​at a specified time interval. The scheduling is handled ​by Twisted
 +NAV provide two jobs in the default configuration, ​''​Inventory''​ and ''​Logging''​. 
 +These provide a sane order for the plug-ins to run. The user may specify their own jobs and schedules ​in ''​jobs.conf''​. The configuration files used by ipdevpoll follows the syntax 
 +demanded by the python ConfigParser-module. 
 +\\  
 +\\  
 +Example jobs.conf 
 +<code ini> 
 +[inventory] 
 +interval: 60s 
 +plugins: 
 + ​typeoid 
 + ​dnsname 
 + ​interfaces 
 + ​vlan 
 + ​prefix 
 +</​code>​
  
-The previous version ​of gDD determined plugin execution order based on numeric values assigned to each plugin, ​this implies that every plugin must know about the value of all other plugins to be sure that it gets executed before and/or after relevant plugins. To avoid this error prone method ​of ordering using a simple dependence system between plugins has been suggested. Each plugin needs to know which plugins need to run before itself. This data can be used to do a topological sort based on the dependence arcs between plugins.+One advantage ​of this way of doing things is easy deployment ​of several ipdevpolld-processes,​ either ​on the same host or distributed over several machines.
  
 ==== Data ==== ==== Data ====
  
-Data storage will to a certain degree be modeled around ​the existing gDD data-plugins. However instead ​of creating a plugin architecture for this we will simply be hardcoding the data-models ​that we needIn proposed implementation the plugins simply create and populate data objects with as much information as they have. Upon creation these objects are added +Data is provided through ​the use of shadow-classes ​of django-models. ​Plugins never write django-objects ​directlyThis is handled by the job the plugin runs in at the end of each job.
-to a class-level array of initiated ​objects. ​Once all plugins complete ​the data-models are told to process ​the data+
  
-  ​Combine data from all instances ​that map to same database ​object. +Shadow-classes are created in ''​nav.ipdevpoll.shadows''​ 
-  Retrieve current data in db+\\  
-  Update if necessary+Example shadowclass specification 
-  Flush global ​instance storage for model.+<code python>​ 
 +class Interface(Shadow):​ 
 +    __shadowclass__ = manage.Interface 
 +    __lookups__ = [('​netbox',​ '​ifname'​),​ ('​netbox',​ '​ifindex'​)] 
 +</​code>​ 
 + 
 +The Shadow-class has two important attributes. ''<​nowiki>​__shadowclass__</​nowiki>''​ specifies 
 +which django model to mimic. This will make any instances ​of the shadowclass 
 +answers ​to the same attributes as the django model. When a property is changed 
 +on the shadowclass,​ the class will set it state to touched. This is needed for 
 +determining wheter to update an object ​or not at the end of a runTo lookup 
 +existing objects in the database the storage system first tries the primary key 
 +field on the object. If the PK is None, the ''<​nowiki>​__lookup__</​nowiki>''​-attribute is used 
 +for the checking. The ''<​nowiki>​__lookup__</​nowiki>''​ attribute is a list of strings and tuples. 
 +Tuples specifies combined lookups, while strings specifies single field 
 +lookups. The different lookups are tried in the order they are defined in the 
 +listIf we look at the example above, it will first try to get an existing 
 +object with only the ''​interface.id''​-fieldIf its unavailable it tries the 
 +first object in the lookup-list, a combined search for objects matching both 
 +the ''​netbox''​ and ''​ifname''​ properties on the shadow ​instance. If 
 +unsuccessful it goes on to the next lookup field. 
 + 
 +There are two major "​gotcha"​s here. First of, the lookups must be unique. If 
 +multiple rows are returned an exception is raised. The second is primary key 
 +lookups where the primary key is not an AutoField; if no rows are found when 
 +using non-AutoField primary keys, a new object is created. If the field is an 
 +AutoField however, and no rows are returned, an exception is raised. 
 + 
 +Deletion of objects in the database is done if the shadowobject has its 
 +''​delete''​-property set to True. 
 + 
 +Once the job is done the storage ​routine is called. The storage system then parses all 
 +the shadow objects contained in ''​job_handler.container''​ and make s sure all foreignkeys needed ​for the storing  
 +an instance are saved before the object it self
  
 ==== Plugins ==== ==== Plugins ====
Line 63: Line 112:
 prefer to use the MIB-2 standard mibs for collecting information,​ falling back to vendor specific mibs. prefer to use the MIB-2 standard mibs for collecting information,​ falling back to vendor specific mibs.
 To maximize the amount of easily testable code, all parsing logic should be extracted into their own methods and not To maximize the amount of easily testable code, all parsing logic should be extracted into their own methods and not
-be entangled into the collection process (methods that returns deferred objects). ​Only IO-bound methods should be done +be entangled into the collection process (methods that returns deferred objects). ​Remember to call the processing 
-asynchronously ​within the plugin itself.+methods ​asynchronously ​as well
  
   /​nav/​ipdevpoll/​plugins   /​nav/​ipdevpoll/​plugins
     - /iftable     - /iftable
       - iftable.py       - iftable.py
 +      - test_iftable.py
       - collector.py       - collector.py
-      - /test 
-        - tests.py 
       - /vendor       - /vendor
         - cisco.py         - cisco.py
 +        - test_cisco.py
         - alcatel.py         - alcatel.py
 +        - test_alcatel.py
         - ..         - ..
                
Line 140: Line 190:
  
   * ModuleMon ​   * ModuleMon ​
 +
 +=== Implemented plugins ===
 +
 +Already implemented plugins can be found on [[devel:​ipdevpoll:​plugins]]
  
 ===== Database changes ===== ===== Database changes =====
Line 157: Line 211:
 The IF-MIB has no concept of an interface/​module relationship. ​ Information about such a relationship will most likely come from proprietary MIBs (or possibly a properly populated ENTITY-MIB),​ and should be treated as a bonus if found. ​ Ie. NAV's model should add to the interface table a mandatory foreign key to netbox, ​ and only an optional foreign key into the module table. The IF-MIB has no concept of an interface/​module relationship. ​ Information about such a relationship will most likely come from proprietary MIBs (or possibly a properly populated ENTITY-MIB),​ and should be treated as a bonus if found. ​ Ie. NAV's model should add to the interface table a mandatory foreign key to netbox, ​ and only an optional foreign key into the module table.
  
-===== Code =====+===== Status ​=====
  
-The mercurial repositories can be found at http://​metanav.uninett.no/​hg/​features/​ipdevpoll/​+The first NAV release to include ipdevpoll was version 3.6.
  
 ===== References ===== ===== References =====
devel/blueprints/ipdevpoll.1246543693.txt.gz · Last modified: 2009/07/02 14:08 by klette