devel:blueprints:ipdevpoll
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
devel:blueprints:ipdevpoll [2008/12/11 09:11] thomaska Don't use ordered list |
devel:blueprints:ipdevpoll [2009/07/13 13:12] eide |
||
|---|---|---|---|
| Line 22: | Line 22: | ||
| The previous version of gDD determined plugin execution order based on numeric values assigned to each plugin, this implies that every plugin must know about the value of all other plugins to be sure that it gets executed before and/or after relevant plugins. To avoid this error prone method of ordering using a simple dependence system between plugins has been suggested. Each plugin needs to know which plugins need to run before itself. This data can be used to do a topological sort based on the dependence arcs between plugins. | The previous version of gDD determined plugin execution order based on numeric values assigned to each plugin, this implies that every plugin must know about the value of all other plugins to be sure that it gets executed before and/or after relevant plugins. To avoid this error prone method of ordering using a simple dependence system between plugins has been suggested. Each plugin needs to know which plugins need to run before itself. This data can be used to do a topological sort based on the dependence arcs between plugins. | ||
| - | FIXME data persistence/containers | + | ==== Data ==== |
| + | |||
| + | Data storage will to a certain degree be modeled around the existing gDD data-plugins. However instead of creating a plugin architecture for this we will simply be hardcoding the data-models that we need. In proposed implementation the plugins simply create and populate data objects with as much information as they have. Upon creation these objects are added | ||
| + | to a class-level array of initiated objects. Once all plugins complete the data-models are told to process the data. | ||
| + | |||
| + | - Combine data from all instances that map to same database object. | ||
| + | - Retrieve current data in db. | ||
| + | - Update if necessary. | ||
| + | - Flush global instance storage for model. | ||
| ==== Plugins ==== | ==== Plugins ==== | ||
| Line 30: | Line 38: | ||
| Installed plugins should be specified as a simple list of module names to import in string form (much like django's INSTALLED_APPS setting). During import each plugin will be responsible for registering itself by calling gdd.plugins.register(<class>), this will add the plugin to a global list of plugins which will be topologically sorted once the plugin list has been processed. | Installed plugins should be specified as a simple list of module names to import in string form (much like django's INSTALLED_APPS setting). During import each plugin will be responsible for registering itself by calling gdd.plugins.register(<class>), this will add the plugin to a global list of plugins which will be topologically sorted once the plugin list has been processed. | ||
| - | Each plugin will be called in order and must itself determine if the netbox in question is of instrest (ie. dues it support the mibs that the plugin knows about, does the netbox vendor match those supported by the plugin...). Once this is decided the plugin can either pass or start collecting more SNMP data and populate the data persistence store and/or modify the allready there (ie. vendor specific interpretations). Once all plugins have run gDD must ensure that the updated data is stored to the database. | + | Each plugin will be called in order and must itself determine if the netbox in question is of instrest (ie. dues it support the mibs that the plugin knows about, does the netbox vendor match those supported by the plugin...). Once this is decided the plugin can either pass or start collecting more SNMP data and populate the data persistence store and/or modify the already there (ie. vendor specific interpretations). Once all plugins have run gDD must ensure that the updated data is stored to the database. |
| To keep the plugin directory somewhat organized the following organization has been proposed: | To keep the plugin directory somewhat organized the following organization has been proposed: | ||
| Line 46: | Line 54: | ||
| - arp.py | - arp.py | ||
| - ... | - ... | ||
| + | |||
| + | ===== New plugin design proposal ===== | ||
| + | |||
| + | Plugins should be divided into their respective responsibilities. Each plugin should be its own python module | ||
| + | with a defined handler-class sub-classed from the nav.ipdevpoll.Plugin-class. The handler should only deal with | ||
| + | storing the information to the database and logic to determine if vendor-specific collections methods should be used. | ||
| + | Each plugin should supply its own test-suite in a folder called test withing the plugin module. Every plugin should | ||
| + | prefer to use the MIB-2 standard mibs for collecting information, falling back to vendor specific mibs. | ||
| + | To maximize the amount of easily testable code, all parsing logic should be extracted into their own methods and not | ||
| + | be entangled into the collection process (methods that returns deferred objects). Remember to call the processing | ||
| + | methods asynchronously as well. | ||
| + | |||
| + | /nav/ipdevpoll/plugins | ||
| + | - /iftable | ||
| + | - iftable.py | ||
| + | - test_iftable.py | ||
| + | - collector.py | ||
| + | - /vendor | ||
| + | - cisco.py | ||
| + | - test_cisco.py | ||
| + | - alcatel.py | ||
| + | - test_alcatel.py | ||
| + | - .. | ||
| + | |||
| + | Example plugin: | ||
| + | <code python> | ||
| + | from nav.ipdevpoll import Plugin | ||
| + | from nav.ipdevpoll.plugins.vlan import collector | ||
| + | |||
| + | class Vlan(Plugin): | ||
| + | def __init__(self, *args, **kwargs): | ||
| + | """ | ||
| + | Initialize the plugin | ||
| + | """ | ||
| + | Plugin.__init__(self, *args, **kwargs) | ||
| + | self.deferred = defer.Deferred() | ||
| + | |||
| + | def handle(self): | ||
| + | """ | ||
| + | This is the entry point for the plugin. | ||
| + | """ | ||
| + | self.logger.debug("Collecting VLAN information") | ||
| + | collector.collect(self.job_handler.agent) | ||
| + | return self.deferred | ||
| + | |||
| + | def process_result(self, result): | ||
| + | """ | ||
| + | Standard processing method for all plugins. | ||
| + | The collections methods should provide data | ||
| + | in the same format to this method. | ||
| + | """ | ||
| + | if not result: | ||
| + | self.logger.debug( | ||
| + | "No VLAN information found using MIB-2 MIBS.\ | ||
| + | Trying vendor specific collection") | ||
| + | # Try vendor specific methods | ||
| + | else: | ||
| + | self.logger.debug("Found %s VLANs. Processing." % len(result)) | ||
| + | for ifIndex, vlan in result: | ||
| + | # Store the information to database or similar | ||
| + | pass | ||
| + | |||
| + | # Plugin-run finished. Exit. | ||
| + | self.deferred.callback(True) | ||
| + | return result | ||
| + | </code> | ||
| === Suggested log plugins === | === Suggested log plugins === | ||
| Line 51: | Line 125: | ||
| * ARP | * ARP | ||
| * CAM (currently in getBoksMacs) | * CAM (currently in getBoksMacs) | ||
| - | * RRD stats (currently provided by crikect) | + | * RRD stats (currently provided by cricket) |
| - | * ModuleMon replacement | + | |
| === Suggested inventory plugins === | === Suggested inventory plugins === | ||
| Line 65: | Line 138: | ||
| * HP support | * HP support | ||
| + | === Suggested status monitoring plugins === | ||
| + | |||
| + | * ModuleMon | ||
| + | |||
| + | === Implemented plugins === | ||
| + | |||
| + | Already implemented plugins can be found on [[devel:ipdevpollplugins]] | ||
| ===== Database changes ===== | ===== Database changes ===== | ||
| - | The following database changes have been suggested FIXME | + | The following database changes have been suggested. |
| + | |||
| + | ==== Merge swport/gwport tables ==== | ||
| + | |||
| + | A long standing issue has been merging the artificially separated ''swport'' and ''gwport'' tables. In the SNMP MIBs there is no such divide, both swports and gwports are referred to as interfaces, and IP-MIB can be used to find which interfaces operate on layer 3 (IP). All interface-generic information should be stored in a single interface table, while specific layer 3 properties should be stored in separate related tables. There are also many attributes from IF-MIB's ''ifTable'' that are not collected and stored by the current gDD solution, but which would be very interesting to store. | ||
| + | |||
| + | Most NAV code that deals with interfaces will have duplicate code to take care of ''swport'' and ''gwport''. Most subsystems only deal with reading from these tables, so during the transition to a new db model one can create ''swport'' and ''gwport'' views. gDD and networkDiscovery (the topology deriver) are the only known systems that change data in the swport/gwport tables. networkDiscovery will need a rewrite to support a new model (or update rules need to be added to the new views). | ||
| + | |||
| + | ==== No module requirement for interfaces ==== | ||
| + | |||
| + | The current NAV data model mandates that all gwports/swports bek related to a module. For netboxes that do not contain physical modules (or whose modules cannot be found by gDD), the current gDD will create an artifical module related to the same device record as the containing netbox and add swports/gwports to this module. | ||
| + | |||
| + | The IF-MIB has no concept of an interface/module relationship. Information about such a relationship will most likely come from proprietary MIBs (or possibly a properly populated ENTITY-MIB), and should be treated as a bonus if found. Ie. NAV's model should add to the interface table a mandatory foreign key to netbox, and only an optional foreign key into the module table. | ||
| ===== Code ===== | ===== Code ===== | ||
| - | The mercurial repositories can be found at FIXME | + | The mercurial repositories can be found at http://metanav.uninett.no/hg/features/ipdevpoll/ |
| ===== References ===== | ===== References ===== | ||
devel/blueprints/ipdevpoll.txt · Last modified: 2012/05/07 11:26 by morten
Page Tools
