Event processing¶
Introduction¶
NetXMS is event based monitoring system. Events can come from different sources (polling processes (status, configuration, discovery, and data collection), SNMP traps, from NXSL scripts and directly from external applications via client library). All events all are forwarded to NetXMS Event Queue.
NetXMS Event Processor can process events from Event Queue in either sequential or parallel mode. In sequential mode events are processed one-by-one which guarantees that events will be processed in the same sequence as they arrive onto NetXMS server. For installation where a lot of events could be generated in a short period of time this mode can be a bottleneck.
Parallel processing mode allows to process events in several parallel threads, thus allowing to scale horizontally and increasing processing performance. Number of threads for parallel processing is set by Events.Processor.PoolSize server configuration parameter.
Event Processing Rules can read/write persistent storage, create/terminate alarms, can have scripts that are checking other node statuses and care shuld be taken to ensure that no race condition would occur when performing parallel processing.
Correct operation is ensured by properly setting Events.Processor.QueueSelector server configuration parameter. This parameter contains macros that are expanded when an event is created. Events that have same QueueSelector string will be processed one-by-one by one and the same event processing thread, thus ensuring that there will be no race condition between these events.
Event Processing Policy¶
Actions taken by event processor for any specific event are determined by a set of rules called Event Processing Policy.
Event Processing Policy Screen
Every rule has two parts - matching part (called Condition in the rule configuration dialog), which determines if the rule is appropriate for the current event, and action part, which determines actions to be taken for matched events.
Each event passes through all rules in the policy, so if it matches more than one rule, actions specified in all matched rules will be executed. You can change this behavior by setting Stop Processing flag actual the rule. If this flag is set for a rule and that rule is matched, processing of current event will be stopped.
You can create and modify Event Processing Policy using
Event Processing Policy Editor. To access the
Event Processing Policy Editor window, press F4 or select
menu.
To create event policy right click on entry before or after which new Event Processing Policy should appear and select Insert before or Insert after. Drag and drop can be used for rule reorganization.
Event Processing Policy item context menu
To edit Event Processing Policy, filter or action click on icon in right corner of an entry, it will open general properties of Event Processing Policy.
Edit buttons
In EPP properties there are following sections:
| Section | Description |
|---|---|
| Condition | Sub-sections of Condition section determine, if the rule is applicable to a particular event. If checkbox Rule is disabled is set, this rule is ignored. |
| Condition –> Source Objects | One or more event’s source objects. This list can be left empty, which matches any node, or contain nodes, subnets, or containers. If you specify subnet or container, any node within it will be matched. |
| Condition –> Events | Event code. This field can be left empty, which matches any event, or list of event codes. |
| Condition –> Severity Filter | Event’s severity. This field contains selection of event severities to be matched. |
| Condition –> Filtering Script | Optional matching script written in NXSL. If this field is empty, no
additional checks are performed. Otherwise, the event will be considered as
matched only if the script returns non-zero (TRUE) return code. For
more information about NetXMS scripting language please refer to the
chapter Scripting in this manual. |
| Action | Sub-sections of Action section determine what actions are performed if an event meets condition. If checkbox Stop event processing is set, then subsequent rules (with higher rule number) will not be processed for a given event. |
| Action –> Alarm | There can be set rules connected with alarm generation. Alarm can be created, resolved or terminated or no action to alarms is done. |
| Action –> Persistent Storage | NXLS Persistent Storage action like add/update or delete can be made. |
| Action –> Server Actions | Here a list of actions is defined to be executed if condition is met. For action configuration refer to Actions chapter. |
| Action –> Timer Cancellations | Here a list of timers is defined that should be canceled if condition is met. |
| Comments | Here a comment can be added. The comment is displayed as a name of the rule. |
Event Processing Policy properties
Description of EPP toolbar form left to right: save changes, expand all, collapse all, horizontal layout, vertical layout, cut EPP, copy EPP, paset EPP, delete EPP
*After all manipulations are done - save changes by pressing save icon.*
Examples¶
This rule defines that for every major or critical event originated from a node named “IPSO” two e-mail actions will be executed.
Example 1
Alarms¶
Alarms Overview¶
As a result of event processing some events can be shown up as alarms. Usually alarm represents something that needs attention of network administrators or network control center operators, for example low free disk space on a server.
All alarm events are logged to alarm log. The number of days the server keeps alarm history can be configured by “AlarmHistoryRetentionTime” server configuration parameter. Alarm log can be viewed in “Alarm Log View”(Alt+F8). This view gives option to query for required information from alarm log.
Every alarm has the following attributes:
| Attribute | Description |
|---|---|
| Creation time | Time when alarm was created. |
| Last change time | Time when alarm was last changed (for example, acknowledged). |
| State | Current state of the alarm, see table bellow |
| Message | Message text (usually derived from originating event’s message text). |
| Severity | Alarm’s severity - Normal, Warning, Minor, Major, or Critical. |
| Source | Source node (derived from originating event). |
| Key | Text string used to identify duplicate alarms and for automatic alarm termination. |
Possible alarm states:
| Outstanding | New alarm. |
| Acknowledged | When network administrator sees an alarm, he may acknowledge it to indicate that somebody already aware of that problem and working on it. A new event with the same alarm ID will reset the alarm state back to outstanding |
| Sticky Acknowledged for time | Alarm will remain acknowledged for given time interval even after new matching events, after time will pass alarm will be moved to outstanding state. This option can be used like snooze. When you know that there will be new matching events, but it will not change the situation. But after some time someone should check this problem. For example, if you have problem that cannot be solved until next week, so this alarm can be sticky acknowledged for 7 days. After 7 days this problem again will be in outstanding state. This type of acknowledge can be disabled by parameter “EnableTimedAlarmAck” in server configuration view. |
| Sticky Acknowledged | Alarm will remain acknowledged event after new matching events. This can be useful when you know that there will be new matching events, but it will not change the situation. For example, if you have network device which will send new SNMP trap every minute until problem solved, sticky acknowledge will help to eliminate unnecessary outstanding alarms. |
| Resolved | Network administrator sets this state when the problem is solved. |
| Terminated | Inactive alarm. When problem is solved, network administrator can terminate alarm. This will remove alarm from active alarms list and it will not be seen in console, but alarm record will remain in database. |
There are 2 types of alarm state flows: strict and not strict. This option can be configured in Preference page of Alarms or on server configuration page, parameter “StrictAlarmStatusFlow”. The difference between them is that in strict mode Terminate can be done only after Resolve state.
Not strict(default)
Strict
Alarm Melodies¶
On each severity of alarm can be set melody to play. This melody will be played when new alarm in state outstanding will occur. Melody that should be played should exist on server in wav format. See instruction there: Upload file on server. By default there are no sounds on alarms.
To set sound open preferences, there select tab. There in drop-down will be seen all possible options. If sound will not be chosen, alarm with this severity will come silently.
To configure sounds, open preferences and select tab. Drop-downs next to each severity level have a list of available sounds. If no sound is chosen, alarm for given severity will come silently.
Alarm Browser¶
When an alarm is generated it will appear in the Alarm Browser where information about currently active alarms can be viewed.
Alarm Comments¶
For each alarm can be created comments in “Alarm Details”
or “Alarm Comments” views.
Comment can be created, edited or deleted. All comments will be deleted after alarm termination.
Alarm Summary Emails¶
It is possible to schedule emails which contain a summary of all currently active alarms, similar to what can be seen in the Alarm Browser.
To enable Alarm Summary Emails it is required to configure the following server parameters:
| Name |
|---|
| SMTPFromAddr |
| SMTPFromName |
| SMTPPort |
| SMTPRetryCount |
| SMTPServer |
| EnableAlarmSummaryEmails |
| AlarmSummaryEmailSchedule |
| AlarmSummaryEmailRecipients |
Further information on server configuration parameters can be found in Server configuration parameters.
Generating Alarms¶
To generate alarms from events, you should edit Alarm field in appropriate rule of Event Processing Policy. Alarm configuration dialog will look like this:
You should select Generate new alarm radio button to enable alarm generation from current rule. In the Message field enter alarm’s text, and in the alarm key enter value which will be used for repeated alarms detection and automatic alarm termination. In both fields you can use macros described in the Macros for Event Processing section.
You can also configure sending of additional event if alarm will stay in
Outstanding state for given period of time. To enable this, enter
desired number of seconds in Seconds field, and select event to be
sent. Entering value of 0 for seconds will disable additional event
sending.
Alarms generated by rules can by categorised to limit what alarms can be seen by what users. This can be done by applying a category in the Alarm Category field, which can be created and configured in the Alarm Category Configurator.
Alarm Category Configurator¶
Alarm categories can be created and configured in the Alarm Category Configurator which can be found in menu:
Alarm Category Configurator
Alarm categories provide the possibility to configure access rights for viewing generated alarms on a per user or per group basis. When creating an alarm category, it is possible to set the Category name, Description.
Alarm Category properties
Alarm category access rights can be configured by adding users or groups to the access list of the category in the Access Control property page.
Alarm Category Access Control
By default, all alarms can be viewed by all users due to the View all alarms system right being set as default to the Everyone user group. In order to limit the viewing of alarms, this system right should be removed and the access rights configured in the categories themselves. When the categories have been configured, they can be applied to the necessary Event Processing Policy rules.
If an alarm category has been applied to an Event Processing Policy rule, it will appear in the Event Processing Policy Editor when a rule is expanded under the Action section.
Event Processing Policy expanded
Automatic Alarm Termination/Resolve¶
You can terminate or resolve all active alarms with given key as a reaction for the event. To do this, select Terminate alarm radio button or Resolve alarm radio button in alarm configuration dialog and enter value for alarm key. For that field you can use macros described in the Macros for Event Processing chapter.
Escalation¶
As it was described in Generating Alarms chapter there is possibility to generate new event if alarm stay in Outstanding state for too long. Escalation is built on this option. When alarm was generated, but no action was done from operator in predefined time, new event can be generated and this time email or notification (SMS, instant message) can be sent to operator or to it’s manager. This escalation process can have as many steps as it is required.
Actions¶
In addition to alarm generation server can perform various types of actions as a reaction to an event. Action types available in NetXMS are described in the following sections. Each action can be separately disabled in action configuration.
After the action is added, it can be edited to add delay time and timer key. This option can be used to prevent notification sending in case if problem solved quickly enough. Key is a free form string that support macro and delay is the delay time in seconds before action is executed.
The next example shows the configuration for the situation when there is no need to notify anyone if node went down and back up in just a minute.
Escalation¶
One EPP rule can contain multiple actions with different delays. Delay timers are canceled by other rule in case of problem resolution.
- The next example shows that if node went down, then
- after 1 minute responsible person will be notified if the problem still persists
- after 30 minutes the support manager will be notified if the problem still persists
- after 1 hour the IT manager will be notified if the problem still persists
Action types¶
Execute command on management server¶
Executes provided command on server node. Check that user under which netxmsd process
run has permission to run this command.
Execute command on remote node¶
Executes provided command name defined in this nodes agent configuration file. To this
command can be given parameters in format: commandName param1 param2 param3...
Check that user under which nxagentd process run has permission to run this
command.
As the Remote Host can be used hostname or object name(int format: @objectName).
Second option allows action execution on node behind proxy.
Send e-mail¶
Send email to one or more recipients. Multiple recipients can be separated by semicolons.
Required server configuration parameters to send emails: SMTPFromAddr, SMTPFromName,
SMTPRetryCount, SMTPServer. For detailed description of parameters check Server configuration parameters.
In message text can be used Macros for Event Processing.
Send notification¶
Send notification, e.g. SMS, to one or more recipients. Multiple recipients can be separated by semicolons. Server will use Notification channels for actual message sending.
In message text can be used Macros for Event Processing.
Send XMPP message¶
Sends XMPP/Jabber message to one or more recipients. Multiple recipients can be separated by semicolons. Required server configuration parameters to send XMPP message: XMPPLogin, XMPPPassword, XMPPPort, XMPPServer, EnableXMPPConnector. For detailed description of parameters check Server configuration parameters.
In message text can be used Macros for Event Processing.
Execute NXSL script¶
This action executes script form scrip library. In action configuration should be defined name of script. Information about scripting and library can be found there.
Forward event¶
NetXMS does not support configuration synchronization between two NetXMS servers(Distributed Monitoring). But it is possible to forward events from one server to another. This option allow synchronize events between servers but there are some limitation.
Configuration¶
- Source server configuration:
- Create new action of type “forward event” - it will have destination server address property.
- Create a rule in event processing policy with filter for events you want to forward and add forwarding action as action.
- Destination server configuration:
- Enable EnableISCListener and ReceiveForwardedEvents in server configuration.
- Open port 4702.
- Check that receiving server have all events as on a sending server
Limitation¶
- Limitations of event forwarding:
- Event template with same event code or event name must exist on recipient server
- Node object with same IP address as event’s source node’s address must exist on recipient server
- Does not work with zones
Events not met these conditions are discarded. It is possible to check if and why incoming events are discarded by turning on level 5 debug on receiving server.
There can be used one of two options if it is required to disable polling of sender server nodes on recipient server: disable all polling protocols or unmanage nodes. Chose depends on how you wish to see node’s status. For unmanaged node, it always be “unmanaged”, regardless of active alarms. If you disable polling, node’s status will be “unknown” unless there will be active alarms for that node - in that case node’s status will change to severity of most critical alarm.
Notification channels¶
New in version 3.0.0.
NetXMS supports concept of notification channel drivers to provide SMS and instant message sending functionality. Role of notification channel driver is to provide level of abstraction on top of different notification sending mechanisms and uniform notification sending interface for server core. It is possible to set up and use several notification channels.
Configuration of notification channels is done in .
Notification channel driver parameters are specified in Driver configuration input field. Each parameter is given on a separate line in format: parameter_name=parameter_value. Meaning of parameters is driver dependent and described separately for each driver. It a parameter is not given, it’s default value will be used.
Once notification channel is created is is seen in channel list with green or read square next to the name - it is channel status identifier. It should be green if driver initialization was successful or read in other cases. Status column displays last sent attempt status and Error message column provide more information about driver initialization or sending error.
Drivers¶
The following drivers are provided by default with NetXMS installation:
| Driver | Description |
|---|---|
| anysms.ncd | SMS driver for any-sms.biz service (http://any-sms.biz). Configuration parameters:
|
| dbtable.ncd | This driver saves notifications to a database. Configuration parameters:
|
| dummy.ncd | Dummy driver for debugging purposes. Does not send any actual notifications and only logs them to server log file. This driver has no configuration parameters. It is necessary to set debug level to debug=6 or higher to get records in the log file. |
| gsm.ncd | Driver for serial or USB attached GSM modems with support for standard GSM AT command set. Configuration parameters:
|
| kannel.ncd | Driver for Kannel SMS gateway (http://www.kannel.org). Configuration parameters:
|
| msteams.ncd | Notification channel driver for Microsoft Teams. Configuration parameters:
Optional configuration section “Channels” should contain list of channels in the following format: channelName=URL, where channelName is an arbitrary name later used as recipient in action configuration. More information about setting up the URL of incoming webhook available there #config example
ThemeColor=FF6A00
UseMessageCards = false
[Channels]
Channel=URL
AnotherChannel=URL
MsTeams requires 2 fields in action configuration:
|
| mymobile.ncd | SMS driver for MyMobile API gateways. Configuration parameters:
|
| nexmo.ncd | SMS driver for Nexmo gateway. Configuration parameters:
|
| nxagent.ncd | Similar to gsm.ncd, but sending is done via GSM modem, attached to NetXMS agent. Configuration parameters:
|
| portech.ncd | Driver for Portech MV-372 and MV-374 GSM gateways (https://www.portech.com.tw/p3-product1_1.asp?Pid=14). Configuration parameters:
|
| slack.ncd | Driver for slack.com service. Configuration parameters:
|
| smseagle.ncd | Driver for SMSEagle Hardware SMS Gateway. Configuration parameters:
|
| telegram.ncd | Notification channel driver for Telegram messenger. Configuration parameters:
Only AuthToken field is mandatory field all others are optional. It is necessary to create a telegram bot that NetXMS server will use to send messages. In order to create a new bot it’s necessary to talk to BotFather and get bot authentication token (AUTH_TOKEN). Set authentication token in notification channel configuration, e.g.: AuthToken=1234567890:jdiAiwdisUsWjvKpDenAlDjuqpx The bot can:
Telegram’s bot can’t initiate conversations with users in a private chat or a group. A user must either add bot to a group or send a private message to the bot first. Chat, group or channel is identified by ID or name (without @ prefix). For private chats only users who configured a Username can be identified by name (without @ prefix). NetXMS stores the correspondence between ID and name when the bot receives a message in chat or group (NetXMS server should be running a that moment). If group, channel name or username is changed, it’s necessary to send any message to the bot so new correspondence could be stored. Telegram notification channel requires 2 fields in action configuration:
If you want to use ID to identify a recipient, you can get it by opening Telegram API URL in your browser, e.g. https://api.telegram.org/bot1234567890:jdiAiwdisUsWjvKpDenAlDjuqpx/getUpdates After sending a message to the bot or adding it to a group you should see chat id there. You might need to temporary deconfigure Telegram notification channel, otherwise if NetXMS server is running, it will read data from Telegram API first. |
| text2reach.ncd | Driver for Text2Reach.com service (http://www.text2reach.com). Configuration parameters:
|
| textfile.ncd | Notification driver that writes messages to text file. Configuration parameter:
|
| websms.ncd | Driver for websms.ru service (https://websms.ru). Configuration parameters:
|
NXLS Persistent Storage¶
NXSL¶
- There are 2 functions:
- ReadPersistentStorage(“key”) - read value by key
- WritePersistentStorage(“key”, “value”) - insert or update value by key. If value will be empty - variable will be deleted.
View¶
Persistent Storage view () provide information about current state of Persistent Storage variables.
Macros for Event Processing¶
On various stages of event processing you may need to use macros to include information like event source, severity, or parameter in your event texts, alarms, or actions. You may use the following macros to accomplish this:
| Macro | Description |
|---|---|
%a |
IP address of event source object. |
%A |
Alarm’s text (can be used only in actions to put text of alarm from the same event processing policy rule). |
%c |
Event’s code. |
%g |
Globally unique identifier (GUID) of event source object. |
%i |
Unique ID of event source object in hexadecimal form. Always prefixed with 0x and contains exactly 8 digits (for example 0x000029AC). |
%I |
Unique ID of event source object in decimal form. |
%K |
Alarm’s key (can be used only in actions to put text of alarm from the same event processing policy rule). |
%m |
Event’s message text (meaningless in event template). |
%M |
Custom message text. Can be set in filtering script by setting CUSTOM_MESSAGE variable. |
%n |
Name of event source object or name of interface for intefrace macro expansion. |
%N |
Event’s name. |
%s |
|
%S |
Event’s severity code as text. |
%t |
Event’s timestamp is a form day-month-year hour:minute:second. |
%T |
Event’s timestamp as a number of seconds since epoch (as returned by time() function). |
%u |
User tag associated with the event. |
%v |
NetXMS server’s version. |
%z |
Zone UIN of event source object. |
%Z |
Zone name of event source object. |
%[name] |
Value returned by script. You should specify name of the script from script library. |
%{name} |
Value of custom attribute. |
%{name:default_value} |
Value of custom attribute. If such custom attribute does not exists on a particular node, default_value is taken. If custom attribute exists, but has empty value, this empty value is taken (if this macro is used somewhere, where it’s value is converted to numeric value - e.g. as threshold value for a numeric DCI - then empty value will be converted to 0). |
%<name> |
Event’s parameter with given name. |
%1 - %99 |
Event’s parameter number 1 .. 99. |
%% |
Insert % character. |
If you need to insert special characters (like carriage return) you can use the following notations:
| Char | Description |
|---|---|
\t |
Tab Character (0x09) |
\n |
New line, CR/LF character pair |
\\ |
Backslash character |
Event’s parameter with given name¶
- Threshold reached/rearmed named parameters:
- %<dciId>
- %<dciName>
- %<dciDescription>
- %<thresholdValue>
- %<currentValue>
- %<instance>
- %<isRepeatedEvent> - set only for DCI reached events
- %<dciValue>