NetXMS Support Forum

Hi!

Last special build uses SO_REUSEADDR option - it makes no difference. As for MaxUserPort, things are different on Windows 2008 - http://support.microsoft.com/kb/929851. So by default we should have about 16000 ports for outgoing connections, which is much more then required.

Best regards,
Victor

It also could be Windows 2008 issue. We have installation in Riga with 700+ agents - and I never seen issues like that. They are running on Windows Server 2003 x86.

Best regards,
Victor

Hi!

I'm still unable to reproduce this issue or find any clue to the source of the problem. Please try to replace libnetxms.dll to attached one - it has another changes in communication code. At least, it may give more meaningful error codes in "agent unreachable" messages in the log.

Also, do you experience this issue from the beginning, or it appears after latest upgrades?

Best regards,
Victor

Could you please try to install updated build: https://www.netxms.org/download/rc/netxms-1.0.8-rc2.exe. If it will not solve the problem, please set logging to file (by setting LogFile = some_file in netxmsd.conf) and run netxmsd.exe with -D 6 command line option (this will enable debug output). It will generate a lot of messages in the log file - I'm interested in messages containing words "agent unreachable, error=".

Best regards,
Victor

Hi!

I plan 1.0.8 to be mostly a bugfix release (plus custom SNMP port feature), and if everything will go as expected, I'll will release it in a week.

Best regards,
Victor

Few additional questions:

Do you have a Windows firewall running on server machine?
Is there any error messages in netxmsd log?

Best regards,
Victor

I suspect that it may hit some limits for open connections. There was a limit of around 4000 outgoing TCP connections in Windows 2003, I'm not sure how it is in Windows 2008. Could you please run netstat -n and count connections to external IPs on port 4700? And how many of them are not in ESTABLISHED state?

Best regards,
Victor

Hi!

Unfortunately I cannot reproduce this on my system. Could you please run nxagentd under gdb and send me a backtrace after crash?

To run agent under gdb:

1. Run command
gdb /opt/netxms/bin/nxagentd

You will get gdb prompt:
(gdb)

Type command:
run -f

Agent will run in foreground. After crash, gdb prompt will appear:
(gdb)

Type command
bt

and send it's output to me.

Best regards,
Victor

Some additional questions:

1. where the packets was captured?
2. is it possible that incoming SNMP packets blocked by firewall at NetXMS server machine?
3. could you run server with debug level set to at least 6 (command line option -D 6) and post log file?

Best regards,
Victor

Added to bugtracker as feature request.

Btw, some message customization can be done using custom attributes for the object. For example, you can define custom attribute MESSAGE and then insert it's value into alarm text by using macro %{MESSAGE}. Of course, this is not a replacement for functionality you was asking for, because it requires manual attribute configuration on each node, but it can help in some cases.

Best regards,
Victor

Queues looks much better now. The only possibly problematic number is data collector's queue. Is it always like this, or it was just a short pike? Btw, by default server create DCIs for internal queue sizes. If those DCIs are configured, could you post a charts of them for last hour or two?

Few things I forgot:

- Remove ProcessAffinityMask option from netxmsd.conf. It does not help with the crash issue, but prevents using of all CPUs in the system.

- Some additional questions:

1. How many memory consumes netxmsd process? Is it growing?
2. What is the state of SQL server? So high value for database writer queue means that either NetXMS server cannot send requests as fast as needed or SQL server cannot process them as fast as needed. Increasing value for NumberOfDatabaseWriters will speed up request sending, but will SQL server be able to handle them?

Best regards,
Victor

Hi!

Numbers are quite high, especially database writer queue. I would suggest to change some server parameters, taking into account number of nodes you have:

NumberOfStatusPollers = 100
NumberOfConfigurationPollers = 25
NumberOfRoutingTablePollers = 25

NumberOfDatabaseWriters = 5


I would also suggest to increase status and configuration polling intervals, which are one minute and one hour respectively. My recommended settings is to set configuration polling interval to at least 4 hours, and status polling interval to 90 seconds. It can be done by setting the following parameters:

StatusPollingInterval = 90
ConfigurationPollingInterval = 14400


Best regards,
Victor

Hi!

No, it's not possible. I can add this as feature request.

Best regards,
Victor

Hi!

Am I understand your correctly - you have one three-node cluster, and each node has two SNMP agents?

Right now the only working solution I could imagine is to configure lot of external parameters on agent which will call nxsnmpget - but this cannot be considered as good solution. I think that easiest way is to allow change of SNMP port on node and DCI basis - this is relatively easy change and will solve this problem. I'll try to implement it in 1.0.8 release.

Best regards,
Victor