We just published patch release 3 for version 5.1. It fixes critical bug in SNMP trap receiver, and contains few other fixes and improvements. Full change log since previous patch release:

-  Fixed critical bug in SNMP trap receiver
-  Server checks for other possible SNMP credentials during configuration poll if node marked as SNMP unreachable
-  Image attributes in Markdown viewer
-  Fixed bug in counter reset detection
-  Fixed issue NX-2685 (nxshell asks for password while using properties file)

Hello, we have a problem with crashing NetXMS Server, it causes segfault after about 1-2 days.

Dmesg 1:

Quote[58727.484875] $POLLERS/WRK[8933]: segfault at 3800 ip 00007f655c27dc61 sp 00007f64fc55e948 error 6 in libc.so.6[7f655c0fe000+195000]
[58727.484904] Code: 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 83 fa 20 72 33 c5 fe 6f 06 48 83 fa 40 0f 87 b5 00 00 00 c5 fe 6f 4c 16 e0 <c5> fe 7f 07 c5 fe 7f 4c 17 e0 0f 01 d6 74 04 c5 fc 77 c3 c5 f8 77

Dmesg 2:

Quote[264395.048604] $DATACOLL/WRK[165295]: segfault at 0 ip 00007f4bb8324b1d sp 00007f4b87a06a58 error 4 in libc.so.6[7f4bb81a2000+195000]
[264395.048638] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3 0f 1e fa 89 f8 48 89 fa c5 f9 ef c0 25 ff 0f 00 00 3d e0 0f 00 00 0f 87 23 01 00 00 <c5> fd 74 0f c5 fd d7 c1 85 c0 74 57 f3 0f bc c0 e9 2c 01 00 00 66

Backtrace after second segfault:

Quoteterminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
terminate called recursively
terminate called recursively

Thread 1837 "$POLLERS/WRK" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffba9c8640 (LWP 340969)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140736324208192) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140736324208192) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140736324208192) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140736324208192, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff751f476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff75057f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff77cab9e in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff77d620c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007ffff77d6277 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007ffff77d64d8 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007ffff77cd265 in std::__throw_bad_alloc() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00005555555e28f6 in handleOOM(unsigned long, bool) ()
#11 0x00007ffff74ced8d in operator() (__closure=<optimized out>, __closure=<optimized out>, var=0x7fffb650bd20) at /build/nxbuild.kEGh9XOvdg/build/netxms-5.1.3/src/snmp/libnxsnmp/snapshot.cpp:125
#12 std::__invoke_impl<unsigned int, SNMP_Snapshot::create(SNMP_Transport*, const uint32_t*, size_t)::<lambda(SNMP_Variable*)>&, SNMP_Variable*> (__f=...) at /usr/include/c++/11/bits/invoke.h:61
#13 std::__invoke_r<unsigned int, SNMP_Snapshot::create(SNMP_Transport*, const uint32_t*, size_t)::<lambda(SNMP_Variable*)>&, SNMP_Variable*> (__fn=...) at /usr/include/c++/11/bits/invoke.h:114
#14 std::_Function_handler<unsigned int(SNMP_Variable*), SNMP_Snapshot::create(SNMP_Transport*, const uint32_t*, size_t)::<lambda(SNMP_Variable*)> >::_M_invoke(const std::_Any_data &, SNMP_Variable *&&) (
    __functor=..., __args#0=<optimized out>) at /usr/include/c++/11/bits/std_function.h:290
#15 0x00007ffff74ce1fd in std::function<unsigned int (SNMP_Variable*)>::operator()(SNMP_Variable*) const (__args#0=<optimized out>, this=0x7fffba9c6a10) at /usr/include/c++/11/bits/std_function.h:590
#16 SnmpWalk(SNMP_Transport*, unsigned int const*, unsigned long, std::function<unsigned int (SNMP_Variable*)>, bool, bool) (transport=transport@entry=0x7fffd1089600, rootOid=rootOid@entry=0x7fffba9c6aa0,
    rootOidLen=rootOidLen@entry=10, handler=..., logErrors=logErrors@entry=false, failOnShutdown=failOnShutdown@entry=false) at /build/nxbuild.kEGh9XOvdg/build/netxms-5.1.3/src/snmp/libnxsnmp/util.cpp:372
#17 0x00007ffff74ce30f in SNMP_Snapshot::create (transport=transport@entry=0x7fffd1089600, baseOid=baseOid@entry=0x7fffba9c6aa0, oidLen=oidLen@entry=10)
    at /build/nxbuild.kEGh9XOvdg/build/netxms-5.1.3/src/snmp/libnxsnmp/snapshot.cpp:124
#18 0x00007ffff7d48ffe in SNMP_Snapshot::create (baseOid=std::initializer_list of length 10 = {...}, transport=0x7fffd1089600) at ../../../include/nxsnmp.h:1134
#19 SnmpGetRoutingTable (snmp=0x7fffd1089600, node=...) at /build/nxbuild.kEGh9XOvdg/build/netxms-5.1.3/src/server/core/snmp.cpp:214
#20 0x00007ffff7c5eb67 in Node::getRoutingTable (this=this@entry=0x7fffe4376810) at /build/nxbuild.kEGh9XOvdg/build/netxms-5.1.3/src/server/core/node.cpp:10071
#21 0x00007ffff7c5edfa in Node::routingTablePoll (this=0x7fffe4376810, poller=<optimized out>, session=<optimized out>, rqId=<optimized out>)
    at /build/nxbuild.kEGh9XOvdg/build/netxms-5.1.3/src/server/core/node.cpp:10223
#22 0x00007ffff7ce25b2 in Pollable::doRoutingTablePoll (this=0x7fffe4377390, poller=0x7fffd4dbdb00) at /build/nxbuild.kEGh9XOvdg/build/netxms-5.1.3/src/server/core/pollable.cpp:206
#23 0x00007ffff7b73f16 in __ThreadPoolExecute_Wrapper_1<Pollable, PollerInfo*> (arg=0x7fffd4cddb80) at ../../../include/nms_threads.h:1191
#24 0x00007ffff79ef0ef in ProcessSerializedRequests (data=0x7fffc86b2740) at /build/nxbuild.kEGh9XOvdg/build/netxms-5.1.3/src/libnetxms/tp.cpp:495
#25 0x00007ffff79ed784 in WorkerThread (threadInfo=0x7ffff1a2ca70) at /build/nxbuild.kEGh9XOvdg/build/netxms-5.1.3/src/libnetxms/tp.cpp:214
#26 0x00007ffff79e787f in ThreadCreate_Wrapper_1<WorkerThreadInfo*> (context=0x7ffff1a2cae0) at ../../include/nms_threads.h:539
#27 0x00007ffff7571ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#28 0x00007ffff7603850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)

Version:

QuoteNetXMS Server Version 5.1.3 Build 5.1-496-g315015358c (UNICODE)
NXCP: 5.62.1.52 (AES-256, 3DES, AES-128)
Built with: g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

OS: Ubuntu Server 22.04.5 LTS

Hi!

From stack trace it looks like netxmsd run out of memory. Can you please show chart of memory consumption by netxmsd?

Best regards,
Victor

Here is a memory graph (LibreNMS) from a virtual machine running only NetXMS and TimescaleDB 15.

Now it has 48 GB allocated, on the older version it ran without problems on 32 GB.

I don't remember exactly what version it was. The problem appears from versions from about October/November. We update about once a month.