nscp test is crashing

[mikep]

I just tested build 156. The service appears to run fine. When I run the following command, I get the errors below.

nscp test

In the following output, I have tested the command twice. The first run gets futher than the second run.

C:\Program Files\NSClient++>nscp test
d    ..\..\..\trunk\service\NSClient++.cpp:374  NSClient++ 0,4,0,156 2012-04-05 x64 Loading settings and logger...
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: base-path
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: exe-path
d ttings_manager\settings_manager_impl.cpp:162  Boot.ini found in: C:/Program Files/NSClient++//boot.ini
d ttings_manager\settings_manager_impl.cpp:179  Boot order: ini://${shared-path}/config
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: shared-path
d ttings_manager\settings_manager_impl.cpp:182  Activating: ini://${shared-path}/config
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini://${shared-path}/config
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: shared-path
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/iis.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/iis.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/iis.ini from ini:///C:/Program Files/NSClient++//config/iis.ini
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/test_app.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/test_app.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/test_app.ini from ini:///C:/Program Files/NSClient++//config/test_app.ini
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/nsclient.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/nsclient.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/nsclient.ini from ini:///C:/Program Files/NSClient++//config/nsclient.ini
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/sql.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/sql.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/sql.ini from ini:///C:/Program Files/NSClient++//config/sql.ini
l    ..\..\..\trunk\service\NSClient++.cpp:385  NSClient++ 0,4,0,156 2012-04-05 x64 booting...
d    ..\..\..\trunk\service\NSClient++.cpp:386  Booted settings subsystem...
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: shared-path
d    ..\..\..\trunk\service\NSClient++.cpp:453  On crash: restart: NSClientpp
d    ..\..\..\trunk\service\NSClient++.cpp:465  Archiving crash dumps in: C:/Program Files/NSClient++//crash-dumps
d    ..\..\..\trunk\service\NSClient++.cpp:532  booting::loading plugins
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: module-path
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: shared-path
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: CheckDisk
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: CheckEventLog
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: CheckHelpers
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: CheckSystem
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: NRPEServer
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: NSCAClient
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: Scheduler
d    ..\..\..\trunk\service\NSClient++.cpp:543  Processing plugin: CheckDisk.dll
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/CheckDisk.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:543  Processing plugin: CheckEventLog.dll
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/CheckEventLog.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:543  Processing plugin: CheckHelpers.dll
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/CheckHelpers.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:543  Processing plugin: CheckSystem.dll
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/CheckSystem.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:543  Processing plugin: NRPEServer.dll
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/NRPEServer.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:543  Processing plugin: NSCAClient.dll
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/NSCAClient.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:543  Processing plugin: Scheduler.dll
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/Scheduler.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:807  Loading plugin: CheckDisk
d    ..\..\..\trunk\service\NSClient++.cpp:807  Loading plugin: Event log Checker.
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/eventlog/real-time/filters
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/eventlog/real-time/filters
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/eventlog/real-time/filters
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/eventlog/real-time/filters
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/eventlog/real-time/filters
d    ..\..\..\trunk\service\NSClient++.cpp:807  Loading plugin: Helper function
d    ..\..\..\trunk\service\NSClient++.cpp:807  Loading plugin: CheckSystem
d dules\CheckEventLog\eventlog_wrapper.cpp:80   Attempting to match: Application with application
d dules\CheckEventLog\eventlog_wrapper.cpp:80   Attempting to match: Hardware Events with application
d dules\CheckEventLog\eventlog_wrapper.cpp:80   Attempting to match: Microsoft Office Alerts with application
d dules\CheckEventLog\eventlog_wrapper.cpp:80   Attempting to match: Security with application
d dules\CheckEventLog\eventlog_wrapper.cpp:80   Attempting to match: System with application
d    ..\..\..\trunk\service\NSClient++.cpp:807  Loading plugin: NRPE server
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: certificate-path
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: shared-path
d \trunk\modules\NRPEServer\NRPEServer.cpp:117  Allowed hosts definition: <IP Removed>(255.255.255.255)
d unk\modules\CheckSystem\PDHCollector.cpp:91   Loading counters...
d unk\modules\CheckSystem\PDHCollector.cpp:94   Loading counter: cpu = \238(_total)\6
d unk\modules\CheckSystem\PDHCollector.cpp:94   Loading counter: memory commit bytes = \4\26
d unk\modules\CheckSystem\PDHCollector.cpp:94   Loading counter: memory commit limit = \4\30
d unk\modules\CheckSystem\PDHCollector.cpp:94   Loading counter: uptime = \2\674
d \..\trunk\include\nrpe\server\server.cpp:40   Using cert: C:/Program Files/NSClient++//security/nrpe_dh_512.pem
d \..\trunk\include\nrpe\server\server.cpp:50   Attempting to bind to: :5666
d \..\trunk\include\nrpe\server\server.cpp:62   Bound to: :5666
d \..\trunk\include\nrpe\server\server.cpp:77   Thredpool containes: 10
d    ..\..\..\trunk\service\NSClient++.cpp:807  Loading plugin: NSCAClient
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/NSCA/client/handlers
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/NSCA/client/handlers
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/NSCA/client/handlers
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/NSCA/client/handlers
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/NSCA/client/handlers
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/NSCA/client/targets
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/NSCA/client/targets
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/NSCA/client/targets
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/NSCA/client/targets
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/NSCA/client/targets

C:\Program Files\NSClient++>nscp test
d    ..\..\..\trunk\service\NSClient++.cpp:374  NSClient++ 0,4,0,156 2012-04-05 x64 Loading settings and logger...
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: base-path
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: exe-path
d ttings_manager\settings_manager_impl.cpp:162  Boot.ini found in: C:/Program Files/NSClient++//boot.ini
d ttings_manager\settings_manager_impl.cpp:179  Boot order: ini://${shared-path}/config
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: shared-path
d ttings_manager\settings_manager_impl.cpp:182  Activating: ini://${shared-path}/config
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini://${shared-path}/config
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: shared-path
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/iis.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/iis.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/iis.ini from ini:///C:/Program Files/NSClient++//config/iis.ini
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/test_app.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/test_app.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/test_app.ini from ini:///C:/Program Files/NSClient++//config/test_app.ini
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/nsclient.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/nsclient.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/nsclient.ini from ini:///C:/Program Files/NSClient++//config/nsclient.ini
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/sql.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/sql.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/sql.ini from ini:///C:/Program Files/NSClient++//config/sql.ini
l    ..\..\..\trunk\service\NSClient++.cpp:385  NSClient++ 0,4,0,156 2012-04-05 x64 booting...
d    ..\..\..\trunk\service\NSClient++.cpp:386  Booted settings subsystem...
d    ..\..\..\trunk\service\NSClient++.cpp:1412 Found key: shared-path
d    ..\..\..\trunk\service\NSClient++.cpp:453  On crash: restart: NSClientpp
d    ..\..\..\trunk\service\NSClient++.cpp:465  Archiving crash dumps in: C:/Program Files/NSClient++//crash-dumps

C:\Program Files\NSClient++>

I have run this both with the nscp service running and stopped. The results are the same.

[mickem]

Just out of curiosity could you try adding the new command line option --log-to-file or something like that (run nscp test --help to see if) could be --enable-file-log.

Also in the C:/Program Files/NSClient++/crash-dumps you should have a crash dump file which you can send to me (email) michael@… which will greatly help me figure out what is crashing...

Michael Medin

[mikep]

Here are my results:

C:\Program Files\NSClient++>nscp test --help
Allowed options (test):

Common options:
  --settings arg        Override (temporarily) settings subsystem to use
  --help                produce help message
  --debug               Set log level to debug (and show debug information)
  --log arg             The log level to use
  --version             Show version information

Test Options:
  --log-to-file         Enable file logger (defaults is console only)


C:\Program Files\NSClient++>nscp test --log-to-file
e source\nscp\trunk\service\cli_parser.hpp:239  Unable to parse command line (settings): unknown option log-to-file

C:\Program Files\NSClient++>

There are no crash-dumps folder generated from either the initially reported crashes or the message above.

mikep

[mickem]

Sorry been away over ester will look at this tonight.

For the crash dumps I need to verify that I start the writer elevated but please try to set the crash dump location to something publicly writable like c:\windows\temp or some such.

For the other stuff (two issues, first the option not working and secondly the crash) I need to investigate and will hopefully have a new build by tomorrow...

Michael Medin

[mikep]

I have emailed you a dump file.

[mickem]

Thank you, dumps worked like a charm.

All three issues resolved in next build.

Michael Medin

[mikep]

New results.

This command appears to work fine. It is logging to the log file and appears to be functioning correctly.

nscp test --log-to-file

This command crashes. I am emailing you the logs.

nscp test

The screen output is shown below.

C:\Program Files\NSClient++>nscp test
d    ..\..\..\trunk\service\NSClient++.cpp:374  NSClient++ 0,4,0,157 2012-04-10 x64 Loading settings and logger...
d ttings_manager\settings_manager_impl.cpp:162  Boot.ini found in: C:/Program Files/NSClient++//boot.ini
d ttings_manager\settings_manager_impl.cpp:179  Boot order: ini://${shared-path}/config
d ttings_manager\settings_manager_impl.cpp:182  Activating: ini://${shared-path}/config
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini://${shared-path}/config
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/iis.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/iis.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/iis.ini from ini:///C:/Program Files/NSClient++//config/iis.ini
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/test_app.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/test_app.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/klondike_app.ini from ini:///C:/Program Files/NSClient++//config/klondike_app.ini
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/nsclient.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/nsclient.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/nsclient.ini from ini:///C:/Program Files/NSClient++//config/nsclient.ini
d ttings_manager\settings_manager_impl.cpp:73   Creating instance for: ini:///C:/Program Files/NSClient++//config/sql.ini
d \trunk\include\settings/settings_ini.hpp:268  Reading INI settings from: C:/Program Files/NSClient++//config/sql.ini
d \trunk\include\settings/settings_ini.hpp:234  Loading: C:/Program Files/NSClient++//config/sql.ini from ini:///C:/Program Files/NSClient++//config/sql.ini
l    ..\..\..\trunk\service\NSClient++.cpp:385  NSClient++ 0,4,0,157 2012-04-10 x64 booting...
d    ..\..\..\trunk\service\NSClient++.cpp:386  Booted settings subsystem...
d    ..\..\..\trunk\service\NSClient++.cpp:453  On crash: restart: NSClientpp
d    ..\..\..\trunk\service\NSClient++.cpp:465  Archiving crash dumps in: c:\temp
IDetected crash...
IRunning: "C:\Program Files\NSClient++\reporter.exe" "archive" "C:\Users\myuser\AppData\Local\Temp\2\\b2057c25-fe26-4061-9774-2cb91b6f8cb1.dmp" "NSClient++" "0,4,0,157" "2012-04-10" "c:\temp"
d    ..\..\..\trunk\service\NSClient++.cpp:532  booting::loading plugins
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: CheckDisk
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: CheckEventLog
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: CheckHelpers
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: CheckSystem
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: NRPEServer
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: NSCAClient
d    ..\..\..\trunk\service\NSClient++.cpp:298  Found: Scheduler
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/CheckDisk.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/CheckEventLog.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/CheckHelpers.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/CheckSystem.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/NRPEServer.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/NSCAClient.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:830  addPlugin(C:/Program Files/NSClient++//modules/Scheduler.dll as )
d    ..\..\..\trunk\service\NSClient++.cpp:807  Loading plugin: CheckDisk
d    ..\..\..\trunk\service\NSClient++.cpp:807  Loading plugin: Event log Checker.
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/eventlog/real-time/filters
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/eventlog/real-time/filters
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/eventlog/real-time/filters
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/eventlog/real-time/filters
d ude\settings/settings_interface_impl.hpp:390  Get sections for: /settings/eventlog/real-time/filters
d    ..\..\..\trunk\service\NSClient++.cpp:807  Loading plugin: Helper function
d    ..\..\..\trunk\service\NSClient++.cpp:807  Loading plugin: CheckSystem
d    ..\..\..\trunk\service\NSClient++.cpp:807  Loading plugin: NRPE server
d \trunk\modules\NRPEServer\NRPEServer.cpp:117  Allowed hosts definition: 10.1.253.16(255.255.255.255)
d dules\CheckEventLog\eventlog_wrapper.cpp:80   Attempting to match: Application with application
d dules\CheckEventLog\eventlog_wrapper.cpp:80   Attempting to match: Hardware Events with application
d dules\CheckEventLog\eventlog_wrapper.cpp:80   Attempting to match: Microsoft Office Alerts with application
d dules\CheckEventLog\eventlog_wrapper.cpp:80   Attempting to match: Security with application
d dules\CheckEventLog\eventlog_wrapper.cpp:80   Attempting to match: System with application
d unk\modules\CheckSystem\PDHCollector.cpp:91   Loading counters...
d unk\modules\CheckSystem\PDHCollector.cpp:94   Loading counter: cpu = \238(_total)\6
d unk\modules\CheckSystem\PDHCollector.cpp:94   Loading counter: memory commit bytes = \4\26
d unk\modules\CheckSystem\PDHCollector.cpp:94   Loading counter: memory commit limit = \4\30
d unk\modules\CheckSystem\PDHCollector.cpp:94   Loading counter: uptime = \2\674
IRunning: "C:\Program Files\NSClient++\reporter.exe" "restart" "NSClientpp"

C:\Program Files\NSClient++>

Mike

[mickem]

Ahh...
Think I found what is the culprit...

Strange that I have never run into this myself.
When I "switch logger" (hat test does) I never kill the background thread...

I am building a version where I added this but I have not tested it at all so use with care. I will run some tests when I get home after works....

[mikep]

With build 159, I'm still seeing the same results as my last reply. I'm emailing you the new dump files.

[mickem]

Is it reproducible? (ie. always happening every time or just intermittently)
Could you also send the config as I said I don't get any crashes at all here so it is difficult to verify :(

[mickem]

Think I nailed it this time...
Building now so a new build out in a bit...

Michael Medin

[mikep]

It looks like you fixed this issue. I'll continue testing the rest of the funcitonality, but the command "nscp test" works.