Configuration & Administration

Expand all | Collapse all

zencommand suddenly using high amounts of memory

  • 1.  zencommand suddenly using high amounts of memory

    Posted 10-08-2018 08:51 AM
    So suddenly about a month ago our zencommand process went from around 5GB memory to 15GB memory. I recycle and it helps, for a little while. How high should this be set to? What would cause it to suddenly use so much memory?
    YZEKqUucSBO69h5ute0L_2018-10-08_8-49-29.png

    ------------------------------
    Paul Giordano
    Senior Systems Engineer
    Zethcon Corporation
    ------------------------------


  • 2.  RE: zencommand suddenly using high amounts of memory

    Posted 10-11-2018 04:49 AM
    A few thoughts...
    Doesn't look like the number of devices polled has changed over the last few hours but has it grown substantially over the last month?

    Check zencommand .log to see if there are significant timeouts going on which might result in lots more processes actually running at one time and in zencommand checking threads to see if an answer has been received.

    Do you have "top" records for your Zenoss server to see if anything else has changed?

    What happened in your Zenoss world "about a month ago" that might have caused this?  Extra devices added?  "Big" devices added eg. big routers with lots of interfaces, VMware devices added?  New ZenPack or version of ZenPack installed eg. updated Linux ZenPack?

    Cheers,
    Jane

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------



  • 3.  RE: zencommand suddenly using high amounts of memory

    Posted 20 days ago
    Apologies for taking forever to respond. There has been no significant increase in the number or types of servers. The zencommand continues to use more and more memory, now requesting 16GB. Just seems to go up and up. I cannot find the zencommand.log file, it's not in /opt/zenoss/log - is there something I need to turn on somewhere? I'm running with all of the defaults.

    ------------------------------
    Paul Giordano
    Senior Systems Engineer
    Zethcon Corporation
    ------------------------------



  • 4.  RE: zencommand suddenly using high amounts of memory

    Posted 20 days ago
    The log file ought to be in /opt/zenoss/log - in the zencommand container.  Do you have separate collectors?  If so, the logfile will be under a subdirectory with the collector's name.

    You might have something going on whereby LOTS of commands are starting and either never timing out or take a long time to time out.  What is your zCommandCommandTimeout zProperty typically set to?

    Have a look with top and see if there are lots of processes that have been started by zencommand.

    Cheers,
    Jane

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------



  • 5.  RE: zencommand suddenly using high amounts of memory

    Posted 20 days ago
    Edited by Paul Giordano 20 days ago
    Thanks Jane - checking the logs. The zCommandTimeout is set to 15 seconds. Top, though, yields over 3000 tasks, almost all of them "Hiding the command line arguments" and "tail -n1". So I think we're onto the problem - but how do I figure out what all of those tasks are? Some of them have been around for days, back to the last time I restarted Zenoss.

    ------------------------------
    Paul Giordano
    Senior Systems Engineer
    Zethcon Corporation
    ------------------------------



  • 6.  RE: zencommand suddenly using high amounts of memory

    Posted 19 days ago
    The logs indicated some issues, which I corrected - but every 5 minutes one of the "Hiding" and "tail -n1" are hanging out there. I came in to several hundred this AM. How do I figure out what they are? Is there some debug I can issue that will generate the command rather than hiding it?

    ------------------------------
    Paul Giordano
    Senior Systems Engineer
    Zethcon Corporation
    ------------------------------



  • 7.  RE: zencommand suddenly using high amounts of memory

    Posted 19 days ago
    So I restarted Zenoss this morning, and lo and behold the problem is not occurring any more. Sheesh. Thanks for all the help.

    ------------------------------
    Paul Giordano
    Senior Systems Engineer
    Zethcon Corporation
    ------------------------------



  • 8.  RE: zencommand suddenly using high amounts of memory

    Posted 19 days ago
    I suspect that the problem has "gone away" because of the restart.  I am afraid itr may come back over time :)

    You end up with lots of processes started by zencommand and then they get orphaned - for some reason, they do not get properly killed off after the zCommandTimeout time.  Hope I am wrong on this but keep yours eyes on it ;)

    Look at the scripts that are being run and consider whether they might be re-written to ensure they get properly killed rather than becoming zombies.  You might look at running a script from cron that kills any such processes that are more than a few minutes old.

    Cheers,
    Jane

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------