Configuration & Administration

Expand all | Collapse all

Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

  • 1.  Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 09-17-2018 01:21 PM
    Since installing Zenoss6 last year we have had random times when MetricShipper and Zope would fail and we would have to restart them to get back into Zenoss Core.  Normally restarting those two containers in Control Center would do the trick.  Last week we did this and while Zope came back up we haven't gotten graphs back.

    In Control Center zenhub and MetricShipper are showing health issues.  We are getting a failed health check for metric_consumer_answering for zenhub,  and fails for store_answering and websocket_opened on MetricShipper.

    Are these failed health checks related to the loss of graphs?  What should we be looking into to troubleshoot this issue further?


    ------------------------------
    Joseph Meslovich
    Network Administrator & IT Security Officer
    Bridgewater College
    Bridgewater VA
    540-828-5343
    ------------------------------


  • 2.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 09-18-2018 01:27 PM
    I'm having the exact same issue with Zenoss 6.2... not sure why.

    Solomon Hill
    Director of Technology
    Ravenswood City School District
    East Palo Alto, CA

    ------------------------------
    Solomon Hill
    ------------------------------



  • 3.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 09-19-2018 05:55 PM
    I am also experiencing this issue since 13/9/18.  Restarting zenhub and metricshipper not helping.

    ------------------------------
    Steven
    ------------------------------



  • 4.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 09-20-2018 02:49 PM
    Hi Joseph

    Yes, this could be.

    MetricShipper
    Inserts metrics into OpenTSDBWriter.
    This component ships with a default threshold. The maximum number of seconds MetricShipper needs to
    process its Redis queue at the current rate is 300. If this value is exceeded, the MetricShipper node on the
    metric pipeline turns gray and flashes.

    OpenTSDB
    Resource Manager no longer uses RRD files on the collectors for data storage. We have created a centralized
    storage framework for this data which uses a Redis key-value store on the collector and then ships that data to
    an OpenTSDB (time series database) instance that runs on Hadoop and HBase.

    Source:
    https://www.zenoss.com/sites/default/files/zenoss-doc/9856/base/admin/monself/self-monitor-components.html

    Cheers

    ------------------------------
    Arthur
    ------------------------------



  • 5.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 09-20-2018 02:52 PM
    Edited by Arthur 09-20-2018 03:01 PM
    To solve it

    Do a backup from the GUI but don't delete or overwrite older, perhabs good ones.

    then try:

    https://support.zenoss.com/hc/en-us/articles/211783563-Zenoss-Master-Staged-Startup-and-Shutdown-Best-Practices-for-Maintenance-

    If it does not help restore a known good backup taken before the failure occured.

    ------------------------------
    Arthur
    ------------------------------



  • 6.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 09-25-2018 06:30 PM
    I have followed the guidance from the staged shutdown/startup article provided.

    I encounter issues starting the MetricShipper service, it fails health checks for store_answering.  Consequently, when trying to start the zenhub service afterwards, it fails health checks for metric_consumer_answering.

    I am not seeing any other problems.

    Any guidance on how to get these services healthy would be greatly appreciated, as I don't have a good backup to resort to... I believe that fixing this will resolve my graphing issues.

    Thanks


    ------------------------------
    Steven
    ------------------------------



  • 7.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 10-01-2018 09:45 PM
    Must have been a resource issue in my case, as I was seeing various memory related errors in the logs.

    After reducing the number of monitored devices from 212 down to 190 (single master host), I tweaked the RAM requested for various services including zenhub, metricconsumer, zenpython, opentsdb and zenmodeler.  Doubled the default amount.

    Then after following the shutdown/startup guide it all came right.

    Very relieved, and have now made a good backup :)


    ------------------------------
    Steven
    ------------------------------



  • 8.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 11-12-2018 09:59 AM
    So it appears we were also having a resource issue here.  We were monitoring 85 Windows servers, 24 Linux servers, and 121 switches.  We had some staffing changes and the new Systems Administrator decided to go with Zabbix instead of Zenoss for server monitoring.  So after removing the Windows servers from Zenoss, the resource requirements dropped enough that MetricShipper and zenhub containers started working normally again.

    So if we had wanted to keep monitoring everything we would have also had to increase the resources of those containers.  When we initially installed we had also gone with the minimum recommend resources.  We did not explore what we would have needed to increase the resources to to properly size the Zenoss master for our environment.  We were only running the master and had not added any other hosts to the Zenoss cluster.


    ------------------------------
    Joseph Meslovich
    Network Administrator & IT Security Officer
    Bridgewater College
    Bridgewater VA
    540-828-5343
    ------------------------------



  • 9.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 11-14-2018 04:38 AM
    Thanks for these updates.  It would appear that Zenoss 6 doesn't always  "degrade nicely" when short of resources.

    It would be helpful if Zenoss could publish guidance on this sort of scenario.

    Cheers,
    Jane

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------



  • 10.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 11-14-2018 09:04 AM
    Edited by Jason Olson 11-14-2018 09:04 AM
    Heh. I think this is the only guidance we're going to get on things like this, Jane. :)

    ------------------------------
    Jason Olson
    ------------------------------



  • 11.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 11-14-2018 09:11 AM
    But there's no harm in asking ;)
    If a management system doesn't "degrade nicely" then I would hope that the vendor is addressing such an issue and would provide advice and guidance meantime.

    Is that unreasonable?

    Cheers,
    Jane

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------



  • 12.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 11-14-2018 09:20 AM
    Not in the slightest. Here's hoping they scan the forums every so often for feedback like this.

    ------------------------------
    Jason Olson
    ------------------------------



  • 13.  RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks

    Posted 03-29-2019 06:43 AM
    I'm also having this issue with Zenoss 6.2.1, check out my post from December:
    Zenoss 6.2.1, Zope stops answering on its own, unprovoked

    ------------------------------
    Jad
    ------------------------------