Configuration & Administration

Expand all | Collapse all

"Unknown connection problem to zenhub" messages in zenhub log

  • 1.  "Unknown connection problem to zenhub" messages in zenhub log

    Posted 05-21-2020 11:43 PM
    Edited by Larry 05-21-2020 11:55 PM
    We are running Zenoss 6.2.1 (Community). Recently we started getting a lot of these messages in the zenhub service log:

    CRITICAL zen.pbclientfactory: Unknown connection problem to zenhub <class 'twisted.spread.pb.PBConnectionLost'>

    We are getting 2-3K such messages a day. This started a few days ago. We recently made some configuration changes to resolve some performance issues:

    1. Increased zenhub instances: 1 -> 2
    2. Increased zenhub workers. Currently at 26.

    I know that the number of zenhub workers is WAY too high, but we're trying to resolve problems with missed runs and discarded events.

    Any ideas on how to resolve this error message?

    We are also seeing a lot of this message:

    INFO zen.ZenHub: Worker 11 (5394) reports 2020-05-22 04:09:28,693 INFO zen.pbclientfactory: Initial connect timed out after 30 seconds

    Thanks in advance,




    ------------------------------
    Larry
    ------------------------------


  • 2.  RE: "Unknown connection problem to zenhub" messages in zenhub log

    Posted 06-05-2020 04:11 PM
    Edited by Ryan Matte 06-05-2020 04:14 PM

    Hi, unfortunately you're likely running in to some known performance issues with the hub on older versions of Resource Manager.  The best I can recommend is to update to the latest version of Core that is available (6.3.2 I believe) which contains some fixes and then to upgrade to the next release (probaby 6.5.0 I think, but possibly  6.4.x, not sure how they plan to do the core release) once it comes out as it will contain further improvements to hub performance.  Having 26 workers is honestly likely counter-productive and you may want to reduce that number a bit.  We do not recommend having multiple zenhub instances as that causes problems of it's own.  In fact in newer versions of RM we don't even make it possible to increase the number of zenhub instances and we break out the workers in to their own services so that they can be increased individually as separate instances.  In any case, upgrades are going to be the way forward for you in terms of improving the performance of this.



    ------------------------------
    Ryan Matte
    ------------------------------



  • 3.  RE: "Unknown connection problem to zenhub" messages in zenhub log

    Posted 06-05-2020 09:30 PM
    Hi Ryan,
       Thanks for the responses. I will try your recommendations. As for upgrading Core, will 6.5.0 be available for Community users or only paying users? Any estimate on when the next Community release will be out?

    Thanks!

    ------------------------------
    Larry
    ------------------------------



  • 4.  RE: "Unknown connection problem to zenhub" messages in zenhub log

    Posted 06-09-2020 04:31 PM
    Edited by Ryan Matte 06-09-2020 04:31 PM
    Generally all Zenoss releases eventually become Core releases, however there is generally a significant delay for that.  The Core version generally trails behind by one version, so the next core release will probably be 6.4.1 and then 6.5.0 would be released sometime after that.  I however can't make any guarantees on that as I'm not involved in that decision making.

    ------------------------------
    Ryan Matte
    ------------------------------



  • 5.  RE: "Unknown connection problem to zenhub" messages in zenhub log

    Posted 06-22-2020 02:45 PM
    Thanks again Ryan. We tried upgrading our DEV installation from 6.2.1 -> 6.3.2, but the upgrade-core.sh script fails:

    ImportError: No module named jira.client

    ERROR:zen.ZenPackCmd:Command '['zenpack', '--install', '/tmp/ZenPacks.zenoss.Dashboard-1.3.3-py2.7.egg']' returned non-zero exit status 3

    We have a custom zenpack that uses jira. No ideas about dashboard. Any ideas?

    Thanks,

    ------------------------------
    Larry
    ------------------------------



  • 6.  RE: "Unknown connection problem to zenhub" messages in zenhub log

    Posted 06-05-2020 04:17 PM
    For the time being I would recommend that you move back to a single instance of zenhub and then increase the workers on that single instance as needed.  There is no benefit to having multiple master zenhub instances.  Only the number of workers should be increased as load increases.

    ------------------------------
    Ryan Matte
    ------------------------------