Configuration & Administration

Expand all | Collapse all

Zenoss Shows Everything Down

  • 1.  Zenoss Shows Everything Down

    Posted 11-21-2018 05:02 PM
    So yesterday my zenoss decided to go nutty. EVERYTHING shows down. I can't ping anything from the GUI but if I go into the CLI and ping the devices, everything works fine.

    Where would a good place to start looking be?

    ------------------------------
    Tim Meads
    NOC Supervisor
    Mountain West Technologies Network
    ------------------------------


  • 2.  RE: Zenoss Shows Everything Down

    Posted 11-22-2018 02:37 AM
    Hi,

    Share your version and share the error in the system log or the error in the screen when you try to login


    ------------------------------
    nandha K
    CompanyName
    ------------------------------



  • 3.  RE: Zenoss Shows Everything Down

    Posted 11-23-2018 12:16 AM

    Hi,

    Share your version and share the error in the system log or the error in the screen when you try to login
    nandha K,  11-22-2018 03:37 AM
    I just upgraded to the latest version, 6.1.2, and I'm not sure which error log you want to see.....I'm not even sure where the CLI logs for zenoss are at.

    I can sign in fine without an issue...I don't have any problems talking to anything from the CLI but I cannot get anything in Zenoss to ping using the WebGUI, so likely there is something broken between the softeware and the networking on the server.

    ------------------------------
    Tim Meads
    NOC Supervisor
    Mountain West Technologies Network
    ------------------------------



  • 4.  RE: Zenoss Shows Everything Down

    Posted 11-23-2018 02:10 AM

    Yes I can ping itself and in zenoss it was showing the server itself being the only device up. Out of all of the services, only the zenmail / zenpop was stopped. Zenmail i figured would have died as it couldn't talk to the mail server.

    I noticed that when I attacked to the zope service, I couldn't ping anything but when I was on the command line before attaching to the service I could. I hadn't a clue about attaching to the service container before running an upgrade to the system yesterday. So there's that for learning something new. After finding out that I couldn't ping inside the container, I restarted docker from the command line and it started working. I could then ping my main switch gateways. Which is shdown below. Before it would fail the pings.

    Now I rebooted the server after I started having this problem and it didn't correct the issue. So I'm not sure how restarting docker would have fixed it....


    root@zenoss [~] : serviced service attach zope/0

    [root@639fccad075a /]# ping 10.58.0.1

    PING 10.58.0.1 (10.58.0.1) 56(84) bytes of data.

    64 bytes from 10.58.0.1: icmp_seq=1 ttl=254 time=0.900 ms

    64 bytes from 10.58.0.1: icmp_seq=2 ttl=254 time=0.775 ms

    64 bytes from 10.58.0.1: icmp_seq=3 ttl=254 time=0.734 ms

    ^C

    --- 10.58.0.1 ping statistics ---

    3 packets transmitted, 3 received, 0% packet loss, time 2001ms

    rtt min/avg/max/mdev = 0.734/0.803/0.900/0.070 ms

    [root@639fccad075a /]# ping 10.32.0.2

    PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data.

    64 bytes from 10.32.0.2: icmp_seq=1 ttl=63 time=0.732 ms

    64 bytes from 10.32.0.2: icmp_seq=2 ttl=63 time=0.429 ms

    ^C

    --- 10.32.0.2 ping statistics ---

    2 packets transmitted, 2 received, 0% packet loss, time 1000ms

    rtt min/avg/max/mdev = 0.429/0.580/0.732/0.153 ms

    [root@639fccad075a /]#



    ------------------------------
    Tim Meads
    NOC Supervisor
    Mountain West Technologies Network
    ------------------------------



  • 5.  RE: Zenoss Shows Everything Down

    Posted 01-31-2019 11:21 PM
    Same thing happened again tonight...Zenoss Crashed...I had hell getting it running again and when I did, nothing would be able to ping. It seems that when iptables was running and docker started, I couldn't get zenoss to talk to anything. Shutting down iptables then allows me to get into zenoss, but it wouldn't ping anything. After restarting docker (With iptables already shut down), everything comes back up. This is bizare.

    ------------------------------
    Tim Meads
    NOC Supervisor
    Mountain West Technologies Network
    ------------------------------



  • 6.  RE: Zenoss Shows Everything Down

    Posted 11-22-2018 03:04 AM
    Can you logon to the GUI?

    Have you run out of disk space?

    Run the toolbox tools - https://support.zenoss.com/hc/en-us/articles/203117595-How-To-Install-And-Use-the-zenoss-toolbox

    Is your Zenoss server a monitored device?  Can it ping itself?

    What version are you using?  Are all the daemons running?

    Check log files - I'd start with zenhub.log.

    Cheers,
    Jane

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------



  • 7.  RE: Zenoss Shows Everything Down

    Posted 11-22-2018 08:52 AM
    Check if any serviced or java processes are using 100% of a core on the host. if so, you'll need to log into Control Center to see which process is pinned and subsequently failing. Also, do any services show as failed in Control Center?

    ------------------------------
    Jason Olson
    ------------------------------



  • 8.  RE: Zenoss Shows Everything Down

    Posted 02-02-2019 01:16 AM
    Consider this about firewall settings: Chapter 3 Networking
    https://www.zenoss.com/services-support/documentation/planning-guide

    ------------------------------
    Arthur
    ------------------------------



  • 9.  RE: Zenoss Shows Everything Down

    Posted 02-21-2019 09:57 AM

    Hey Tim,

    When you say "everything shows as down" do you mean you actually have device down events under /Status/Ping for every device in your system, or simply that all of the devices show a status of DOWN if you navigate to them?  I ask because there is currently a known issue which hasn't been addressed yet where the presence of a Critical level event for a device anywhere under the /Status event class will cause that device to be marked as down.  That should hopefully be fixed soon in an upcoming release, but for the time being you probably want to add an event transform to /Status to downgrade the severity of all events from Critical to Error level except for the ones in /Status/Ping.



    ------------------------------
    Ryan Matte
    ------------------------------



  • 10.  RE: Zenoss Shows Everything Down

    Posted 02-21-2019 10:00 AM
    Just to add, if your issue is related to what I described in my last note then clearing any Critical events from /Status by hand (closing them) will get the devices marked as Up again until any further events like that come in.  The transform I mentioned would be to prevent further events coming in as Critical under /Status except for /Status/Ping (which is required for normal ping monitoring to operate properly).

    ------------------------------
    Ryan Matte
    ------------------------------