Configuration & Administration

Expand all | Collapse all

Zenoss 4.2.5 Zenup 732 - Production State problem

  • 1.  Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 05-09-2018 11:30 AM

    A change in production state does not take effect in regards to monitoring until a daemon restart?

    I've recently noticed some funny behaviour regarding the production states and how they affect monitoring of devices after moving to Zenoss 4.2.5 Zenup SUP 732

     

    Previously, on Zenoss 425 SUP 671  if a device is set to decommissioned (i.e. -1, or any production state value less than its zProdStateThreshold value – which defaults to 300 for devices) it would no longer monitor the device via ping or by snmp and no further events would be raised until the production state was set back to something above the zProdStateThreshold.

     

    However, I've noticed on SUP732 that this is no longer the case and setting the production state lower than the zProdStateThreshold does not prevent monitoring or events being raised.

    With further experiments I discovered that once prod state is lower than the threshold, restarting the zenping will stop ping monitoring and restarting the zenperfsnmp daemons will stop snmp monitoring.

     

    This behaviour is also similar once re-raising the production state whilst monitoring is off. Until the ping and snmp daemons are restarted monitoring for each protocol does not begin again…

     

    Is this a bug that has crept in during the 732 update? – I noticed that 732 has added a whole set of things around production state under zenutils and introduced a ProdStateManager.

     

    Has anyone else noticed this behaviour or are they able to recreate it to confirm my findings?

     

    And any ideas of how to rectify it so that reducing the production state below the zProdStateThreshold once again stops monitoring as it is supposed to?



    ------------------------------
    Thanks
    Pheripheral

    ------------------------------


  • 2.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 05-16-2018 09:18 AM
    I have certainly seen glitches with SUP732 if a device does not have its productionState attribute set and needed to manually fix a few devices, but I haven't seen the problems you describe.

    You might do a little digging around with zendmd to see what is going on; for example:


    >>> for d in dmd.Devices.getSubDevices():
                if d.getProductionState() < 1000:
                        print d.id, d.getProductionState()

    There is also a setProdState method, eg:
    d.setProdState(300)

    Cheers,
    Jane

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------



  • 3.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 05-17-2018 10:54 AM
    I've taken a look at things in zendmd and this seems to only confirm the behaviour is not working on my zenoss with 732 installed

    I can see in zendmd that the production state is set to -1 (Decommisioned) and yet i'm still getting events raised on my device.
    Once I restart the zenping / zenperfsnmp daemons the  events stop.

    Setting the production state back to 1000 (Production) similarly does not restart events until the daemon(s) are restarted.

    I can also produce this behaviour when setting the production states via zendmd rather than in the gui:

    for d in dmd.Devices.getSubDevices():
        d.setProdState(-1)
        commit()

    again monitoring does not cease until daemon restart unfortunately.

    I've compared this with an instance of zenoss I have with SUP 671 installed and the production state changes operates as it should, with monitoring ceasing when state is changed without needing daemon restarts...

    I may have to roll back to Sup 671 if i can't seem to shake this strange behaviour in the 732 instance.








    ------------------------------
    Pheripheral Pheripheral
    CompanyName
    ------------------------------



  • 4.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 05-18-2018 03:07 AM
    Sounds like you have found a real issue. Have you logged a JIRA ticket for this?
    Cheers,
    Jane

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------



  • 5.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 05-21-2018 05:45 AM
    Edited by Pheripheral Pheripheral 05-22-2018 04:43 AM
    Hi,

    Jira ticket raised:
    ZEN-30167 - Zenoss Core 4.2.5 with SUP 732 - Production State < zProdThreshold does not prevent monitoring
    (although it doens't seem to be viewable by the public)

    I recently create a new build of zenoss but only installed SUP 671 and the production state performs as expected without needing to restart daemons to stop / start monitoring when setting things to decommissioned / Production.
    I then upgraded to SUP 732 and the production state ceased to perform as expected, so i'm reasonably convinced that this is a problem introduced in 732...

    ------------------------------
    Pheripheral Pheripheral
    CompanyName
    ------------------------------



  • 6.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 05-22-2018 02:55 AM
    Yup - that is the correct Jira.  Also copy John Boyle -  @John -, the Community manager; otherwise it will end up as a private ticket that no-one else can see.  Please could you post the ticket number here when you have it?

    Thanks for this - I will try and find time to confirm I see the same thing.

    Cheers,
    Jane​

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------



  • 7.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 05-22-2018 05:48 AM
    Ah oops , I updated my above post before spotting your reply...

    Yes I've raised a ticket - ZEN-30167


    ------------------------------
    Pheripheral Pheripheral
    CompanyName
    ------------------------------



  • 8.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 05-22-2018 07:50 AM
    Thanks for sharing the ticket. I updated the status so it is accessible to you. Cheers! - JB

    ------------------------------
    John Boyle
    Technology Partnerships & Community Manager
    Zenoss
    Austin TX
    jboyle@zenoss.com
    http://www.zenoss.com
    ------------------------------



  • 9.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 05-23-2018 03:29 AM
    The link to this ticket is  https://jira.zenoss.com/browse/ZEN-30167 (and it is public - thanks John).
    Cheers,
    Jane

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------



  • 10.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 05-31-2018 01:40 AM
    Hi

    I have done some testing on this issue (as I needed to upgrade to latest RPS to support OpenSSH 7.x), and it appears that if you change the productionstate on a device, then you need to issue a "Push Changes" before it takes effect.

    So I guess the problem is that the ProductionState change functionality has been changed and now it just doesnt push the changes to collectors

    Thomas

    ------------------------------
    Thomas Luther
    Senior Consultant
    Netic A/S
    ------------------------------



  • 11.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 06-05-2018 12:14 AM
    Maybe we should create telegram group for more discussing about 
    1. How to install zenoss on Ubuntu, CentOS, etc
    2. Configuration and the other things that belongs about zenoss core





  • 12.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 06-06-2018 04:48 AM
    Hi Willy,
    Why do you suggest a new forum for discussing installation and configuration of Zenoss?  Isn't that what this forum is for?

    Cheers,
    Jane

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------



  • 13.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 07-05-2018 06:12 AM
    I created a patch for this, it seems to work in my lab, but if anyone else wants to test I can post the patch file.

    ------------------------------
    jstanley
    ------------------------------



  • 14.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 07-05-2018 11:42 AM
    Edited by Pheripheral Pheripheral 07-05-2018 11:42 AM

    Yes, i'd be very interested in seeing the patch!

    I did some quick playing around with this adding an extra line or two to push the changes in my custom bit of code i was using to toggle between 2  production state via one custom button press, and i think it seems to work once i added the line device._p_changed = True which seems to be what pushes it to the collectors. This may be the wrong way to do it though, and ti certianly won't fix the normal prod state change functionality.

            prodState = device.productionState
            if prodState == self.PRODSTATE1:        

    device.setProdState(self.PRODSTATE2)    
    device._p_changed = True                 
    successFlag = True





    ------------------------------
    Pheripheral Pheripheral
    CompanyName
    ------------------------------



  • 15.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 07-05-2018 12:15 PM
      |   view attached
    Yup, you were on the right track. Just a one line edit.

    drop this in your $ZENHOME, then
    patch -p0 < ZEN-30167.patch

    restart zenoss
    (push to remote hubs and collectors)

    ------------------------------
    jstanley
    ------------------------------

    Attachment(s)

    txt
    ZEN-30167.patch.txt   451B 1 version


  • 16.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 07-06-2018 03:52 AM
    Fantastic... I have just tested it on our development zenoss 4.2.5 with SP732 applied... and now when I change productionState on a device from Production (1000) to Decomissioned (-1) then after a short period, the checks stops... and when changing back to any higher productionState (300 + ) then checks starts coming in again after a short periode...

    So thanks a lot for this very simple fix... I have added a comment to the JIRA https://jira.zenoss.com/browse/ZEN-30167

    ------------------------------
    Thomas Luther
    Senior Consultant
    Netic A/S
    ------------------------------



  • 17.  RE: Zenoss 4.2.5 Zenup 732 - Production State problem

    Posted 07-06-2018 09:56 AM
    You fantastic fellow! Thanks Jay.

    ------------------------------
    Jane Curry
    Skills 1st United Kingdom
    jane.curry@skills-1st.co.uk
    ------------------------------