InstallationGetting Started

Expand all | Collapse all

Upgrade looks like it worked but

  • 1.  Upgrade looks like it worked but

    Posted 03-18-2021 09:50 AM
    web page is getting the error - no public endpoint endpoints available - I attached the upgrade log the only error I see is a time out for line 7
    Script started on Thu 18 Mar 2021 09:36:32 AM EDT
    I0318 09:36:33.317283 26411 runner.go:159] executing step 0: DESCRIPTION Persist current version number of Zenoss Core
    I0318 09:36:33.317387 26411 runner.go:159] executing step 1: REQUIRE_SVC
    I0318 09:36:33.317399 26411 eval.go:340] checking service requirement
    I0318 09:36:33.317413 26411 eval.go:344] verifying service a29uwksqyxmb3q6a63y56env8
    I0318 09:36:33.317737 26411 eval.go:350] found a29uwksqyxmb3q6a63y56env8 tenant id for service a29uwksqyxmb3q6a63y56env8
    I0318 09:36:33.317756 26411 runner.go:159] executing step 2: SVC_USE zenoss/core_5.1:5.1.3_1
    I0318 09:36:33.317772 26411 eval.go:70] preparing to use image: zenoss/core_5.1:5.1.3_1
    I0318 09:36:33.317794 26411 service_client.go:32] Pulling zenoss/core_5.1:5.1.3_1, tagging to latest, and pushing to registry localhost:5000 - this may take a while
    I0318 09:36:34.809740 26411 eval.go:79] Successfully pulled and tagged new image zenoss/core_5.1:5.1.3_1
    I0318 09:36:34.809773 26411 runner.go:159] executing step 3: SVC_START Zenoss.core/Infrastructure/mariadb
    I0318 09:36:35.297059 26411 eval.go:170] starting service Zenoss.core/Infrastructure/mariadb 4095hvjxzyvsazpdt3l79z28h
    I0318 09:36:35.383577 26411 runner.go:159] executing step 4: SVC_START Zenoss.core/Infrastructure/RabbitMQ
    I0318 09:36:35.929092 26411 eval.go:170] starting service Zenoss.core/Infrastructure/RabbitMQ 7sek1x9h67fm0v3hljnrmyeex
    I0318 09:36:36.130112 26411 runner.go:159] executing step 5: SVC_START Zenoss.core/Zenoss/Events/zeneventserver
    I0318 09:36:36.624419 26411 eval.go:170] starting service Zenoss.core/Zenoss/Events/zeneventserver 89y96jsm0z6n6l6erw9nkos67
    I0318 09:36:37.247918 26411 runner.go:159] executing step 6: SVC_START Zenoss.core/Infrastructure/redis
    I0318 09:36:38.502775 26411 eval.go:170] starting service Zenoss.core/Infrastructure/redis abhn75z8hhv872zd1j5whbo5d
    I0318 09:36:38.822773 26411 runner.go:159] executing step 7: SVC_WAIT Zenoss.core/Infrastructure/mariadb Zenoss.core/Infrastructure/RabbitMQ Zenoss.core/Zenoss/Events/zeneventserver Zenoss.core/Infrastructure/redis started 1200
    I0318 09:36:41.388952 26411 eval.go:134] waiting 1200 for services Zenoss.core/Infrastructure/mariadb, Zenoss.core/Infrastructure/RabbitMQ, Zenoss.core/Zenoss/Events/zeneventserver, Zenoss.core/Infrastructure/redis to be started
    
    E0318 09:56:41.419410 26411 runner.go:161] error executing step 7: SVC_WAIT: timeout
    I0318 09:56:41.419452 26411 runner.go:151] Executing exit functions
    timeout

    Script done on Thu 18 Mar 2021 09:56:41 AM EDT





    ------------------------------
    License Administrator
    Ithaca CSD
    ------------------------------


  • 2.  RE: Upgrade looks like it worked but

    Posted 03-18-2021 11:08 AM
    Edited by License 03-18-2021 11:14 AM
    Looks like a restart fixed that issue or at least restarted the Zenoss 5.12 version.

    ------------------------------
    License Administrator
    Ithaca CSD
    ------------------------------



  • 3.  RE: Upgrade looks like it worked but

    Posted 03-18-2021 01:37 PM
    Well tried it again through putty and it worked better. May have been the fact that I had tried the script again after renaming the original folder. This time I used the original files and got to step 12 with out a error - not sure what to do with this error.

    File "/usr/lib64/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    urllib2.HTTPError: HTTP Error 401: Unauthorized
    exit status 1
    E0318 14:21:02.605105 03080 runner.go:161] error executing step 12: SVC_EXEC: exit status 1
    I0318 14:21:02.605184 03080 runner.go:151] Executing exit functions
    I0318 14:21:02.605223 03080 eval.go:51] restoring snapshot a29uwksqyxmb3q6a63y56env8_20210318-181849.210
    exit status 1

    Script done on Thu 18 Mar 2021 02:24:44 PM EDT

    ------------------------------
    License Administrator
    Ithaca CSD
    ------------------------------



  • 4.  RE: Upgrade looks like it worked but

    Posted 03-23-2021 03:48 PM
    Edited by Arthur 03-23-2021 03:56 PM
      |   view attached
    Hi Lic

    The error tells you that the upgrade tried to access an URL and it got a 401 (Unauthorized) Error back. If I Iook at the ugrade_core.txt script, step 12 tries to start the zenevenserver.
    SVC_WAIT Zenoss.core/Infrastructure/solr Zenoss.core/Zenoss/Events/zeneventserver started 600

    So what you can do is, try the upgrade again and if it stays at the same line login to the control center and  try to restart zeneventserver by hand. The services which have to be up for the upgrade are:

    # Start all our dependent services
    SVC_START Zenoss.core/Infrastructure/mariadb
    SVC_START Zenoss.core/Infrastructure/RabbitMQ
    SVC_START Zenoss.core/Zenoss/Events/zeneventserver
    SVC_START Zenoss.core/Infrastructure/redis

    I've attached the two upgrade scripts which are located in the zope container.



    ------------------------------
    Arthur
    ------------------------------

    Attachment(s)

    txt
    upgrade-core.txt   3 KB 1 version


  • 5.  RE: Upgrade looks like it worked but

    Posted 03-23-2021 03:51 PM
    Edited by Arthur 03-23-2021 04:06 PM
    upgrade-core.sh for reference. This script starts the upgrade-core.txt



    #!/bin/sh
    #
    ###############################################################################
    #
    # Upgrade Zenoss Core
    #
    ###############################################################################
    set -e
    export SERVICE_TENANT_ID="`serviced service list core --format='{{.ID}}'`"
    export UPGRADE_REINDEX
    export REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS
    serviced snapshot untag $SERVICE_TENANT_ID preupgrade-core-6.3.2
    
    FROM_VERSION=$(serviced service list core --format='{{.Version}}')
    TO_VERSION=6.3.2
    
    if [ -z "$FROM_VERSION" ];
    then
        FROM_VERSION=$(serviced service list core --format '{{.ImageID}}' | awk '{ print "localhost:5000/"$0" cat /opt/zenoss/Products/ZenModel/ZVersion.py" }' | xargs docker run --rm | grep VERSION | cut -d'"' -f2)
    fi
    
    if [ -z "$FROM_VERSION" ];
    then
        FROM_VERSION=$(grep 'SVC_USE zenoss/resmgr_[^:_]*:.*' /root/6.3.x/current_version.txt | cut -d':' -f2 | cut -d'_' -f1)
        if [ -z "$FROM_VERSION" ];
        then
            echo "Could not get the previous Core version number from serviced, please edit current_version.txt to set it."
            echo "Change this line 'SVC_USE zenoss/core_REPLACE_1' to reflect the pre-upgrade version, for example:"
            echo "SVC_USE zenoss/core_5.2:5.2.6_1"
            exit 1
        else
            echo "Using FROM_VERSION provided by change to current_version.txt"
        fi
    fi
    
    FROM_VERSION_SHORT=$(cut -d'.' -f1,2 <<< $FROM_VERSION)
    sed -i 's/REPLACE/'"$FROM_VERSION_SHORT:$FROM_VERSION"'/g' /root/6.3.x/current_version.txt
    sed -i "s/%FROM_VERSION%/$FROM_VERSION/g" /root/6.3.x/current_version.txt
    serviced script run /root/6.3.x/current_version.txt --service Zenoss.core
    
    if [ $(echo "${FROM_VERSION}" | cut -c1) -eq 5 ] && [ $(echo "${TO_VERSION}" | cut -c1) -eq 6 ];
    then
        UPGRADE_REINDEX='SVC_EXEC NO_COMMIT "Zenoss.core/Zenoss/User Interface/Zope" su - zenoss -c "/opt/zenoss/bin/upgrade_reindex.sh Zenoss.core"'
        REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS='SVC_EXEC NO_COMMIT "Zenoss.core/Zenoss/User Interface/Zope" /opt/zenoss/bin/zenmigrate --step=RemoveEmptyProductClassRelationships --dont-bump'
    fi
    
    sed -i "s@%UPGRADE_REINDEX%@${UPGRADE_REINDEX}@g; s@%REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS%@${REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS}@g;" /root/6.3.x/upgrade-core.txt
    
    serviced script run /root/6.3.x/upgrade-core.txt --service Zenoss.core
    /opt/serviced/bin/serviced-set-version Zenoss.core 6.3.2
    ​




  • 6.  RE: Upgrade looks like it worked but

    Posted 03-26-2021 08:40 AM
    Issue one with this script "# This script can be used to upgrade a Zenoss deployment to a new image
    #
    # To minimize downtime and avoid potential out of space disk issues,
    # preseed the master with the images (that the upgrade will automatically
    # pull from the internet) before the upgrade maintenance window:
    # ssh MASTER
    # sudo docker pull zenoss/core_6.3:6.3.2 # about 10-20 minutes
    # sudo docker pull zenoss/hbase:24.0.8 # about 5-10 minutes
    # sudo docker pull zenoss/opentsdb:24.0.8 # about 7-15 minutes
    # serviced docker sync # about 10m:core 5m:hbase 8m:opentsdb"

    Script started, file is /var/log/serviced/script-2021-03-26-091436-root.log
    could not parse line 60: %UPGRADE_REINDEX%
    Script done, file is /var/log/serviced/script-2021-03-26-091436-root.log

    and the other one is
    urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
    exit status 1

    I updated the certificate following the steps
    Control Center (zenoss.com) and
    Control Center (zenoss.com) 

    When I access the web interface the certificate is valid. I included the log files. The 4 services started correctly in the script.

    Thank you

    ------------------------------
    License Administrator
    Ithaca CSD
    ------------------------------



  • 7.  RE: Upgrade looks like it worked but

    Posted 03-27-2021 07:32 AM
    Edited by Arthur 03-27-2021 06:51 PM
    Hi Lic

    Based on this logfie script-2021-03-26-092205-root.txt  the upagrade passed not steps 0 -11. Then zenmigrate starte to add the Solr service which is new in 6.X. After that it failed trying to open a URL during the crtification verification.
    You said you updated the certificate. My I ask you why? And did you got the same error with your previous certificate?

    ------------------------------
    Arthur
    ------------------------------



  • 8.  RE: Upgrade looks like it worked but

    Posted 03-31-2021 06:52 AM
    I updated the certificate because it had expired.

    ------------------------------
    License Administrator
    Ithaca CSD
    ------------------------------



  • 9.  RE: Upgrade looks like it worked but

    Posted 04-03-2021 11:21 AM
    Hi Lic

    That's fair enough. If I understand it right you are  still using a self signed cert and your brwoser will probably complain it. The only advantage you have is that cert does not show as expired and the communication is secured based on your cert/key pair. Currently my knowlege about the migrate code is very little and I can't tell you where your problem is.

    My advices is to try the update with Zenoss internal cert and see if you fail at the same point.

    It also not worth to check your cert again and compair your  Zeoss endpoints with the one listed in the certificate because if you followed the procedure to create the certificate your endpoint names (Subject Alternative Name:) should look like:

    (In case your CC hostname is master.X.Y)

    X509v3 Subject Alternative Name:
    DNS:master.X.Y, DNS:*.master.X.Y

    So for the Solr endpoint it would be solr.master.X.Y and it should be resolvable.







    ------------------------------
    Arthur
    ------------------------------



  • 10.  RE: Upgrade looks like it worked but

    Posted 21 days ago
    Thanks for the help. We decided to create a new server using the OVF file.

    ------------------------------
    License Administrator
    Ithaca CSD
    ------------------------------



  • 11.  RE: Upgrade looks like it worked but

    Posted 19 days ago
    Hi Lic

    Thanks to let us know.

    Regards,

    ------------------------------
    Arthur
    ------------------------------