Upgrade looks like it worked but

6. RE: Upgrade looks like it worked but

Recommend

License

Posted 14 days ago

| view attached (3)

Issue one with this script "# This script can be used to upgrade a Zenoss deployment to a new image
#
# To minimize downtime and avoid potential out of space disk issues,
# preseed the master with the images (that the upgrade will automatically
# pull from the internet) before the upgrade maintenance window:
# ssh MASTER
# sudo docker pull zenoss/core_6.3:6.3.2 # about 10-20 minutes
# sudo docker pull zenoss/hbase:24.0.8 # about 5-10 minutes
# sudo docker pull zenoss/opentsdb:24.0.8 # about 7-15 minutes
# serviced docker sync # about 10m:core 5m:hbase 8m:opentsdb"

Script started, file is /var/log/serviced/script-2021-03-26-091436-root.log
could not parse line 60: %UPGRADE_REINDEX%
Script done, file is /var/log/serviced/script-2021-03-26-091436-root.log

and the other one is
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
exit status 1

I updated the certificate following the steps
Control Center (zenoss.com) and
Control Center (zenoss.com)

When I access the web interface the certificate is valid. I included the log files. The 4 services started correctly in the script.

Thank you

------------------------------
License Administrator
Ithaca CSD
------------------------------

Attachment(s)

script-2021-03-26-091436-root.txt 146 B 1 version

script-2021-03-26-092137-root.txt 4 KB 1 version

script-2021-03-26-092205-root.txt 7 KB 1 version

Original Message

Original Message:
Sent: 03-23-2021 03:51 PM
From: Arthur
Subject: Upgrade looks like it worked but

upgrade-core.sh for reference. This script starts the upgrade-core.txt

#!/bin/sh
#
###############################################################################
#
# Upgrade Zenoss Core
#
###############################################################################
set -e
export SERVICE_TENANT_ID="`serviced service list core --format='{{.ID}}'`"
export UPGRADE_REINDEX
export REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS
serviced snapshot untag $SERVICE_TENANT_ID preupgrade-core-6.3.2

FROM_VERSION=$(serviced service list core --format='{{.Version}}')
TO_VERSION=6.3.2

if [ -z "$FROM_VERSION" ];
then
    FROM_VERSION=$(serviced service list core --format '{{.ImageID}}' | awk '{ print "localhost:5000/"$0" cat /opt/zenoss/Products/ZenModel/ZVersion.py" }' | xargs docker run --rm | grep VERSION | cut -d'"' -f2)
fi

if [ -z "$FROM_VERSION" ];
then
    FROM_VERSION=$(grep 'SVC_USE zenoss/resmgr_[^:_]*:.*' /root/6.3.x/current_version.txt | cut -d':' -f2 | cut -d'_' -f1)
    if [ -z "$FROM_VERSION" ];
    then
        echo "Could not get the previous Core version number from serviced, please edit current_version.txt to set it."
        echo "Change this line 'SVC_USE zenoss/core_REPLACE_1' to reflect the pre-upgrade version, for example:"
        echo "SVC_USE zenoss/core_5.2:5.2.6_1"
        exit 1
    else
        echo "Using FROM_VERSION provided by change to current_version.txt"
    fi
fi

FROM_VERSION_SHORT=$(cut -d'.' -f1,2 <<< $FROM_VERSION)
sed -i 's/REPLACE/'"$FROM_VERSION_SHORT:$FROM_VERSION"'/g' /root/6.3.x/current_version.txt
sed -i "s/%FROM_VERSION%/$FROM_VERSION/g" /root/6.3.x/current_version.txt
serviced script run /root/6.3.x/current_version.txt --service Zenoss.core

if [ $(echo "${FROM_VERSION}" | cut -c1) -eq 5 ] && [ $(echo "${TO_VERSION}" | cut -c1) -eq 6 ];
then
    UPGRADE_REINDEX='SVC_EXEC NO_COMMIT "Zenoss.core/Zenoss/User Interface/Zope" su - zenoss -c "/opt/zenoss/bin/upgrade_reindex.sh Zenoss.core"'
    REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS='SVC_EXEC NO_COMMIT "Zenoss.core/Zenoss/User Interface/Zope" /opt/zenoss/bin/zenmigrate --step=RemoveEmptyProductClassRelationships --dont-bump'
fi

sed -i "s@%UPGRADE_REINDEX%@${UPGRADE_REINDEX}@g; s@%REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS%@${REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS}@g;" /root/6.3.x/upgrade-core.txt

serviced script run /root/6.3.x/upgrade-core.txt --service Zenoss.core
/opt/serviced/bin/serviced-set-version Zenoss.core 6.3.2

Original Message:
Sent: 03-23-2021 03:48 PM
From: Arthur
Subject: Upgrade looks like it worked but

Hi Lic

The erro tells you that the upgrade tried to access an URL and it got a 401 (Unauthorized) Error back. If Iook at the ugrade_core.txt script step 12 tries to start the zenevenserver.
SVC_WAIT Zenoss.core/Infrastructure/solr Zenoss.core/Zenoss/Events/zeneventserver started 600

So what you can do is, try the upgrade again and if it stays at the same line login to the control center and try to restart zeneventserver by hand. The services which have to be up for the upgrade are:

# Start all our dependent services
SVC_START Zenoss.core/Infrastructure/mariadb
SVC_START Zenoss.core/Infrastructure/RabbitMQ
SVC_START Zenoss.core/Zenoss/Events/zeneventserver
SVC_START Zenoss.core/Infrastructure/redis

I've attached the two upgrade scripts which are located in the zope container.

------------------------------
Arthur

Original Message:
Sent: 03-18-2021 01:36 PM
From: License Administrator
Subject: Upgrade looks like it worked but

Well tried it again through putty and it worked better. May have been the fact that I had tried the script again after renaming the original folder. This time I used the original files and got to step 12 with out a error - not sure what to do with this error.

File "/usr/lib64/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: Unauthorized
exit status 1
[91mE0318 14:21:02.605105 03080 runner.go:161] error executing step 12: SVC_EXEC: exit status 1
[0m[94mI0318 14:21:02.605184 03080 runner.go:151] Executing exit functions
[0m[94mI0318 14:21:02.605223 03080 eval.go:51] restoring snapshot a29uwksqyxmb3q6a63y56env8_20210318-181849.210
[0mexit status 1

Script done on Thu 18 Mar 2021 02:24:44 PM EDT

------------------------------
License Administrator
Ithaca CSD

Original Message:
Sent: 03-18-2021 09:50 AM
From: License Administrator
Subject: Upgrade looks like it worked but

web page is getting the error - no public endpoint endpoints available - I attached the upgrade log the only error I see is a time out for line 7

Script started on Thu 18 Mar 2021 09:36:32 AM EDT
[94mI0318 09:36:33.317283 26411 runner.go:159] executing step 0: DESCRIPTION Persist current version number of Zenoss Core
[0m[94mI0318 09:36:33.317387 26411 runner.go:159] executing step 1: REQUIRE_SVC
[0m[94mI0318 09:36:33.317399 26411 eval.go:340] checking service requirement
[0m[94mI0318 09:36:33.317413 26411 eval.go:344] verifying service a29uwksqyxmb3q6a63y56env8
[0m[94mI0318 09:36:33.317737 26411 eval.go:350] found a29uwksqyxmb3q6a63y56env8 tenant id for service a29uwksqyxmb3q6a63y56env8
[0m[94mI0318 09:36:33.317756 26411 runner.go:159] executing step 2: SVC_USE zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:33.317772 26411 eval.go:70] preparing to use image: zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:33.317794 26411 service_client.go:32] Pulling zenoss/core_5.1:5.1.3_1, tagging to latest, and pushing to registry localhost:5000 - this may take a while
[0m[94mI0318 09:36:34.809740 26411 eval.go:79] Successfully pulled and tagged new image zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:34.809773 26411 runner.go:159] executing step 3: SVC_START Zenoss.core/Infrastructure/mariadb
[0m[94mI0318 09:36:35.297059 26411 eval.go:170] starting service Zenoss.core/Infrastructure/mariadb 4095hvjxzyvsazpdt3l79z28h
[0m[94mI0318 09:36:35.383577 26411 runner.go:159] executing step 4: SVC_START Zenoss.core/Infrastructure/RabbitMQ
[0m[94mI0318 09:36:35.929092 26411 eval.go:170] starting service Zenoss.core/Infrastructure/RabbitMQ 7sek1x9h67fm0v3hljnrmyeex
[0m[94mI0318 09:36:36.130112 26411 runner.go:159] executing step 5: SVC_START Zenoss.core/Zenoss/Events/zeneventserver
[0m[94mI0318 09:36:36.624419 26411 eval.go:170] starting service Zenoss.core/Zenoss/Events/zeneventserver 89y96jsm0z6n6l6erw9nkos67
[0m[94mI0318 09:36:37.247918 26411 runner.go:159] executing step 6: SVC_START Zenoss.core/Infrastructure/redis
[0m[94mI0318 09:36:38.502775 26411 eval.go:170] starting service Zenoss.core/Infrastructure/redis abhn75z8hhv872zd1j5whbo5d
[0m[94mI0318 09:36:38.822773 26411 runner.go:159] executing step 7: SVC_WAIT Zenoss.core/Infrastructure/mariadb Zenoss.core/Infrastructure/RabbitMQ Zenoss.core/Zenoss/Events/zeneventserver Zenoss.core/Infrastructure/redis started 1200
[0m[94mI0318 09:36:41.388952 26411 eval.go:134] waiting 1200 for services Zenoss.core/Infrastructure/mariadb, Zenoss.core/Infrastructure/RabbitMQ, Zenoss.core/Zenoss/Events/zeneventserver, Zenoss.core/Infrastructure/redis to be started
[0m
[91mE0318 09:56:41.419410 26411 runner.go:161] error executing step 7: SVC_WAIT: timeout
[0m[94mI0318 09:56:41.419452 26411 runner.go:151] Executing exit functions
[0mtimeout

Script done on Thu 18 Mar 2021 09:56:41 AM EDT

------------------------------
License Administrator
Ithaca CSD
------------------------------

7. RE: Upgrade looks like it worked but

Recommend

Arthur

Posted 13 days ago
Edited by Arthur 13 days ago

Hi Lic

Based on this logfie script-2021-03-26-092205-root.txt the upagrade passed not steps 0 -11. Then zenmigrate starte to add the Solr service which is new in 6.X. After that it failed trying to open a URL during the crtification verification.
You said you updated the certificate. My I ask you why? And did you got the same error with your previous certificate?

------------------------------
Arthur
------------------------------

Original Message

Original Message:
Sent: 03-26-2021 08:39 AM
From: License Administrator
Subject: Upgrade looks like it worked but

Issue one with this script "# This script can be used to upgrade a Zenoss deployment to a new image
#
# To minimize downtime and avoid potential out of space disk issues,
# preseed the master with the images (that the upgrade will automatically
# pull from the internet) before the upgrade maintenance window:
# ssh MASTER
# sudo docker pull zenoss/core_6.3:6.3.2 # about 10-20 minutes
# sudo docker pull zenoss/hbase:24.0.8 # about 5-10 minutes
# sudo docker pull zenoss/opentsdb:24.0.8 # about 7-15 minutes
# serviced docker sync # about 10m:core 5m:hbase 8m:opentsdb"

Script started, file is /var/log/serviced/script-2021-03-26-091436-root.log
could not parse line 60: %UPGRADE_REINDEX%
Script done, file is /var/log/serviced/script-2021-03-26-091436-root.log

and the other one is
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
exit status 1

I updated the certificate following the steps
Control Center (zenoss.com) and
Control Center (zenoss.com)

When I access the web interface the certificate is valid. I included the log files. The 4 services started correctly in the script.

Thank you

------------------------------
License Administrator
Ithaca CSD

Original Message:
Sent: 03-23-2021 03:51 PM
From: Arthur
Subject: Upgrade looks like it worked but

upgrade-core.sh for reference. This script starts the upgrade-core.txt

#!/bin/sh
#
###############################################################################
#
# Upgrade Zenoss Core
#
###############################################################################
set -e
export SERVICE_TENANT_ID="`serviced service list core --format='{{.ID}}'`"
export UPGRADE_REINDEX
export REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS
serviced snapshot untag $SERVICE_TENANT_ID preupgrade-core-6.3.2

FROM_VERSION=$(serviced service list core --format='{{.Version}}')
TO_VERSION=6.3.2

if [ -z "$FROM_VERSION" ];
then
    FROM_VERSION=$(serviced service list core --format '{{.ImageID}}' | awk '{ print "localhost:5000/"$0" cat /opt/zenoss/Products/ZenModel/ZVersion.py" }' | xargs docker run --rm | grep VERSION | cut -d'"' -f2)
fi

if [ -z "$FROM_VERSION" ];
then
    FROM_VERSION=$(grep 'SVC_USE zenoss/resmgr_[^:_]*:.*' /root/6.3.x/current_version.txt | cut -d':' -f2 | cut -d'_' -f1)
    if [ -z "$FROM_VERSION" ];
    then
        echo "Could not get the previous Core version number from serviced, please edit current_version.txt to set it."
        echo "Change this line 'SVC_USE zenoss/core_REPLACE_1' to reflect the pre-upgrade version, for example:"
        echo "SVC_USE zenoss/core_5.2:5.2.6_1"
        exit 1
    else
        echo "Using FROM_VERSION provided by change to current_version.txt"
    fi
fi

FROM_VERSION_SHORT=$(cut -d'.' -f1,2 <<< $FROM_VERSION)
sed -i 's/REPLACE/'"$FROM_VERSION_SHORT:$FROM_VERSION"'/g' /root/6.3.x/current_version.txt
sed -i "s/%FROM_VERSION%/$FROM_VERSION/g" /root/6.3.x/current_version.txt
serviced script run /root/6.3.x/current_version.txt --service Zenoss.core

if [ $(echo "${FROM_VERSION}" | cut -c1) -eq 5 ] && [ $(echo "${TO_VERSION}" | cut -c1) -eq 6 ];
then
    UPGRADE_REINDEX='SVC_EXEC NO_COMMIT "Zenoss.core/Zenoss/User Interface/Zope" su - zenoss -c "/opt/zenoss/bin/upgrade_reindex.sh Zenoss.core"'
    REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS='SVC_EXEC NO_COMMIT "Zenoss.core/Zenoss/User Interface/Zope" /opt/zenoss/bin/zenmigrate --step=RemoveEmptyProductClassRelationships --dont-bump'
fi

sed -i "s@%UPGRADE_REINDEX%@${UPGRADE_REINDEX}@g; s@%REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS%@${REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS}@g;" /root/6.3.x/upgrade-core.txt

serviced script run /root/6.3.x/upgrade-core.txt --service Zenoss.core
/opt/serviced/bin/serviced-set-version Zenoss.core 6.3.2

Original Message:
Sent: 03-23-2021 03:48 PM
From: Arthur
Subject: Upgrade looks like it worked but

Hi Lic

The erro tells you that the upgrade tried to access an URL and it got a 401 (Unauthorized) Error back. If Iook at the ugrade_core.txt script step 12 tries to start the zenevenserver.
SVC_WAIT Zenoss.core/Infrastructure/solr Zenoss.core/Zenoss/Events/zeneventserver started 600

So what you can do is, try the upgrade again and if it stays at the same line login to the control center and try to restart zeneventserver by hand. The services which have to be up for the upgrade are:

# Start all our dependent services
SVC_START Zenoss.core/Infrastructure/mariadb
SVC_START Zenoss.core/Infrastructure/RabbitMQ
SVC_START Zenoss.core/Zenoss/Events/zeneventserver
SVC_START Zenoss.core/Infrastructure/redis

I've attached the two upgrade scripts which are located in the zope container.

------------------------------
Arthur

Original Message:
Sent: 03-18-2021 01:36 PM
From: License Administrator
Subject: Upgrade looks like it worked but

Well tried it again through putty and it worked better. May have been the fact that I had tried the script again after renaming the original folder. This time I used the original files and got to step 12 with out a error - not sure what to do with this error.

File "/usr/lib64/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: Unauthorized
exit status 1
[91mE0318 14:21:02.605105 03080 runner.go:161] error executing step 12: SVC_EXEC: exit status 1
[0m[94mI0318 14:21:02.605184 03080 runner.go:151] Executing exit functions
[0m[94mI0318 14:21:02.605223 03080 eval.go:51] restoring snapshot a29uwksqyxmb3q6a63y56env8_20210318-181849.210
[0mexit status 1

Script done on Thu 18 Mar 2021 02:24:44 PM EDT

------------------------------
License Administrator
Ithaca CSD

Original Message:
Sent: 03-18-2021 09:50 AM
From: License Administrator
Subject: Upgrade looks like it worked but

web page is getting the error - no public endpoint endpoints available - I attached the upgrade log the only error I see is a time out for line 7

Script started on Thu 18 Mar 2021 09:36:32 AM EDT
[94mI0318 09:36:33.317283 26411 runner.go:159] executing step 0: DESCRIPTION Persist current version number of Zenoss Core
[0m[94mI0318 09:36:33.317387 26411 runner.go:159] executing step 1: REQUIRE_SVC
[0m[94mI0318 09:36:33.317399 26411 eval.go:340] checking service requirement
[0m[94mI0318 09:36:33.317413 26411 eval.go:344] verifying service a29uwksqyxmb3q6a63y56env8
[0m[94mI0318 09:36:33.317737 26411 eval.go:350] found a29uwksqyxmb3q6a63y56env8 tenant id for service a29uwksqyxmb3q6a63y56env8
[0m[94mI0318 09:36:33.317756 26411 runner.go:159] executing step 2: SVC_USE zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:33.317772 26411 eval.go:70] preparing to use image: zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:33.317794 26411 service_client.go:32] Pulling zenoss/core_5.1:5.1.3_1, tagging to latest, and pushing to registry localhost:5000 - this may take a while
[0m[94mI0318 09:36:34.809740 26411 eval.go:79] Successfully pulled and tagged new image zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:34.809773 26411 runner.go:159] executing step 3: SVC_START Zenoss.core/Infrastructure/mariadb
[0m[94mI0318 09:36:35.297059 26411 eval.go:170] starting service Zenoss.core/Infrastructure/mariadb 4095hvjxzyvsazpdt3l79z28h
[0m[94mI0318 09:36:35.383577 26411 runner.go:159] executing step 4: SVC_START Zenoss.core/Infrastructure/RabbitMQ
[0m[94mI0318 09:36:35.929092 26411 eval.go:170] starting service Zenoss.core/Infrastructure/RabbitMQ 7sek1x9h67fm0v3hljnrmyeex
[0m[94mI0318 09:36:36.130112 26411 runner.go:159] executing step 5: SVC_START Zenoss.core/Zenoss/Events/zeneventserver
[0m[94mI0318 09:36:36.624419 26411 eval.go:170] starting service Zenoss.core/Zenoss/Events/zeneventserver 89y96jsm0z6n6l6erw9nkos67
[0m[94mI0318 09:36:37.247918 26411 runner.go:159] executing step 6: SVC_START Zenoss.core/Infrastructure/redis
[0m[94mI0318 09:36:38.502775 26411 eval.go:170] starting service Zenoss.core/Infrastructure/redis abhn75z8hhv872zd1j5whbo5d
[0m[94mI0318 09:36:38.822773 26411 runner.go:159] executing step 7: SVC_WAIT Zenoss.core/Infrastructure/mariadb Zenoss.core/Infrastructure/RabbitMQ Zenoss.core/Zenoss/Events/zeneventserver Zenoss.core/Infrastructure/redis started 1200
[0m[94mI0318 09:36:41.388952 26411 eval.go:134] waiting 1200 for services Zenoss.core/Infrastructure/mariadb, Zenoss.core/Infrastructure/RabbitMQ, Zenoss.core/Zenoss/Events/zeneventserver, Zenoss.core/Infrastructure/redis to be started
[0m
[91mE0318 09:56:41.419410 26411 runner.go:161] error executing step 7: SVC_WAIT: timeout
[0m[94mI0318 09:56:41.419452 26411 runner.go:151] Executing exit functions
[0mtimeout

Script done on Thu 18 Mar 2021 09:56:41 AM EDT

------------------------------
License Administrator
Ithaca CSD
------------------------------

8. RE: Upgrade looks like it worked but

Recommend

License

Posted 10 days ago

I updated the certificate because it had expired.

------------------------------
License Administrator
Ithaca CSD
------------------------------

Original Message

Original Message:
Sent: 03-27-2021 07:31 AM
From: Arthur
Subject: Upgrade looks like it worked but

Hi Lic

Based on this logfie script-2021-03-26-092205-root.txt the upagrade passed not steps 0 -11. Then zenmigrate starte to add the Solr service which is new in 6.X. After that it failed trying to open a URL during the crtification verification.
You said you updated the certificate. My I ask you why? And did you got the same error with your previous certificate?

------------------------------
Arthur

Original Message:
Sent: 03-26-2021 08:39 AM
From: License Administrator
Subject: Upgrade looks like it worked but

Issue one with this script "# This script can be used to upgrade a Zenoss deployment to a new image
#
# To minimize downtime and avoid potential out of space disk issues,
# preseed the master with the images (that the upgrade will automatically
# pull from the internet) before the upgrade maintenance window:
# ssh MASTER
# sudo docker pull zenoss/core_6.3:6.3.2 # about 10-20 minutes
# sudo docker pull zenoss/hbase:24.0.8 # about 5-10 minutes
# sudo docker pull zenoss/opentsdb:24.0.8 # about 7-15 minutes
# serviced docker sync # about 10m:core 5m:hbase 8m:opentsdb"

Script started, file is /var/log/serviced/script-2021-03-26-091436-root.log
could not parse line 60: %UPGRADE_REINDEX%
Script done, file is /var/log/serviced/script-2021-03-26-091436-root.log

and the other one is
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
exit status 1

I updated the certificate following the steps
Control Center (zenoss.com) and
Control Center (zenoss.com)

When I access the web interface the certificate is valid. I included the log files. The 4 services started correctly in the script.

Thank you

------------------------------
License Administrator
Ithaca CSD

Original Message:
Sent: 03-23-2021 03:51 PM
From: Arthur
Subject: Upgrade looks like it worked but

upgrade-core.sh for reference. This script starts the upgrade-core.txt

#!/bin/sh
#
###############################################################################
#
# Upgrade Zenoss Core
#
###############################################################################
set -e
export SERVICE_TENANT_ID="`serviced service list core --format='{{.ID}}'`"
export UPGRADE_REINDEX
export REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS
serviced snapshot untag $SERVICE_TENANT_ID preupgrade-core-6.3.2

FROM_VERSION=$(serviced service list core --format='{{.Version}}')
TO_VERSION=6.3.2

if [ -z "$FROM_VERSION" ];
then
    FROM_VERSION=$(serviced service list core --format '{{.ImageID}}' | awk '{ print "localhost:5000/"$0" cat /opt/zenoss/Products/ZenModel/ZVersion.py" }' | xargs docker run --rm | grep VERSION | cut -d'"' -f2)
fi

if [ -z "$FROM_VERSION" ];
then
    FROM_VERSION=$(grep 'SVC_USE zenoss/resmgr_[^:_]*:.*' /root/6.3.x/current_version.txt | cut -d':' -f2 | cut -d'_' -f1)
    if [ -z "$FROM_VERSION" ];
    then
        echo "Could not get the previous Core version number from serviced, please edit current_version.txt to set it."
        echo "Change this line 'SVC_USE zenoss/core_REPLACE_1' to reflect the pre-upgrade version, for example:"
        echo "SVC_USE zenoss/core_5.2:5.2.6_1"
        exit 1
    else
        echo "Using FROM_VERSION provided by change to current_version.txt"
    fi
fi

FROM_VERSION_SHORT=$(cut -d'.' -f1,2 <<< $FROM_VERSION)
sed -i 's/REPLACE/'"$FROM_VERSION_SHORT:$FROM_VERSION"'/g' /root/6.3.x/current_version.txt
sed -i "s/%FROM_VERSION%/$FROM_VERSION/g" /root/6.3.x/current_version.txt
serviced script run /root/6.3.x/current_version.txt --service Zenoss.core

if [ $(echo "${FROM_VERSION}" | cut -c1) -eq 5 ] && [ $(echo "${TO_VERSION}" | cut -c1) -eq 6 ];
then
    UPGRADE_REINDEX='SVC_EXEC NO_COMMIT "Zenoss.core/Zenoss/User Interface/Zope" su - zenoss -c "/opt/zenoss/bin/upgrade_reindex.sh Zenoss.core"'
    REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS='SVC_EXEC NO_COMMIT "Zenoss.core/Zenoss/User Interface/Zope" /opt/zenoss/bin/zenmigrate --step=RemoveEmptyProductClassRelationships --dont-bump'
fi

sed -i "s@%UPGRADE_REINDEX%@${UPGRADE_REINDEX}@g; s@%REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS%@${REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS}@g;" /root/6.3.x/upgrade-core.txt

serviced script run /root/6.3.x/upgrade-core.txt --service Zenoss.core
/opt/serviced/bin/serviced-set-version Zenoss.core 6.3.2

Original Message:
Sent: 03-23-2021 03:48 PM
From: Arthur
Subject: Upgrade looks like it worked but

Hi Lic

The erro tells you that the upgrade tried to access an URL and it got a 401 (Unauthorized) Error back. If Iook at the ugrade_core.txt script step 12 tries to start the zenevenserver.
SVC_WAIT Zenoss.core/Infrastructure/solr Zenoss.core/Zenoss/Events/zeneventserver started 600

So what you can do is, try the upgrade again and if it stays at the same line login to the control center and try to restart zeneventserver by hand. The services which have to be up for the upgrade are:

# Start all our dependent services
SVC_START Zenoss.core/Infrastructure/mariadb
SVC_START Zenoss.core/Infrastructure/RabbitMQ
SVC_START Zenoss.core/Zenoss/Events/zeneventserver
SVC_START Zenoss.core/Infrastructure/redis

I've attached the two upgrade scripts which are located in the zope container.

------------------------------
Arthur

Original Message:
Sent: 03-18-2021 01:36 PM
From: License Administrator
Subject: Upgrade looks like it worked but

Well tried it again through putty and it worked better. May have been the fact that I had tried the script again after renaming the original folder. This time I used the original files and got to step 12 with out a error - not sure what to do with this error.

File "/usr/lib64/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: Unauthorized
exit status 1
[91mE0318 14:21:02.605105 03080 runner.go:161] error executing step 12: SVC_EXEC: exit status 1
[0m[94mI0318 14:21:02.605184 03080 runner.go:151] Executing exit functions
[0m[94mI0318 14:21:02.605223 03080 eval.go:51] restoring snapshot a29uwksqyxmb3q6a63y56env8_20210318-181849.210
[0mexit status 1

Script done on Thu 18 Mar 2021 02:24:44 PM EDT

------------------------------
License Administrator
Ithaca CSD

Original Message:
Sent: 03-18-2021 09:50 AM
From: License Administrator
Subject: Upgrade looks like it worked but

web page is getting the error - no public endpoint endpoints available - I attached the upgrade log the only error I see is a time out for line 7

Script started on Thu 18 Mar 2021 09:36:32 AM EDT
[94mI0318 09:36:33.317283 26411 runner.go:159] executing step 0: DESCRIPTION Persist current version number of Zenoss Core
[0m[94mI0318 09:36:33.317387 26411 runner.go:159] executing step 1: REQUIRE_SVC
[0m[94mI0318 09:36:33.317399 26411 eval.go:340] checking service requirement
[0m[94mI0318 09:36:33.317413 26411 eval.go:344] verifying service a29uwksqyxmb3q6a63y56env8
[0m[94mI0318 09:36:33.317737 26411 eval.go:350] found a29uwksqyxmb3q6a63y56env8 tenant id for service a29uwksqyxmb3q6a63y56env8
[0m[94mI0318 09:36:33.317756 26411 runner.go:159] executing step 2: SVC_USE zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:33.317772 26411 eval.go:70] preparing to use image: zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:33.317794 26411 service_client.go:32] Pulling zenoss/core_5.1:5.1.3_1, tagging to latest, and pushing to registry localhost:5000 - this may take a while
[0m[94mI0318 09:36:34.809740 26411 eval.go:79] Successfully pulled and tagged new image zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:34.809773 26411 runner.go:159] executing step 3: SVC_START Zenoss.core/Infrastructure/mariadb
[0m[94mI0318 09:36:35.297059 26411 eval.go:170] starting service Zenoss.core/Infrastructure/mariadb 4095hvjxzyvsazpdt3l79z28h
[0m[94mI0318 09:36:35.383577 26411 runner.go:159] executing step 4: SVC_START Zenoss.core/Infrastructure/RabbitMQ
[0m[94mI0318 09:36:35.929092 26411 eval.go:170] starting service Zenoss.core/Infrastructure/RabbitMQ 7sek1x9h67fm0v3hljnrmyeex
[0m[94mI0318 09:36:36.130112 26411 runner.go:159] executing step 5: SVC_START Zenoss.core/Zenoss/Events/zeneventserver
[0m[94mI0318 09:36:36.624419 26411 eval.go:170] starting service Zenoss.core/Zenoss/Events/zeneventserver 89y96jsm0z6n6l6erw9nkos67
[0m[94mI0318 09:36:37.247918 26411 runner.go:159] executing step 6: SVC_START Zenoss.core/Infrastructure/redis
[0m[94mI0318 09:36:38.502775 26411 eval.go:170] starting service Zenoss.core/Infrastructure/redis abhn75z8hhv872zd1j5whbo5d
[0m[94mI0318 09:36:38.822773 26411 runner.go:159] executing step 7: SVC_WAIT Zenoss.core/Infrastructure/mariadb Zenoss.core/Infrastructure/RabbitMQ Zenoss.core/Zenoss/Events/zeneventserver Zenoss.core/Infrastructure/redis started 1200
[0m[94mI0318 09:36:41.388952 26411 eval.go:134] waiting 1200 for services Zenoss.core/Infrastructure/mariadb, Zenoss.core/Infrastructure/RabbitMQ, Zenoss.core/Zenoss/Events/zeneventserver, Zenoss.core/Infrastructure/redis to be started
[0m
[91mE0318 09:56:41.419410 26411 runner.go:161] error executing step 7: SVC_WAIT: timeout
[0m[94mI0318 09:56:41.419452 26411 runner.go:151] Executing exit functions
[0mtimeout

Script done on Thu 18 Mar 2021 09:56:41 AM EDT

------------------------------
License Administrator
Ithaca CSD
------------------------------

9. RE: Upgrade looks like it worked but

Recommend

Arthur

Posted 6 days ago

Hi Lic

That's fair enough. If I understand it right you are still using a self signed cert and your brwoser will probably complain it. The only advantage you have is that cert does not show as expired and the communication is secured based on your cert/key pair. Currently my knowlege about the migrate code is very little and I can't tell you where your problem is.

My advices is to try the update with Zenoss internal cert and see if you fail at the same point.

It also not worth to check your cert again and compair your Zeoss endpoints with the one listed in the certificate because if you followed the procedure to create the certificate your endpoint names (Subject Alternative Name:) should look like:

(In case your CC hostname is master.X.Y)

X509v3 Subject Alternative Name:
DNS:master.X.Y, DNS:*.master.X.Y

So for the Solr endpoint it would be solr.master.X.Y and it should be resolvable.

------------------------------
Arthur
------------------------------

Original Message

Original Message:
Sent: 03-31-2021 06:52 AM
From: License Administrator
Subject: Upgrade looks like it worked but

I updated the certificate because it had expired.

------------------------------
License Administrator
Ithaca CSD

Original Message:
Sent: 03-27-2021 07:31 AM
From: Arthur
Subject: Upgrade looks like it worked but

Hi Lic

Based on this logfie script-2021-03-26-092205-root.txt the upagrade passed not steps 0 -11. Then zenmigrate starte to add the Solr service which is new in 6.X. After that it failed trying to open a URL during the crtification verification.
You said you updated the certificate. My I ask you why? And did you got the same error with your previous certificate?

------------------------------
Arthur

Original Message:
Sent: 03-26-2021 08:39 AM
From: License Administrator
Subject: Upgrade looks like it worked but

Issue one with this script "# This script can be used to upgrade a Zenoss deployment to a new image
#
# To minimize downtime and avoid potential out of space disk issues,
# preseed the master with the images (that the upgrade will automatically
# pull from the internet) before the upgrade maintenance window:
# ssh MASTER
# sudo docker pull zenoss/core_6.3:6.3.2 # about 10-20 minutes
# sudo docker pull zenoss/hbase:24.0.8 # about 5-10 minutes
# sudo docker pull zenoss/opentsdb:24.0.8 # about 7-15 minutes
# serviced docker sync # about 10m:core 5m:hbase 8m:opentsdb"

Script started, file is /var/log/serviced/script-2021-03-26-091436-root.log
could not parse line 60: %UPGRADE_REINDEX%
Script done, file is /var/log/serviced/script-2021-03-26-091436-root.log

and the other one is
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
exit status 1

I updated the certificate following the steps
Control Center (zenoss.com) and
Control Center (zenoss.com)

When I access the web interface the certificate is valid. I included the log files. The 4 services started correctly in the script.

Thank you

------------------------------
License Administrator
Ithaca CSD

Original Message:
Sent: 03-23-2021 03:51 PM
From: Arthur
Subject: Upgrade looks like it worked but

upgrade-core.sh for reference. This script starts the upgrade-core.txt

#!/bin/sh
#
###############################################################################
#
# Upgrade Zenoss Core
#
###############################################################################
set -e
export SERVICE_TENANT_ID="`serviced service list core --format='{{.ID}}'`"
export UPGRADE_REINDEX
export REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS
serviced snapshot untag $SERVICE_TENANT_ID preupgrade-core-6.3.2

FROM_VERSION=$(serviced service list core --format='{{.Version}}')
TO_VERSION=6.3.2

if [ -z "$FROM_VERSION" ];
then
    FROM_VERSION=$(serviced service list core --format '{{.ImageID}}' | awk '{ print "localhost:5000/"$0" cat /opt/zenoss/Products/ZenModel/ZVersion.py" }' | xargs docker run --rm | grep VERSION | cut -d'"' -f2)
fi

if [ -z "$FROM_VERSION" ];
then
    FROM_VERSION=$(grep 'SVC_USE zenoss/resmgr_[^:_]*:.*' /root/6.3.x/current_version.txt | cut -d':' -f2 | cut -d'_' -f1)
    if [ -z "$FROM_VERSION" ];
    then
        echo "Could not get the previous Core version number from serviced, please edit current_version.txt to set it."
        echo "Change this line 'SVC_USE zenoss/core_REPLACE_1' to reflect the pre-upgrade version, for example:"
        echo "SVC_USE zenoss/core_5.2:5.2.6_1"
        exit 1
    else
        echo "Using FROM_VERSION provided by change to current_version.txt"
    fi
fi

FROM_VERSION_SHORT=$(cut -d'.' -f1,2 <<< $FROM_VERSION)
sed -i 's/REPLACE/'"$FROM_VERSION_SHORT:$FROM_VERSION"'/g' /root/6.3.x/current_version.txt
sed -i "s/%FROM_VERSION%/$FROM_VERSION/g" /root/6.3.x/current_version.txt
serviced script run /root/6.3.x/current_version.txt --service Zenoss.core

if [ $(echo "${FROM_VERSION}" | cut -c1) -eq 5 ] && [ $(echo "${TO_VERSION}" | cut -c1) -eq 6 ];
then
    UPGRADE_REINDEX='SVC_EXEC NO_COMMIT "Zenoss.core/Zenoss/User Interface/Zope" su - zenoss -c "/opt/zenoss/bin/upgrade_reindex.sh Zenoss.core"'
    REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS='SVC_EXEC NO_COMMIT "Zenoss.core/Zenoss/User Interface/Zope" /opt/zenoss/bin/zenmigrate --step=RemoveEmptyProductClassRelationships --dont-bump'
fi

sed -i "s@%UPGRADE_REINDEX%@${UPGRADE_REINDEX}@g; s@%REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS%@${REMOVE_EMPTY_PRODUCT_CLASS_RELATIONSHIPS}@g;" /root/6.3.x/upgrade-core.txt

serviced script run /root/6.3.x/upgrade-core.txt --service Zenoss.core
/opt/serviced/bin/serviced-set-version Zenoss.core 6.3.2

Original Message:
Sent: 03-23-2021 03:48 PM
From: Arthur
Subject: Upgrade looks like it worked but

Hi Lic

The erro tells you that the upgrade tried to access an URL and it got a 401 (Unauthorized) Error back. If Iook at the ugrade_core.txt script step 12 tries to start the zenevenserver.
SVC_WAIT Zenoss.core/Infrastructure/solr Zenoss.core/Zenoss/Events/zeneventserver started 600

So what you can do is, try the upgrade again and if it stays at the same line login to the control center and try to restart zeneventserver by hand. The services which have to be up for the upgrade are:

# Start all our dependent services
SVC_START Zenoss.core/Infrastructure/mariadb
SVC_START Zenoss.core/Infrastructure/RabbitMQ
SVC_START Zenoss.core/Zenoss/Events/zeneventserver
SVC_START Zenoss.core/Infrastructure/redis

I've attached the two upgrade scripts which are located in the zope container.

------------------------------
Arthur

Original Message:
Sent: 03-18-2021 01:36 PM
From: License Administrator
Subject: Upgrade looks like it worked but

Well tried it again through putty and it worked better. May have been the fact that I had tried the script again after renaming the original folder. This time I used the original files and got to step 12 with out a error - not sure what to do with this error.

File "/usr/lib64/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 401: Unauthorized
exit status 1
[91mE0318 14:21:02.605105 03080 runner.go:161] error executing step 12: SVC_EXEC: exit status 1
[0m[94mI0318 14:21:02.605184 03080 runner.go:151] Executing exit functions
[0m[94mI0318 14:21:02.605223 03080 eval.go:51] restoring snapshot a29uwksqyxmb3q6a63y56env8_20210318-181849.210
[0mexit status 1

Script done on Thu 18 Mar 2021 02:24:44 PM EDT

------------------------------
License Administrator
Ithaca CSD

Original Message:
Sent: 03-18-2021 09:50 AM
From: License Administrator
Subject: Upgrade looks like it worked but

web page is getting the error - no public endpoint endpoints available - I attached the upgrade log the only error I see is a time out for line 7

Script started on Thu 18 Mar 2021 09:36:32 AM EDT
[94mI0318 09:36:33.317283 26411 runner.go:159] executing step 0: DESCRIPTION Persist current version number of Zenoss Core
[0m[94mI0318 09:36:33.317387 26411 runner.go:159] executing step 1: REQUIRE_SVC
[0m[94mI0318 09:36:33.317399 26411 eval.go:340] checking service requirement
[0m[94mI0318 09:36:33.317413 26411 eval.go:344] verifying service a29uwksqyxmb3q6a63y56env8
[0m[94mI0318 09:36:33.317737 26411 eval.go:350] found a29uwksqyxmb3q6a63y56env8 tenant id for service a29uwksqyxmb3q6a63y56env8
[0m[94mI0318 09:36:33.317756 26411 runner.go:159] executing step 2: SVC_USE zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:33.317772 26411 eval.go:70] preparing to use image: zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:33.317794 26411 service_client.go:32] Pulling zenoss/core_5.1:5.1.3_1, tagging to latest, and pushing to registry localhost:5000 - this may take a while
[0m[94mI0318 09:36:34.809740 26411 eval.go:79] Successfully pulled and tagged new image zenoss/core_5.1:5.1.3_1
[0m[94mI0318 09:36:34.809773 26411 runner.go:159] executing step 3: SVC_START Zenoss.core/Infrastructure/mariadb
[0m[94mI0318 09:36:35.297059 26411 eval.go:170] starting service Zenoss.core/Infrastructure/mariadb 4095hvjxzyvsazpdt3l79z28h
[0m[94mI0318 09:36:35.383577 26411 runner.go:159] executing step 4: SVC_START Zenoss.core/Infrastructure/RabbitMQ
[0m[94mI0318 09:36:35.929092 26411 eval.go:170] starting service Zenoss.core/Infrastructure/RabbitMQ 7sek1x9h67fm0v3hljnrmyeex
[0m[94mI0318 09:36:36.130112 26411 runner.go:159] executing step 5: SVC_START Zenoss.core/Zenoss/Events/zeneventserver
[0m[94mI0318 09:36:36.624419 26411 eval.go:170] starting service Zenoss.core/Zenoss/Events/zeneventserver 89y96jsm0z6n6l6erw9nkos67
[0m[94mI0318 09:36:37.247918 26411 runner.go:159] executing step 6: SVC_START Zenoss.core/Infrastructure/redis
[0m[94mI0318 09:36:38.502775 26411 eval.go:170] starting service Zenoss.core/Infrastructure/redis abhn75z8hhv872zd1j5whbo5d
[0m[94mI0318 09:36:38.822773 26411 runner.go:159] executing step 7: SVC_WAIT Zenoss.core/Infrastructure/mariadb Zenoss.core/Infrastructure/RabbitMQ Zenoss.core/Zenoss/Events/zeneventserver Zenoss.core/Infrastructure/redis started 1200
[0m[94mI0318 09:36:41.388952 26411 eval.go:134] waiting 1200 for services Zenoss.core/Infrastructure/mariadb, Zenoss.core/Infrastructure/RabbitMQ, Zenoss.core/Zenoss/Events/zeneventserver, Zenoss.core/Infrastructure/redis to be started
[0m
[91mE0318 09:56:41.419410 26411 runner.go:161] error executing step 7: SVC_WAIT: timeout
[0m[94mI0318 09:56:41.419452 26411 runner.go:151] Executing exit functions
[0mtimeout

Script done on Thu 18 Mar 2021 09:56:41 AM EDT

------------------------------
License Administrator
Ithaca CSD
------------------------------

InstallationGetting Started

License22 days ago

License22 days ago

License22 days ago

Arthur17 days ago

Arthur17 days ago

License14 days ago

Arthur13 days ago

License10 days ago

Arthur6 days ago

1. Upgrade looks like it worked but

2. RE: Upgrade looks like it worked but

3. RE: Upgrade looks like it worked but

4. RE: Upgrade looks like it worked but

5. RE: Upgrade looks like it worked but

6. RE: Upgrade looks like it worked but

7. RE: Upgrade looks like it worked but

8. RE: Upgrade looks like it worked but

9. RE: Upgrade looks like it worked but