ZenPacks, JSON API, and Integrations

Expand all | Collapse all

serviced snapshot commit

  • 1.  serviced snapshot commit

    Posted 03-06-2019 04:05 PM
    Hello,
    I am installing the toolbox,
    when I try a commit, here's the message back !

    API error (500): {"message":"max depth exceeded"}

    I have a multiple node configuration:
    1 control center
    2 resources pool

    Regard




    ------------------------------
    JOANA SIANARD
    SYSADMIN
    GLOBECAST
    PARIS FRANCE
    Zenoss 5.3.3 r419
    Control Center ControlCenter 1.4.2
    ------------------------------


  • 2.  RE: serviced snapshot commit

    Posted 03-07-2019 01:19 PM
    Hello
    Did you follow the steps described in this article (Permanent or Semi-Permanent)?
    https://support.zenoss.com/hc/en-us/articles/207610516-How-to-add-tools-or-scripts-into-a-Resource-Manager-5-x-Docker-Container

    ------------------------------
    Arthur
    ------------------------------



  • 3.  RE: serviced snapshot commit

    Posted 03-07-2019 04:27 PM
    hello
    I have the same problem when I install a zenpack or I restore !

    INFO:zen.ZenPackCmd:Successfully restored zenpacks!
    FATA[0339] Unable to commit container containerid=b8236d11c88d6ec302d0c9b2881f053a0b0fccdf2f05a2c34232f2eafea9cb80 containername=4jpVZOxxuVDkEB9egl44Dw error=API error (500): {"message":"max depth exceeded"}
    location=shell.go:264 logger=cli.api

    ------------------------------
    JOANA SIANARD
    SYSADMIN
    GLOBECAST
    PARIS FRANCE
    Zenoss 5.3.3 r419
    Control Center ControlCenter 1.4.2
    ------------------------------



  • 4.  RE: serviced snapshot commit

    Posted 03-08-2019 03:35 PM
    Can you post the output of the following commands:

    # docker info && serviced volume status
    # img=$(docker images |grep core |grep latest | awk {'print $1'}) && docker history $img

    ​​​​

    ------------------------------
    Arthur
    ------------------------------



  • 5.  RE: serviced snapshot commit

    Posted 03-08-2019 05:05 PM
    Hello Arthur,

    the files.

    Regards,
    JOANA

    ------------------------------
    JOANA SIANARD
    GLOBECAST
    ------------------------------

    Attachment(s)

    txt
    docker-history.txt   13 KB 1 version
    txt
    docker-info.txt   1 KB 1 version
    txt
    volume-status.txt   647 B 1 version


  • 6.  RE: serviced snapshot commit

    Posted 03-09-2019 02:01 AM
    Hello Joana

    It looks like your image has too many layers (126). As I haven't seen this message so far this is onl my assumption and I can't give you a solution yet. You can google for docker AND "message":"max depth exceeded" for example to find some help how to solve it.

    Anyhow I would be interested about your solution/findings. If I have some free time I'll try to reproduce it, but I can't give you any date so far.

    ------------------------------
    Arthur
    ------------------------------



  • 7.  RE: serviced snapshot commit

    Posted 03-13-2019 01:35 PM

    This happens when your docker container has a lot of layers.  Have you added a lot of things to this before?  (Zenpacks etc).  I would suggest flattening it if so.  Note that you may lose the ability to roll back.  

    https://tuhrig.de/flatten-a-docker-container-or-image/



    ------------------------------
    Andrew Kirch
    Senior Solutions Engineer
    GoVanguard
    ------------------------------



  • 8.  RE: serviced snapshot commit

    Posted 26 days ago
    Has anyone found a workable solution to this issue?  I'm running into the same problem on Zenoss Core 6.3.2.

    ------------------------------
    Duane Wylie
    ------------------------------



  • 9.  RE: serviced snapshot commit

    Posted 26 days ago
    Duane,

    The following steps should take care of things.

    0) Note - There is a restart of the Zenoss application in this process under step 10

    1) First we need to find out what our Zenoss Core image is by running:

    [root@core ~]# ZCORE="$(docker images | grep core_ | awk '{print $1}' | head -1)"
    [root@core ~]# echo $ZCORE
    localhost:5000/5ipo4ged9ffy4gze3zqkq2tdh/core_6.1

    2) Check current image layer count:

    [root@core ~]# docker history $ZCORE | wc -l
    37

    3) We need the sha256 hash of the latest layer from the docker history once we have that we run:

    [root@core ~]# IMAGE_ID="$(docker history --no-trunc -q $ZCORE | head -1)"
    [root@core ~]# echo $IMAGE_ID
    sha256:bcc1c00a7c9fd08c19566fd7e1ccbffe61477801279aac8b0356076716005834

    4) Create a backup of the current image:

    [root@core ~]# docker tag $ZCORE:latest $ZCORE:latest-backup
    [root@core ~]# docker images | grep backup
    localhost:5000/5ipo4ged9ffy4gze3zqkq2tdh/core_6.1 latest-backup bcc1c00a7c9f 8 weeks ago 2.22GB

    5) Create a new container based on the latest layer:

    [root@core ~]# docker run --name "corecontainer" $IMAGE_ID /bin/bash
    [root@core ~]# docker ps -a
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    09efecb488be bcc1c00a7c9f "/bin/bash" 7 seconds ago Exited (0) 5 seconds ago corecontainer

    * If serviced is running you will also see other containers including each isvcs

    6) Use this manually started container to flatten the current image by performing an export/import (this will take a few minutes):

    [root@core ~]# docker export corecontainer | docker import - flat-corecontainer
    sha256:d99d9c93c6e65bfe905079382bbc9f081b58efaf420b678b1a93c9ebbdfb5874

    * The sha256 line will show when complete

    7) Override the current image with the new, flattened image:

    [root@core ~]# serviced docker override $ZCORE:latest flat-corecontainer:latest
    [root@core ~]# docker images | grep latest | head -n3
    flat-corecontainer latest d99d9c93c6e6 2 minutes ago 1.96GB
    localhost:5000/5ipo4ged9ffy4gze3zqkq2tdh/core_6.1 latest d99d9c93c6e6 2 minutes ago 1.96GB
    localhost:5000/5ipo4ged9ffy4gze3zqkq2tdh/core_6.1 latest-backup bcc1c00a7c9f 8 weeks ago 2.22GB

    8) Verify your image has been flattened:

    [root@core ~]# docker history $ZCORE | wc -l
    2

    9) Asynchronously push all images from the serviced Docker registry index into the Docker registry (may take a minute to complete):

    [root@core ~]# serviced docker sync

    * Command produces no output

    10) Restart Zenoss Core to pull the latest image for each container:

    serviced service restart Zenoss.core

    * Restarting immediately isn't necessarily required, but plan to perform a restart before performing any further image edits.

    If the process fails, your original container will still be backed up with the latest-backup tag from Step 4.

    Let us know if this helps?

    ------------------------------
    Michael J. Rogers
    Senior Instructor - Zenoss
    Austin TX
    ------------------------------



  • 10.  RE: serviced snapshot commit

    Posted 25 days ago
    Thanks, Michael.

    I've run through this process without issue through step 9.  (I'll restart a bit later).

    On the off chance that this detonates in my face, I imagine the rollback would look something like this?

    [root@core ~]# serviced docker override $ZCORE:latest $ZCORE:latest-backup
    [root@core ~]# serviced docker sync
    serviced service restart Zenoss.core


    ------------------------------
    Duane Wylie
    ------------------------------



  • 11.  RE: serviced snapshot commit

    Posted 25 days ago
    Duane,

    I should have included the instructions for rollback in my last post, but that looks perfect.

    ------------------------------
    Michael J. Rogers
    Senior Instructor - Zenoss
    Austin TX
    ------------------------------



  • 12.  RE: serviced snapshot commit

    Posted 25 days ago
    After restarting with the flattened image, stats from zenperfsnmp weren't being graphed.  (but stats from zenprocess were.)

    I tried reverting to the $ZCORE:latest-backup, but this didn't resolve my issue.

    I'm restoring the system from backup (hopefully I haven't completely lost the system).

    I was seeing a lot of errors like this in various container logs:

    error="Identity token signature cannot be verified"

    ------------------------------
    Duane Wylie
    ------------------------------