Configuration & Administration

Expand all | Collapse all

howto extend size in /dev/mapper/docker- ?

  • 1.  howto extend size in /dev/mapper/docker- ?

    Posted 12-21-2018 05:08 AM
    Hello
    typical, main zenoss system breaks down 3h before christmas hollidays

    i tried a number of different commands but with no luck
    serviced-storage resize -d /opt/serviced/var/volumes \ -o dm.thinpooldev=docker-253:0-44154960-pool c2zb7s2kdi6onzyaan9qg5vdr 50G

    this is how df looks

    Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/mapper/centos_core--master-root 470785024 124272500 346512524 27% /
    devtmpfs 20021600 0 20021600 0% /dev
    tmpfs 20033516 0 20033516 0% /dev/shm
    tmpfs 20033516 1096 20032420 1% /run
    tmpfs 20033516 0 20033516 0% /sys/fs/cgroup
    /dev/sda1 1038336 278408 759928 27% /boot
    /dev/mapper/centos_core--master-home 450462820 32992 450429828 1% /home
    tmpfs 4006704 0 4006704 0% /run/user/0
    /dev/mapper/docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 103080888 92144932 5676692 95% /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr
    overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/ea5dfcbf97149f48d8b4ce6718d78226c80aa448368470a29dde1e6d44be1f58/merged
    overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/f825f5dacd5f4e3d02d3d5d59c855737d8bd1fe15e4115c882fbac197f8454d9/merged
    overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/636a889a6761257bfb6299f852295bb7075b5baf202b440595eb0a13e69b172c/merged
    overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/fcacda6324191eaab132ef475f053f2ba5e74347c2054d311cb0478d50b818e2/merged
    overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/597863313de7fcc96f55d3d2351fd9c44b9c91208094a95d1ea6193a5f75fd14/merged
    shm 65536 0 65536 0% /var/lib/docker/containers/1aa99077701a25784cdf75d160b6fae93889a89650c3c1e7410ba7cafb8bbd4a/shm
    shm 65536 0 65536 0% /var/lib/docker/containers/1223c1084f59cbd67926dd453cbc35bf61629768e4b5873fd4d1794cac5262fe/shm
    shm 65536 0 65536 0% /var/lib/docker/containers/683700f17d8886319fcdd5efec270fa39a2f919e8efd40a9b36ccbc37d03e070/shm
    shm 65536 0 65536 0% /var/lib/docker/containers/43d616ac5e7e6e49fdff592ee15da41c3fc78c047da7caae1adf27098a2c25d5/shm
    shm 65536 0 65536 0% /var/lib/docker/containers/3e9e3a57aaceb2a9bb08a04685ccb6e89c3b7ccb466d80d45812018a78850f95/shm
    overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/192058a9c7b123a854bf3e2e117c451e36e19404aca4e6a563e6b72a9b44737b/merged
    overlay 470785024 124272500 346512524 27% /var/lib/docker/overlay/ecc5179fc308a221d633946a970519590f6833e3cfe3b89854bdddcacf7cb353/merged
    shm 65536 0 65536 0% /var/lib/docker/containers/c58549d867a35e6362f306041dbee31a52ac6f67dc9b472cfa25c0e69645f443/shm
    shm 65536 0 65536 0% /var/lib/docker/containers/df9284cd9f0e965d31fe50fdc6aa9e5ca2ae8048dc5780bc72fcb271d610664d/shm



    can someone help me with what commands to run in order to get more space in the 95% section.

    regards /F

    ------------------------------
    Fredrik

    ------------------------------


  • 2.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 12-22-2018 09:05 AM
    Hi Fredrik

    Maybe you need to first add additional storage to the volume group.

    https://www.zenoss.com/sites/default/files/zenoss-doc/6386/feature/manage/resize.html

    serviced volume status and lsblk -a output would also be of interest.

    Cheers

    ------------------------------
    Arthur
    ------------------------------



  • 3.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 12-26-2018 11:25 AM

    /dev/mapper/docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 103080888 92144932 5676692 95% /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

    So this means you ran out of storage space on your serviced volume.

    So first off, it's possible that space is not actually being used by files and is just being taken up by old data which hasn't been cleared from the volume yet.  The first thing you should try doing as root is this...

    fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

    See if that frees up a significant amount of space on the volume.  If it doesn't then move on to the following steps to extend the volume size (make sure you add more storage space to the underlying storage device beforehand)...

    The following is an example of the steps required to extend a serviced volume...

    First, stop serviced on all of your hosts including the master host.  Then do...

    vgdisplay # to locate the physical device associated with that volume group, in the case of this example /dev/sdf.  Replace that with the appropriate device for your instance.

    # Also, below where it says +200G, replace that with the amount of storage that you plan to add to it once the additional storage is available on the underlying device.

    pvresize /dev/sdf
    vgextend serviced /dev/sdf
    lvextend /dev/serviced/serviced-pool -L +200G
    fstrim /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr

    Update the following in /etc/default/serviced on the master with the new total volume size before starting serviced back up.

    SERVICED_DM_BASESIZE=300G

    Hope this helps.  Cheers.



    ------------------------------
    Ryan Matte
    ------------------------------



  • 4.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 12-26-2018 12:29 PM

    Some useful commands while working on this...

    lsblk # to list all the block devices
    lvs # to list the actual volume usage (both data and metadata)

    ​​

    ------------------------------
    Ryan Matte
    ------------------------------



  • 5.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 01-02-2019 02:24 AM
    Edited by Fredrik 25 days ago
    Hi and thank you for your response..however im cinda stuck.

    vgdisplay
    gives me

    [root@zenoss ~]# vgdisplay
    File descriptor 3 (/root/appliance/adminmenu.options) leaked on vgdisplay invocation. Parent PID 5646: bash
    --- Volume group ---
    VG Name centos_core-master
    System ID
    Format lvm2
    Metadata Areas 3
    Metadata Sequence No 11
    VG Access read/write
    VG Status resizable
    MAX LV 0
    Cur LV 3
    Open LV 3
    Max PV 0
    Cur PV 3
    Act PV 3
    VG Size <898.99 GiB
    PE Size 4.00 MiB
    Total PE 230141
    Alloc PE / Size 229886 / 897.99 GiB
    Free PE / Size 255 / 1020.00 MiB
    VG UUID KE3VkC-nSXJ-DHGV-Hgiz-tnp7-eIQu-1vv11T

    does that meen it´s "centos_core-master" i should use in my case?


    pvresize is that on my newly added disk (sdd)
    pvresize /dev/sdd
    File descriptor 3 (/root/appliance/adminmenu.options) leaked on pvresize invocation. Parent PID 5646: bash
    Failed to find physical volume "/dev/sdd".
    0 physical volume(s) resized or updated / 0 physical volume(s) not resized



    i dont have /dev/serviced what should i use?

    [root@zenoss ~]# ls /dev/
    autofs centos_core-master crash fb0 hugepages loop-control network_latency ppp rtc0 sdc sg2 sr0 tty1 tty15 tty20 tty26 tty31 tty37 tty42 tty48 tty53 tty59 tty7 ttyS3 vcs vcs6 vcsa5 zero
    block char disk fd hwrng mapper network_throughput ptmx sda sdc3 sg3 stderr tty10 tty16 tty21 tty27 tty32 tty38 tty43 tty49 tty54 tty6 tty8 uhid vcs1 vcsa vcsa6
    bsg console dm-0 full initctl mcelog null pts sda1 sdd sg4 stdin tty11 tty17 tty22 tty28 tty33 tty39 tty44 tty5 tty55 tty60 tty9 uinput vcs2 vcsa1 vfio
    btrfs-control core dm-1 fuse input mem nvram random sda2 sdd1 shm stdout tty12 tty18 tty23 tty29 tty34 tty4 tty45 tty50 tty56 tty61 ttyS0 urandom vcs3 vcsa2 vga_arbiter
    bus cpu dm-2 hidraw0 kmsg mqueue oldmem raw sdb sg0 snapshot tty tty13 tty19 tty24 tty3 tty35 tty40 tty46 tty51 tty57 tty62 ttyS1 usbmon0 vcs4 vcsa3 vhci
    cdrom cpu_dma_latency dri hpet log net port rtc sdb1 sg1 snd tty0 tty14 tty2 tty25 tty30 tty36 tty41 tty47 tty52 tty58 tty63 ttyS2 usbmon1 vcs5 vcsa4 vhost-net


    much regards /F

    ------------------------------
    Fredrik
    ------------------------------



  • 6.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 01-03-2019 09:36 AM

    Sorry, use pvdisplay instead or lsblk to determine the device for the serviced pool.  The pvdisplay output should look something like...

    [root@zenoss5 rmatte]# pvdisplay
    --- Physical volume ---
    PV Name /dev/sda1
    VG Name docker
    PV Size 90.00 GiB / not usable 4.00 MiB
    Allocatable yes
    PE Size 4.00 MiB
    Total PE 23039
    Free PE 2304
    Allocated PE 20735
    PV UUID C7LCve-mYBU-VYlr-jnLv-NEw9-Jq6Y-qaLIJS

    --- Physical volume ---
    PV Name /dev/sda2
    VG Name serviced
    PV Size 100.00 GiB / not usable 3.00 MiB
    Allocatable yes
    PE Size 4.00 MiB
    Total PE 25599
    Free PE 2335
    Allocated PE 23264
    PV UUID 6PAzCU-gwgd-bDjD-uAja-6py1-bAAi-dWmbru

    In the example output above the serviced device would be /dev/sda2.  That's the device you would reference when completing the other steps.  Run pvdisplay on your instance to locate the correct device then proceed with the rest of the steps.  If your serviced host is a vm I'd recommend cloning it beforehand in case something goes wrong (though the steps should work fine, I've used them many times).



    ------------------------------
    Ryan Matte
    ------------------------------



  • 7.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 25 days ago
    Edited by Fredrik 25 days ago
    Hello
    there seems to be something different with my installation, they are all named "VG Name               centos_core-master"

    which one should i extend?, centos_core-master wouldent be right i guess..or?
    it´s hdd1 that´s the new 200g device

    pvdisplay
    File descriptor 3 (/root/appliance/adminmenu.options) leaked on pvdisplay invocation. Parent PID 19959: bash
      --- Physical volume ---
      PV Name               /dev/sda2
      VG Name               centos_core-master
      PV Size               <499.00 GiB / not usable 3.00 MiB
      Allocatable           yes (but full)
      PE Size               4.00 MiB
      Total PE              127743
      Free PE               0
      Allocated PE          127743
      PV UUID               GxdaGE-PgLF-szht-Q2ha-VsJp-hG7i-s64db2

      --- Physical volume ---
      PV Name               /dev/sdb1
      VG Name               centos_core-master
      PV Size               <100.00 GiB / not usable 2.00 MiB
      Allocatable           yes (but full)
      PE Size               4.00 MiB
      Total PE              25599
      Free PE               0
      Allocated PE          25599
      PV UUID               9SPUZH-1gFZ-58ff-UGjd-qlf9-G0dW-SuV94a

      --- Physical volume ---
      PV Name               /dev/sdc3
      VG Name               centos_core-master
      PV Size               <300.00 GiB / not usable 2.00 MiB
      Allocatable           yes
      PE Size               4.00 MiB
      Total PE              76799
      Free PE               255
      Allocated PE          76544
      PV UUID               EPZB4v-Ijrg-PeqG-Nmpl-7ZCd-byck-qfIn3X​

    regards /F

    ------------------------------
    Fredrik
    ------------------------------



  • 8.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 25 days ago
    Normally I would expect those volumes to be labeled according to what they are for, not sure why they are all labeled the same in this case.  They have different sizes though so perhaps you can use that as a clue as to which one you need to extend?

    ------------------------------
    Ryan Matte
    ------------------------------



  • 9.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 24 days ago
    bare with me ;)..
    It is this one i want to extend

      --- Physical volume ---
      PV Name               /dev/sdb1
      VG Name               centos_core-master
      PV Size               <100.00 GiB / not usable 2.00 MiB
      Allocatable           yes (but full)
      PE Size               4.00 MiB
      Total PE              25599
      Free PE               0
      Allocated PE          25599
      PV UUID               9SPUZH-1gFZ-58ff-UGjd-qlf9-G0dW-SuV94a

    but i cant use "centos_core-master" then i´ll just add it go root partition, how do i add it to *docker partition?



    lvextend /dev/centos_core-master/ -L +200G
    /dev/centos_core-master/home  /dev/centos_core-master/root  /dev/centos_core-master/swap
    [root@zenoss5 ~]# lvextend /dev/centos_core-master/ -L +200G

    ------------------------------
    Fredrik
    Umea
    ------------------------------



  • 10.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 19 days ago
    /dev/sdb1 is the device name of that device.  Look at the PV Name.

    ------------------------------
    Ryan Matte
    ------------------------------



  • 11.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 18 days ago
    Edited by Arthur 18 days ago
    Hi Fredrik / Ryan

    As Zenoss V5/6 normally uses the docker devicemapper storage driver, I don't think we can extend the fs by only extend the logical volume.
    Based on your pvdisplay output you have 3 physical volumes in your vg "centos_core-master". sda2, sdb1, sdc3.
    sda2 and sdb1 have zero free PEs. sdc3 has 255 free PE's. Wth a PE size of 4MB  you  have around 1GB free space which is not a lot.

    If you still have the need to extend your serviced volume I need to know a bit more about your configuration.

    Please post the following outputs:

    # lsblk -a
    # serviced volume status
    # docker info
    # grep FS /etc/default/serviced
    # grep DM_ /etc/default/serviced
    # vgs && lvs

    Cheers

    ​​​​​

    ------------------------------
    Arthur
    ------------------------------



  • 12.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 18 days ago
    Hello Arthur and thankyou for your response

    here is the out of the commands you requesten

    [root@zenoss5 ~]# lsblk -a
    NAME                                             MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    sda                                                8:0    0   500G  0 disk
    â"œâ"€sda1                                             8:1    0     1G  0 part /boot
    â""â"€sda2                                             8:2    0   499G  0 part
      â"œâ"€centos_core--master-root                     253:0    0   449G  0 lvm  /
      â"œâ"€centos_core--master-swap                     253:1    0  19.2G  0 lvm  [SWAP]
      â""â"€centos_core--master-home                     253:2    0 429.8G  0 lvm  /home
    sdb                                                8:16   0   100G  0 disk
    â""â"€sdb1                                             8:17   0   100G  0 part
      â""â"€centos_core--master-root                     253:0    0   449G  0 lvm  /
    sdc                                                8:32   0   300G  0 disk
    â""â"€sdc3                                             8:35   0   300G  0 part
      â""â"€centos_core--master-root                     253:0    0   449G  0 lvm  /
    sdd                                                8:48   0   200G  0 disk
    â""â"€sdd1                                             8:49   0   200G  0 part
    sr0                                               11:0    1   4.4G  0 rom
    loop0                                              7:0    0   100G  0 loop
    â""â"€docker-253:0-44154960-pool                     253:3    0   100G  0 dm
      â""â"€docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 253:4    0   100G  0 dm   /exports/serviced_volumes_v2/c2zb7s2kdi6onzyaan9qg5vdr
    loop1                                              7:1    0     2G  0 loop
    â""â"€docker-253:0-44154960-pool                     253:3    0   100G  0 dm
      â""â"€docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 253:4    0   100G  0 dm   /exports/serviced_volumes_v2/c2zb7s2kdi6onzyaan9qg5vdr
    [root@zenoss5 ~]# serviced volume status
    Status for volume /opt/serviced/var/volumes:
    Driver:      devicemapper
    Driver Type: loop-lvm
    Volume Path: /opt/serviced/var/volumes
    Thin Pool
    ---------
    Logical Volume:               docker-253:0-44154960-pool
    Metadata (total/used/avail):  2 GiB   / 60.91 MiB (3%)  / 1.941 GiB (97%)
    Data (total/used/avail):      100 GiB / 95.14 GiB (95%) / 4.855 GiB (4.9%)
    c2zb7s2kdi6onzyaan9qg5vdr Application Data
    -----------------------------------------
    Volume Mount Point:            /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr
    Filesystem (total/used/avail): 98.31 GiB / 83.64 GiB (85%) / 9.653 GiB (9.8%)
    Virtual device size:           100 GiB

    [root@zenoss5 ~]# docker info
    Containers: 38
     Running: 38
     Paused: 0
     Stopped: 0
    Images: 4
    Server Version: 17.09.0-ce
    Storage Driver: overlay
     Backing Filesystem: xfs
     Supports d_type: true
    Logging Driver: json-file
    Cgroup Driver: cgroupfs
    Plugins:
     Volume: local
     Network: bridge host macvlan null overlay
     Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
    Swarm: inactive
    Runtimes: runc
    Default Runtime: runc
    Init Binary: docker-init
    containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
    runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
    init version: 949e6fa
    Security Options:
     seccomp
      Profile: default
    Kernel Version: 3.10.0-957.1.3.el7.x86_64
    Operating System: CentOS Linux 7 (Core)
    OSType: linux
    Architecture: x86_64
    CPUs: 12
    Total Memory: 38.21GiB
    Name: zenoss5.net.umea.se
    ID: PC7S:LOW2:J6GL:RFAC:OYTM:T7R4:HVGV:6WLV:CCI4:ZXAY:HD7Q:3QWN
    Docker Root Dir: /var/lib/docker
    Debug Mode (client): false
    Debug Mode (server): false
    Registry: https://index.docker.io/v1/
    Experimental: false
    Insecure Registries:
     127.0.0.0/8
    Live Restore Enabled: false

    [root@zenoss5 ~]# grep FS /etc/default/serviced
    # SERVICED_FS_TYPE=devicemapper
    # Set to 0 in order to prevent this host from attempting to mount the DFS
    # SERVICED_NFS_CLIENT=1
    # The timeout for performing a DFS snapshot (in seconds)
    # SERVICED_MAX_DFS_TIMEOUT=300
    [root@zenoss5 ~]# grep DM_ /etc/default/serviced
    # SERVICED_DM_ARGS=
    SERVICED_DM_BASESIZE=300G
    #SERVICED_DM_LOOPDATASIZE=100G
    # SERVICED_DM_LOOPMETADATASIZE=2G
    # SERVICED_DM_THINPOOLDEV=
    [root@zenoss5 ~]# vgs && lvs
    File descriptor 3 (/root/appliance/adminmenu.options) leaked on vgs invocation. Parent PID 3620: bash
      VG                 #PV #LV #SN Attr   VSize VFree
      centos_core-master   4   3   0 wz--n- 1.07t 200.99g
    File descriptor 3 (/root/appliance/adminmenu.options) leaked on lvs invocation. Parent PID 3620: bash
      LV   VG                 Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
      home centos_core-master -wi-ao---- 429.80g
      root centos_core-master -wi-ao---- 449.00g
      swap centos_core-master -wi-ao---- <19.19g


    regards /F

    ------------------------------
    Fredrik
    Umea
    ------------------------------



  • 13.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 17 days ago

    Hi Fredrik

    Let's start from the top. For the docker stuff you are using the overlay storage driver. It's not recommended by Zenoss but it's not the point here.

    For the serviced stuff you are using the docker devicemapper storage driver in loopback-lvm mode (loop-lvm mode is not supported) – but ok.

    https://docs.docker.com/storage/storagedriver/select-storage-driver/

    The devicemapper storage driver uses a thin-pool which has two storage areas, Data and Metadata.

    In your case it looks like you have two files mounted (loop0 100GB/Data and loop1 2GBMetadata)

    loop0                                                                             7:0   0   100G  0 loop

    docker-253:0-44154960-pool                     253:3    0   100G  0 dm  

      docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 253:4    0   100G  0 dm   /exports/serviced_volumes_v2/c2zb7s2kdi6onzyaan9qg5vdr

     

    loop1                                                                              7:1    0     2G  0 loop

    docker-253:0-44154960-pool                     253:3    0   100G  0 dm  

      docker-253:0-44154960-2ojXvaTRZkYNXo9ErVL9T9 253:4    0   100G  0 dm   /exports/serviced_volumes_v2/c2zb7s2kdi6onzyaan9qg5vdr

     

    If you look at your serviced volume status output, you can see that your data volume has only 4.855GiB space left. The Metadata is only 3% used which is still good enough for me.

     Volume Path: /opt/serviced/var/volumes
    Thin Pool
    ---------
    Logical Volume:               docker-253:0-44154960-pool
    Metadata (total/used/avail):  2 GiB   / 60.91 MiB (3%)  / 1.941 GiB (97%)
    Data (total/used/avail):      100 GiB / 95.14 GiB (95%) / 4.855 GiB (4.9%)
    c2zb7s2kdi6onzyaan9qg5vdr Application Data
    -----------------------------------------
    Volume Mount Point:            /opt/serviced/var/volumes/c2zb7s2kdi6onzyaan9qg5vdr
    Filesystem (total/used/avail): 98.31 GiB / 83.64 GiB (85%) / 9.653 GiB (9.8%)
    Virtual device size:           100 GiB

    So, from my point of view have to extend the loop1 device. Unfortunately I haven't any experience in resizing loop devices. I found some information on the below site under "Resize a loop-lvm thin pool"

    https://docs.docker.com/storage/storagedriver/device-mapper-driver/

     

    Maybe some other readers do have and are willing to jump in!

     

    In the meantime I will check out if I can find some more information.

     

    Cheers Arthur

     

    P.S I saw that you already added 200GB (sdd)  to the environment.




  • 14.  RE: howto extend size in /dev/mapper/docker- ?

    Posted 9 days ago
    Edited by Arthur 9 days ago
    Hi Fredrik

    The metadata and data sparse file building the servcied thin-pool are probably located under /opt/serviced/var/volumes/.devicemapper/devicemapper.

    Can you please run the following commands and post the results?

    # df -h | grep volumes
    # ll -alh /opt/serviced/var/volumes/.devicemapper/devicemapper/
    # grep -v ^# /etc/default/serviced | grep .

    # dmsetup ls |grep pool | sort
    # dmsetup status |grep pool | sort

    Cheers




    ------------------------------
    Arthur
    ------------------------------