Configuration & Administration

Expand all | Collapse all

serviced fails to start due to devicemapper error

Jump to Best Answer
  • 1.  serviced fails to start due to devicemapper error

    Posted 15 days ago
    Edited by Jad Baz 15 days ago
      |   view attached
    I'm an error with Zenoss 6.2.1 (serviced 1.5.1) on a fresh Centos 7.5 with docker 17.09.0-ce.
    The error is the same as this thread but appeared here on startup.

    The weird thing is, it was working quite well until I logged in to check something and found that it was slowing down tremendously (I'm working on a 12 GB machine).
    I rebooted the machine and started getting the error.
    Oct 30 15:20:05 LDLBN019WMZNS01 serviced: E1030 13:20:05.438238 05007 devicemapper.go:477] Error mounting device "5DCUDNVNWnT9m0zluVxXQ1" on "/opt/serviced/var/volumes/9u3l0amg9fhm9ps59qtu71xqt" for volume "9u3l0amg9fhm9ps59qtu71xqt": devmapper: Error mounting '/dev/mapper/docker-253:0-34009442-5DCUDNVNWnT9m0zluVxXQ1' on '/opt/serviced/var/volumes/9u3l0amg9fhm9ps59qtu71xqt': input/output error
    Oct 30 15:20:05 LDLBN019WMZNS01 serviced: time="2018-10-30T13:20:05Z" level=fatal msg="Could not get volume for tenant" error="devmapper: Error mounting '/dev/mapper/docker-253:0-34009442-5DCUDNVNWnT9m0zluVxXQ1' on '/opt/serviced/var/volumes/9u3l0amg9fhm9ps59qtu71xqt': input/output error" location="daemon.go:515" logger=cli.api tenantid=9u3l0amg9fhm9ps59qtu71xqt​

    here is my lsblk:
    [root@LDLBN019WMZNS01 ~]# lsblk
    NAME                                               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    sda                                                  8:0    0   40G  0 disk
    ├─sda1                                               8:1    0    1G  0 part /boot
    └─sda2                                               8:2    0   39G  0 part
      ├─centos-root                                    253:0    0   35G  0 lvm  /
      └─centos-swap                                    253:1    0    4G  0 lvm  [SWAP]
    sdb                                                  8:16   0   40G  0 disk
    ├─zenoss-docker--pool_tmeta                        253:2    0  816M  0 lvm
    │ └─zenoss-docker--pool                            253:4    0 19.6G  0 lvm
    ├─zenoss-docker--pool_tdata                        253:3    0 19.6G  0 lvm
    │ └─zenoss-docker--pool                            253:4    0 19.6G  0 lvm
    ├─zenoss-serviced--pool_tmeta                      253:5    0   20M  0 lvm
    │ └─zenoss-serviced--pool                          253:7    0 16.9G  0 lvm
    │   └─docker-253:0-34009442-5DCUDNVNWnT9m0zluVxXQ1 253:15   0  100G  0 dm
    └─zenoss-serviced--pool_tdata                      253:6    0 16.9G  0 lvm
      └─zenoss-serviced--pool                          253:7    0 16.9G  0 lvm
        └─docker-253:0-34009442-5DCUDNVNWnT9m0zluVxXQ1 253:15   0  100G  0 dm
    sr0                                                 11:0    1 1024M  0 rom​

    I'm also attaching the relevant part of /var/log/messages

    PS: how can I only subscribe to threads I participate in (like this one) and not subscribe to the entire forum in which it is posted?

    ------------------------------
    Jad
    ------------------------------

    Attachment(s)

    txt
    var_log_messages.txt   40K 1 version


  • 2.  RE: serviced fails to start due to devicemapper error

    Posted 7 days ago
    Anyone?

    ------------------------------
    Jad
    ------------------------------



  • 3.  RE: serviced fails to start due to devicemapper error
    Best Answer

    Posted 6 days ago
    Hi

    Looks like you have a file-system issue (Dirty inode writeback failed for block device dm-15) and then the follwoing mount fails.

    Oct 30 15:20:05 LDLBN019WMZNS01 kernel: Buffer I/O error on dev dm-15, logical block 13107214, lost async page write
    Oct 30 15:20:05 LDLBN019WMZNS01 kernel: JBD2: recovery failed
    Oct 30 15:20:05 LDLBN019WMZNS01 kernel: EXT4-fs (dm-15): error loading journal
    Oct 30 15:20:05 LDLBN019WMZNS01 kernel: VFS: Dirty inode writeback failed for block device dm-15 (err=-5).
    Oct 30 15:20:05 LDLBN019WMZNS01 serviced: E1030 13:20:05.438238 05007 devicemapper.go:477] Error mounting device "5DCUDNVNWnT9m0zluVxXQ1" on "/opt/serviced/var/volumes/9u3l0amg9fhm9ps59qtu71xqt" for volume "9u3l0amg9fhm9ps59qtu71xqt": devmapper: Error mounting '/dev/mapper/docker-253:0-34009442-5DCUDNVNWnT9m0zluVxXQ1' on '/opt/serviced/var/volumes/9u3l0amg9fhm9ps59qtu71xqt': input/output error

    Next step would by trying to fix the fs with fsck. Looks like you are using EXT4 where XFS is recommended as far I know :-)

    https://www.zenoss.com/services-support/documentation/planning-guide


    ​​

    ------------------------------
    Cheers
    Arthur
    ------------------------------



  • 4.  RE: serviced fails to start due to devicemapper error

    Posted 6 days ago
    Thanks.

    I couldn't get this to work so I wiped my server and started from scratch.
    I will keep this in my if it happens again

    ------------------------------
    Jad
    ------------------------------



  • 5.  RE: serviced fails to start due to devicemapper error

    Posted 4 days ago
    Hi Jad

    Yep, that's another possibility to solve it. :-)

    Thanks for replay.

    ------------------------------
    Arthur
    ------------------------------