Configuration & Administration

Expand all | Collapse all

ZenOSS 6 thin pool storage filling up quickly

  • 1.  ZenOSS 6 thin pool storage filling up quickly

    Posted 03-01-2019 02:19 PM
      |   view attached
    It fills up by about 30GB per day, I attached a screenshot showing the thin pool storage growth.  Is there a way to determine which control center container is filling up the thin pool?

    Its a fresh install of ZenOSS 6, below is the version info for control center.


    Version: 1.5.1
    GoVersion: go1.7.4
    Date: Fri_May_18_18:35:47_UTC_2018
    Gitcommit: 9ccf1f2-dirty
    Gitbranch: HEAD
    Buildtag: jenkins-ControlCenter-support-1.5.x-merge-rpm-build-7
    Release: 1.5.1-1

    - David


  • 2.  RE: ZenOSS 6 thin pool storage filling up quickly

    Posted 03-01-2019 05:30 PM
    Have seen this if you have any snapshots left over. Check it with

    # serviced snapshot list


    ​​

    ------------------------------
    Arthur
    ------------------------------



  • 3.  RE: ZenOSS 6 thin pool storage filling up quickly

    Posted 03-02-2019 02:46 AM
    Edited by Arthur 03-02-2019 03:58 AM
    This could help also:
    Display the amount of disk space each container is using:

    for d in `docker ps | awk '{print $1}' | tail -n +2`; do
        d_name=`docker inspect -f {{.Name}} $d`
        echo "========================================================="
        echo "$d_name ($d) container size:"
        sudo du --max-depth 2 -h /var/lib/docker/devicemapper | grep `docker inspect -f "{{.Id}}" $d`
        echo "$d_name ($d) volumes:"
        for mount in `docker inspect -f "{{range .Mounts}} {{.Source}}:{{.Destination}}                                                                                                                                                      
        {{end}}" $d`; do
            size=`echo $mount | cut -d':' -f1 | sudo xargs du -d 0 -h`
            mnt=`echo $mount | cut -d':' -f2`
            echo "$size mounted on $mnt"
        done
    done

    https://github.com/control-center/serviced/wiki/Control-Center-Tips-and-Tricks



  • 4.  RE: ZenOSS 6 thin pool storage filling up quickly

    Posted 03-07-2019 02:05 PM
    Tried @Arthur 's bash snippet to show each containers usage, it doesn't look like its using anything.  thinpool is showing 89GB used.

    ==================================================================================================================/serviced-isvcs_logstash (b7ce42e58129) container size:/serviced-isvcs_logstash (b7ce42e58129) volumes:54M /var/log/serviced mounted on /var/log/serviced16M /opt/serviced/isvcs/resources mounted on /usr/local/serviced/resources=========================================================/serviced-isvcs_kibana (4dd8b50e8ff7) container size:/serviced-isvcs_kibana (4dd8b50e8ff7) volumes:16M /opt/serviced/isvcs/resources mounted on /usr/local/serviced/resources=========================================================/serviced-isvcs_opentsdb (f9ace2ec0df8) container size:/serviced-isvcs_opentsdb (f9ace2ec0df8) volumes:2.4G /opt/serviced/var/isvcs/opentsdb/hbase mounted on /opt/zenoss/var/hbase16M /opt/serviced/isvcs/resources mounted on /usr/local/serviced/resources=========================================================/serviced-isvcs_elasticsearch-logstash (d034ad354b89) container size:/serviced-isvcs_elasticsearch-logstash (d034ad354b89) volumes:1.8G /opt/serviced/var/isvcs/elasticsearch-logstash/data mounted on /opt/elasticsearch-logstash/data16M /opt/serviced/isvcs/resources mounted on /usr/local/serviced/resources=========================================================/serviced-isvcs_docker-registry (a4f2ada3dc2e) container size:/serviced-isvcs_docker-registry (a4f2ada3dc2e) volumes:975M /opt/serviced/var/isvcs/docker-registry/v2 mounted on /tmp/registry-dev16M /opt/serviced/isvcs/resources mounted on /usr/local/serviced/resources=========================================================/serviced-isvcs_elasticsearch-serviced (8f241689c8ea) container size:/serviced-isvcs_elasticsearch-serviced (8f241689c8ea) volumes:5.5M /opt/serviced/var/isvcs/elasticsearch-serviced/data mounted on /opt/elasticsearch-serviced/data16M /opt/serviced/isvcs/resources mounted on /usr/local/serviced/resources=========================================================/serviced-isvcs_zookeeper (075190ebbe36) container size:/serviced-isvcs_zookeeper (075190ebbe36) volumes:3.1M /opt/serviced/var/isvcs/zookeeper/data mounted on /var/zookeeper16M /opt/serviced/isvcs/resources mounted on /usr/local/serviced/resources

    There were two volumes in /opt/serviced/var/volumes/.  I tried fstrim on both, output shown below.  It cleared my usage down to 7GB after running fstrim, so maybe a cron will fix my issue.

    # /sbin/fstrim /opt/serviced/var/volumes/4hoc5vjm0doqdvzvapq13vi9g
    fstrim: /opt/serviced/var/volumes/4hoc5vjm0doqdvzvapq13vi9g: the discard operation is not supported
    # /sbin/fstrim /opt/serviced/var/volumes/d5ig9lx0u0hnap5fuuxxq62ei

    Thanks for your help.  I will report back if this issue continues, but I think fstrim may resolve it.

    ------------------------------
    FirstName LastName
    CompanyName
    ------------------------------



  • 5.  RE: ZenOSS 6 thin pool storage filling up quickly

    Posted 03-07-2019 01:57 PM
    No snapshots.

    # serviced snapshot list
    no snapshots found

    ------------------------------
    FirstName LastName
    CompanyName
    ------------------------------



  • 6.  RE: ZenOSS 6 thin pool storage filling up quickly

    Posted 03-07-2019 02:57 PM
      |   view attached
    Hi David

    I have attached the /etc/cron.weekly/serviced-fstrim from Zenoss for reference.

    ------------------------------
    Arthur
    ------------------------------

    Attachment(s)

    txt
    serviced-fstrim.txt   2 KB 1 version


  • 7.  RE: ZenOSS 6 thin pool storage filling up quickly

    Posted 03-03-2019 12:37 PM

    This will happen over time with a thinpool.  What you need to do is crontab a job to perform an fstrim on the volume periodically.  For instance on my lab box I did crontab -e to edit the crontab as root and then I put the following line...

    */15 * * * * /sbin/fstrim /opt/serviced/var/volumes/6zbnssjd86cva5rtoa5ifuhsa

    You would need to replace 6zbnssjd86cva5rtoa5ifuhsa with your instance id.  That runs an fstrim against the volume every 15 minutes.  When something is deleted from a thinpool the old data remains.  An fstrim is required to clean up that old data and bring the utilization back down.  You can try just running the fstrim command by hand as root to see if it causes the utilization to drop back down.

    If it doesn't cause it to drop back down then it's likely legitimate usage at which point you need to cd in to /opt/serviced/var/volumes/<instance id> and then do a "du -sh *" to figure out which of the directories is taking up the most space.  Try the fstrim like I mentioned above and then follow up to let us know if that resolves it or not.



    ------------------------------
    Ryan Matte
    ------------------------------



  • 8.  RE: ZenOSS 6 thin pool storage filling up quickly

    Posted 05-04-2021 03:32 PM
    Hi Ryan,

    I see similar issue on 6.2.3 and in hbase-master folder i see oldWAls to be around 333GB. How to get rid of that cleanly? Below is data i see

    3.3G archive
    4.0K corrupt
    221G data
    4.0K hbase.id
    4.0K hbase.version
    4.0K MasterProcWALs
    333G oldWALs
    832M WALs



    ------------------------------
    krishna reddy
    cox
    ------------------------------



  • 9.  RE: ZenOSS 6 thin pool storage filling up quickly

    Posted 05-04-2021 03:47 PM
    Stop then start the hbase services. That should cause it to clear up.

    ------------------------------
    Ryan Matte
    ------------------------------



  • 10.  RE: ZenOSS 6 thin pool storage filling up quickly

    Posted 05-04-2021 04:09 PM
    Thanks Ryan. If for some reason, that doesn't clean, is it fine to manullay delete those files as they seem to be log files from replication?

    see this error:

    2021-05-04 16:40:01,811 ERROR [localhost,60000,1618593951858_ChoreService_1] zookeeper.ZooKeeperWatcher: replicationLogCleaner-0x178dbb7a27b0003, quorum=zk1:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:835) at org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80) at org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99) at org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.getDeletableFiles(ReplicationLogCleaner.java:70) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:233) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:157) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745), replicationLogCleaner-0x178dbb7a27b0003, quorum=zk1:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:835) at org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80) at org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99) at org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.getDeletableFiles(ReplicationLogCleaner.java:70) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:233) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:157) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124) at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

    ------------------------------
    krishna reddy
    cox
    ------------------------------



  • 11.  RE: ZenOSS 6 thin pool storage filling up quickly

    Posted 05-04-2021 04:19 PM
    Thanks Ryan. Restart fixed it...

    ------------------------------
    krishna reddy
    cox
    ------------------------------