Configuration & Administration

Back to discussions
Expand all | Collapse all

Zenoss 4.2.5 Database issue

  • 1.  Zenoss 4.2.5 Database issue

    Posted 01-21-2021 07:54 AM
      |   view attached

    I'm having a SQL issue with Zenoss Core 4.2.5 on Centos likely as the result of a disk full and a few power events.  It's a pretty standard install and monitors about 300 devices.  It runs really well without persistent issues.

    ****

    There is a serious problem with the MySQL event database

    Below is the detailed information provided:

    (145, "Table './zodb/new_oid' is marked as crashed and should be repaired")

    ****

    The installation is mostly functional but I get an error page intermittently on new page views and can't add/remove/modify devices.  ("unable to connect to server" prompt).

    I'm not super interested in preserving event or performance data.  Can I just drop and add a database? which one?

    My SQL skills are modest, but I did get through POSKey error (dropping and adding zodb_session).



    ------------------------------
    Greg Sullivan
    Appoquinimink School District
    Odessa DE
    3024493832
    ------------------------------

    Attachment(s)

    rtf
    zenoss problem error.rtf   5 KB 1 version


  • 2.  RE: Zenoss 4.2.5 Database issue

    Posted 01-22-2021 08:51 AM
    ​Hi,

    I'm pretty sure i've seen this error in amougst the various errors i've seen with corrupted event databases.

    I've got a pair of scripts that totally removes then rebuilds the event database if you want to give them a try...

    Main script, save this with whatever name you fancy in a top level dir called /fixme/ (or edit script to work in whatever location you want)
    #!/bin/sh
set -x #echo on

# STOP ZENOSS
sudo -i -u zenoss zenoss stop

# TRY AND REMOVE ZENOSS_ZEP
# MAY WELL FAIL AS THIS SCRIPT IS TO BE USED WHEN ZENOSS_ZEP CORRUPTED
mysql -u root << DROPZEPSQL
show databases;
drop database zenoss_zep;
\! echo "zenoss_zep drop run"
show databases;
DROPZEPSQL

# STOP MYSQL
service mysql stop

# EMPTY AND REMOVE ZENOSS_ZEP DIRECTORY ENTIRELY
rm /var/lib/mysql/zenoss_zep/*.*
rmdir /var/lib/mysql/zenoss_zep/

# START MYSQL
service mysql start

# DROP ZENOSS_ZEP IF IT STILL EXISTS
mysql -u root << DROPZEPSQL
show databases;
drop database zenoss_zep;
\! echo "zenoss_zep drop run"
show databases;
DROPZEPSQL

# FORCE CREATE ZENOSS_ZEP - Often fails after full remove
sudo -i -u zenoss /fixme/create_zepdb.sh

# STOP MYSQL
service mysql stop

# EMPTY POSSIBLE PARTIALLY CREATED ZENOSS_ZEP DIRECTORY
rm /var/lib/mysql/zenoss_zep/*.*

# START MYSQL
service mysql start

# FORCE CREATE ZENOSS_ZEP
sudo -i -u zenoss /fixme/create_zepdb.sh

# START ZENOSS
sudo -i -u zenoss zenoss start​


    Secondary script (called via sudo as zenoss user from main script) - save this as create_zepdb.sh in the same /fixme/ directory

    #/bin/sh
set -x #echo on

zengc=/opt/zenoss/bin/zenglobalconf

# read configuration from global.conf
dbtype=$(${zengc} -p zodb-db-type)
host=$(${zengc} -p zodb-host)
port=$(${zengc} -p zodb-port)
user=$(${zengc} -p zodb-user)
userpass=$(${zengc} -p zodb-password)
admin=$(${zengc} -p zodb-admin-user)
adminpass=$(${zengc} -p zodb-admin-password)
dbname=$(${zengc} -p zodb-db)

# Drop and recreate the ZEP event database
zeneventserver-create-db --dbtype $dbtype --dbhost $host --dbport $port --dbadminuser $admin --dbadminpass "${adminpass}" --dbuser $user --dbpass "${userpass}" --force
​

    Run the main script as root and it should drop the event database, remove the directories and recreate.
    It does the recreate twice as during tests I saw it fail under some circumstances on the first go. This one seems to work everytime.

    Hope this helps.

    Cheers

    ------------------------------
    Pheripheral Pheripheral
    ------------------------------

    Original Message


  • 3.  RE: Zenoss 4.2.5 Database issue

    Posted 01-22-2021 12:07 PM
      |   view attached
    That worked ... I attached the output.

    Those error pages went away I believe, but I'm still not able to modify or delete devices.  I still get the small black window dead center on the screen that says "server exception" ... unable to connect to server.

    The UI responds to navigation but not functions that modify devices (the radio buttons to turn services off/on in advanced >> daemons work fine though).

    Can't change production state, IP address etc.

    Could this just be missing host info?


    ------------------------------
    Greg Sullivan
    Appoquinimink School District
    Odessa DE
    3024493832
    ------------------------------

    Attachment(s)

    rtf
    fixme-output.rtf   6 KB 1 version
    Original Message


  • 4.  RE: Zenoss 4.2.5 Database issue

    Posted 23 days ago
    Yeah that script did complete but it didn't fix the problem.
    **********************************

    [zenoss@appomonitor01 ~]$ zodbscan

    Traceback (most recent call last):

      File "/opt/zenoss/bin/zodbscan", line 8, in <module>

        load_entry_point('zenoss.toolbox==2.3.1', 'console_scripts', 'zodbscan')()

      File "/opt/zenoss/lib/python/pkg_resources.py", line 318, in load_entry_point

        return get_distribution(dist).load_entry_point(group, name)

      File "/opt/zenoss/lib/python/pkg_resources.py", line 2221, in load_entry_point

        return ep.load()

      File "/opt/zenoss/lib/python/pkg_resources.py", line 1954, in load

        entry = __import__(self.module_name, globals(),globals(), ['__name__'])

      File "/opt/zenoss/lib/python2.7/site-packages/zenoss.toolbox-2.3.1-py2.7.egg/zenoss/toolbox/zodbscan.py", line 28, in <module>

        from zodbpickle.pickle import Unpickler as UnpicklerBase

    ImportError: No module named zodbpickle.pickle

    ******************************************************
    I downloaed the zenoss toolbox and zodbscan gives an error.  zenrelationscan and findposkeyerror are having issues too with fixing.

    I took it from the top... dropped and created zodb_session and went back to the toolbox but still no luck.  I'm attaching the logs from this


    ------------------------------
    Greg Sullivan
    Appoquinimink School District
    Odessa DE
    3024493832
    ------------------------------

    Original Message