Configuration & Administration

 View Only
  • 1.  Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events

    Posted 10 days ago
    2022-01-12T13:31:20.926 [INDEXER_EVENT_SUMMARY] ERROR org.springframework.transaction.interceptor.T
    ransactionInterceptor - Application exception overridden by rollback exception
    org.springframework.dao.DeadlockLoserDataAccessException: PreparedStatementCallback; SQL [INSERT IN
    TO event_trigger_signal_spool (created,event_summary_uuid,event_trigger_subscription_uuid,flush_tim
    e,event_count,sent_signal,uuid) VALUES(?,?,?,?,?,?,?)]; Deadlock found when trying to get lock; try
     restarting transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollba
    ckException: Deadlock found when trying to get lock; try restarting transaction
            at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLError
    CodeSQLExceptionTranslator.java:265) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]

    This error is followed by bunch of other jdbc errors. It seems due to the deadlock, a transaction failure happens and then it gets rolled back at lease that's what I see in the errors following it.    However the major impact is that lots of events gets stacked up in the RabbitMqueue until the deadlock goes away.

    Is there something needed to tune mysql or something to avoid this deadlock?

    Please advise,
    Thanks





    ------------------------------
    Sam
    ------------------------------


  • 2.  RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events

    Posted 10 days ago
    Sam,

    If you check the archived Zenoss 4.2.5 Installation Manual, step 2.5 directs you to the MySQL Tuner script at mysqltuner.pl.

    Later versions point toward the Percona Toolkit instead.  It's been a while since I've laid a hand on either of them, so I can't recommend one over the other.

    ------------------------------
    Michael J. Rogers
    Senior Instructor - Zenoss
    Austin TX
    ------------------------------



  • 3.  RE: Zenoss 4.2.5 - Zeneventserver error - Deadlock detected - General failure indexing events

    Posted yesterday
    Here is the deadlock information provided by

    mysql> show engine innodb status\G

    *************************** 1. row ***************************
      Type: InnoDB
      Name:
    Status:
    =====================================
    220119 10:10:36 INNODB MONITOR OUTPUT
    =====================================
    Per second averages calculated from the last 38 seconds
    -----------------
    BACKGROUND THREAD
    -----------------
    srv_master_thread loops: 7878348 1_second, 7878344 sleeps, 787831 10_second, 42 background, 42 flush
    srv_master_thread log flush and writes: 7893321
    ----------
    SEMAPHORES
    ----------
    OS WAIT ARRAY INFO: reservation count 29621057, signal count 44703141
    Mutex spin waits 251001014, rounds 448583868, OS waits 2082513
    RW-shared spins 14501169, rounds 154491848, OS waits 3185354
    RW-excl spins 36891245, rounds 770873766, OS waits 24055859
    Spin rounds per wait: 1.79 mutex, 10.65 RW-shared, 20.90 RW-excl
    ------------------------
    LATEST DETECTED DEADLOCK
    ------------------------
    220119  8:52:33
    *** (1) TRANSACTION:
    TRANSACTION 35583B36, ACTIVE 0 sec starting index read
    mysql tables in use 1, locked 1
    LOCK WAIT 6 lock struct(s), heap size 1248, 2 row lock(s)
    MySQL thread id 922890, OS thread handle 0x7fd30c07e700, query id 210213559 localhost 127.0.0.1 zenoss statistics
    SELECT event_count,first_seen,last_seen,details_json,status_id,uuid FROM event_summary WHERE fingerprint_hash=x'A784                            D0E9A2E2B0C19116665D777B252FDEA93E9F' FOR UPDATE
    *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
    RECORD LOCKS space id 0 page no 1117028 n bits 72 index `PRIMARY` of table `zenoss_zep`.`event_summary` trx id 35583                            B36 lock_mode X locks rec but not gap waiting
    Record lock, heap no 4 PHYSICAL RECORD: n_fields 40; compact format; info bits 0
    
    
    *** (2) TRANSACTION:
    TRANSACTION 35583B35, ACTIVE 0 sec updating or deleting
    mysql tables in use 1, locked 1
    9 lock struct(s), heap size 1248, 11 row lock(s), undo log entries 11
    MySQL thread id 922930, OS thread handle 0x7fd2f0895700, query id 210213568 localhost 127.0.0.1 zenoss Updating
    UPDATE event_summary SET status_id=4,status_change=1642611153281,update_time=1642611153281,current_user_uuid=null,cu                            rrent_user_name=null,cleared_by_event_uuid=x'5254007B971A89FF11EC794832755829',fingerprint_hash=x'9E0AFDD08B913BAF0D                            5B8C0FB3AAC023488CE954',audit_json='{"timestamp":1642611153281,"new_status":4}' WHERE uuid=x'5254007B971A89FF11EC794                            83274BBE8'
    *** (2) HOLDS THE LOCK(S):
    RECORD LOCKS space id 0 page no 1117028 n bits 72 index `PRIMARY` of table `zenoss_zep`.`event_summary` trx id 35583                            B35 lock_mode X locks rec but not gap
    Record lock, heap no 4 PHYSICAL RECORD: n_fields 40; compact format; info bits 0
    
    
    *** (2) WAITING FOR THIS LOCK TO BE GRANTED:
    RECORD LOCKS space id 0 page no 4360710 n bits 320 index `fingerprint_hash` of table `zenoss_zep`.`event_summary` tr                            x id 35583B35 lock_mode X locks rec but not gap waiting
    Record lock, heap no 250 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
     0: len 20; hex a784d0e9a2e2b0c19116665d777b252fdea93e9f; asc           f]w{%/  > ;;
     1: len 16; hex 5254007b971a89ff11ec79483274bbe8; asc RT {      yH2t  ;;
    
    *** WE ROLL BACK TRANSACTION (1)
    ​


    ------------------------------
    Sam
    ------------------------------