Configuration & Administration

Expand all | Collapse all

Redis not answering to collection services

  • 1.  Redis not answering to collection services

    Posted 12-15-2018 07:39 PM
    We are on RM 5.3.1 and after restarting today all of our collection service is failing the health check of redis answering. Redis seems fine and is passing all health checks. All other services are passing health checks as well except the collection services. What could be the issue? We didn't have to increase memory on collectorredis from 8 to 10gb and seem to continue to increase. We have restarted all services and services and docker to see if that would help and hasn't. We are seeing events but don't see any data on the graphs. Please help!

    ------------------------------
    Devon Solomon
    CompanyName
    ------------------------------


  • 2.  RE: Redis not answering to collection services

    Posted 12-17-2018 10:14 AM
    Memory leaks. Memory leaks everywhere in version 5.x daemons. I'd recommend moving to 6.2.1, as there are far fewer issues with the various collectors. It's still not perfect, nor as stable as version 4.2.5, but far better than 5.x.

    ------------------------------
    Jason Olson
    ------------------------------



  • 3.  RE: Redis not answering to collection services

    Posted 26 days ago

    That could be caused by a number of things.  Maybe the .rdb file for that redis instance is corrupted and needs to be re-created.  You would do that by stopping redis then finding the corresponding .rdb file for it and deleting it, then starting it again.  Example...

    For collectorredis on the localhost collector:

    /opt/serviced/var/volumes/6zbnssjd86cva5rtoa5ifuhsa/localhost/localhost_collectorredis/dump.rdb

    For the main redis instance:

    /opt/serviced/var/volumes/6zbnssjd86cva5rtoa5ifuhsa/redis/dump.rdb

    If the redis instance is for a remote collector on a delegate host it may be stored on that host's filesystem rather than inside the main serviced volume if NFS / DFS isn't mounted on the remote host.

    If it's a collectorredis that's failing it might be maxed out with metrics.  You could check the graphs for it in Control Center to see if the queue graph shows that it's filled up.  If it's the main redis instance you could check the same thing on it's graphs.  If it's getting backed up it means that there's likely a problem somewhere in the pipeline.



    ------------------------------
    Ryan Matte
    ------------------------------