I am running Zenoss Core 5.1.5 and Control Center 1.1.7. My initial installation was 5.1.1/1.1.1, I upgraded to 5.1.4/1.1.6, then to 5.1.5/1.1.7. I am running a single machine.
After about an hour of running, I no longer get performance data in Control Center. I'm not seeing any graphs.In the logs I am seeing this message when the graphs stop showing any data.
serviced: E0801 19:20:20.237395 02642 status.go:116] Unable to look up instance memory stats(received error requesting performance data: Too many attempts: HBaseRpc(method=getClosestRowBefore, table="-ROOT-", key=".META.,tsdb,\x00\x00\x04W\x9D\x80,:,:", region=RegionInfo(table="-ROOT-", region_name="-ROOT-,,0", stop_key=""), attempt=11, timeout=-1, hasTimedout=false))
Any ideas on what could be the issue
Going to bump this. I'm still having this issue after an hour or so. My logs are then spammed with the message in my previous post and:
Jan 24 16:29:45 <hostname removed> serviced: time="2017-01-24T21:29:45Z" level=warning msg="Could not look up memory metrics for instances on service" error="received error requesting performance data: 10000 RPCs waiting on \"-ROOT-,,0\" to come back online" location="instance.go:382" logger=facade serviceids=8j9apqmcv6gow03b158epy1lw
I would say the issue is related to an IP change. I had originally built my server in the lab with a lab IP. When I moved the server into production, I changed the IP. When I deleted my host from control center I got an error that listed my lab IP.
I wasn't able to find the IP and eventually did a backup/restore to a new server and I'm not longer getting this error.