Reporting, Analysis and Analytics

Topic: Zenoss 4.2.5 auto deploy script feedback 

1.  Zenoss 4.2.5 auto deploy script feedback

Posted 12-01-2017 05:01 PM
https://github.com/jcurry/Zenoss_4.2.5_core-autodeploy/tree/master

I tried using the script on a minimal CentOS 6.9 VM.

I had to install wget and remove the mysql-libs that was already installed.

And I guess you renamed the script to "core-autodeploy.sh" instead of whatever you have on the GitHub page.

The script ran fine until I ran into this error.

Complete!
Installing optimal /etc/my.cnf settings
Configuring MySQL
Starting MySQL.. SUCCESS!
Enabling rpmforge repo...
--2017-12-01 09:06:44-- http://apt.sw.be/redhat/el6/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
Resolving apt.sw.be... failed: Name or service not known.
wget: unable to resolve host address "apt.sw.be"
Command failure: wget http://apt.sw.be/redhat/el6/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm

------------------------------
Clay
Network Computer Technician
------------------------------


2.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-04-2017 06:54 AM
Hi Clay,
The install script shouldn't be using the rpmforge repo.  Could I suggest you follow the instructions in the README to run the yum commands to remove pre-req software, making sure you also do the "yum clean all".  I sometimes find that the zenossdeps-4.2.x-1.el6.noarch doesn't get removed by this - not sure why - but you can shift it with:
rpm -e zenossdeps-4.2.x-1.el6.noarch

I have just run this through again on my CentOS 6.3 build and it worked fine.  It may be that you have that rpmforge.repo enabled for something else and it is getting invoked anyway?  I found a google hit here - https://github.com/repoforge/rpms/issues/378  that might be helpful.

Since I was checking things, I have also updated the CentOS autodeploy script (in the git master branch), to install the latest SUP732, rather than the earlier SUP671.  That also ran clean.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------



3.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-04-2017 05:30 PM
Thanks Jane.

I went through and did all the "yum remove" commands.  Then I edited the script to go to "http://ftp.tu-chemnitz.de/pub/linux/dag/redhat/el6/en/x86_64/rpmforge/RPMS/" to get the rpmforge package.

And hey, it works!

Do you have a magic script for moving my data?

------------------------------
Clayton
Network Computer Technician
------------------------------



4.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-05-2017 04:17 AM
Magic is in the eye of the beholder ;)

Depends on where you are starting from (Zenoss version) and what you really want to move.

Earlier releases of Zenoss 4.x you may simply be able to run a full zenbackup on the earlier version and then zenrestore on the new build.  Depending on your exact versions, watch for the --mibs and --libexec flags which backup the mibs directory and the $ZENHOME/libexec directory - you want these if you have a late enough zenbackup to support them.  The other trick you need to ensure is that you have installed the same ZenPacks on your old and new systems - don't think they have to be identical versions but they must be there.

Other than that, I have done a number of these migration jobbies, even going back as far as Zenoss 2.x.  zenbatchdump / zenbatchload is your friend to move your Zope database - with the possibility to prune stuff you don't want any more.

If you have customised templates, scripts, events, mibs then I would create a migration ZenPack.

Depending on how old you "old" is, you may or may not get your events or  performance data moved.  If you are starting with Zenoss 3 then I think you have to give up on your existing events as the database architecture changed completely with 4.  I have a feeling that you can't migrate performance data if you are going from a 32-bit Zenoss to a 64-bit.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------



5.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-05-2017 04:34 PM
In my case, all I want to do is get my list of devices, with all their attributes, from the v4.2.5 that I created with the OVA (CentOS 5, not patched) to the v4.2.5 I created from your script (CentOS 6, patched to 732).

The only ZenPacks that the old server has that the new server doesn't are:

ZenPacks.community.Brocade
ZenPacks.community.ConstructionKit
ZenPacks.community.deviceAdvDetail
ZenPacks.vaibhav.brocadeswitches

I know why I have the two Brocade ZenPacks but the other two I don't recognize.  Are you familiar with them?

------------------------------
Clayton
Network Computer Technician
------------------------------



6.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-06-2017 03:00 AM
ZenPacks.community.ConstructionKit is a community development kit from Joseph Anderson, long before Zenoss produced their zenpacklib / SDK.  He then built some of his ZenPacks on top of that Construction Kit so it may be a pre-req to one of your brocade ZenPacks.

ZenPacks.community.deviceAdvDetail in some ways is similar but even earlier, built by "Big Egor", an amazing early contributor to the Zenoss community.  Again, it was a prereq for some of his ZenPacks and I think other developers also built on top of it.

Either way, if you are planning to go with zenbackup / zenrestore, I still think you will need the SAME ZenPacks on old and new, even if they are not being used.  If you can confirm that ZenPacks are not needed in your old environment then, having taken a full system backup, I would try and remove them, check things are clean, and then start on the zenbackup process.

If you are going from 4.2.5 to 4.2.5 then you shouldn't run into any version incompatibilities.  I think you will find that the --mibs --libexec flags do exist on your old Zenoss.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------



7.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-06-2017 05:15 PM
Should I be using zenbackup/zenrestore or zenbatchdump/zenbatchload?

And how would I be able to confirm that a particular zenpack wasn't needed in my old environment?


------------------------------
Clay
Network Computer Technician
------------------------------



8.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-06-2017 05:28 PM
FYI: I had to create a "Blade Switches" under Devices before the ZenPacks.vaibhav.brocadeswitches would install.

And it looks like ZenPacks.community.ConstructionKit and ZenPacks.community.deviceAdvDetail are dependencies for ZenPacks.community.Brocade which I don't think I need anymore.



------------------------------
Clayton
Network Computer Technician
------------------------------



9.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-07-2017 03:02 AM
If you really dont need the Brocade ZenPack then it would make sense to remove it and those 2 pre-reqs, before moving forward.  Personally, I would be VERY sure that I had a VMware snapshot (if your Zenoss server is a VM) or an operating system backup, before doing this.  Sounds like you have already seen some odd things around your new build with those ZenPacks :(   Actually, if you got them installed in the new environment, back that up and test removal on the new system first.  I just have a feeling that I have struggled removing stuff built on the ConstructionKit ZenPack in the past - that is why I am suggesting you have a very safe plan B.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------



10.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-07-2017 04:47 PM
Thanks.  I took a snapshot and removed 3 of the 4 ZenPacks and everything seems to be working.  I still have all 611 of my devices and I can drill down into at least some of their details.

Should I be using zenbackup/zenrestore or zenbatchdump/zenbatchload for moving my data from my old server to my new one?

------------------------------
Clayton
Network Computer Technician
------------------------------



11.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-08-2017 02:04 AM
I would try zenbackup / zenrestore.  That will get all of your zodb database (all devices, components, event definitions, mibs,, templates, reports,...).  You can use the --no-eventsdb if you dont want to backup your events.  You can use --no-perfdata if you dont want to backup your performance data.  zenbackup also backs up several of the Zenoss directories with configuration data - it really is a backup of your entire Zenoss.

Zenbatchdump / zenbatchload will ONLY bring devices across - not templates, not event definitions......  It's the next option if zenbackup/zenrestore won't work (like if you are going from very different versions of Zenoss or you are going from, say, an Ubuntu platform to CentOS.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------



12.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-08-2017 02:02 PM
I'm mostly interested with getting the devices with all their data and not so much their historical data so I went with the --noeventsdb and the --noperfdata but I'm getting what looks like a mysql password error when I try to restore.

Restoring ZODB database.
WARNING:zenbackupbase:ERROR 1045 (28000): Access denied for user 'zenoss'@'localhost' (using password: YES)

ERROR 1045 (28000): Access denied for user 'zenoss'@'localhost' (using password: YES)

gzip: stdout: Broken pipe
Done Restoring ZODB database.

Any suggestions?

------------------------------
Clayton
Network Computer Technician
------------------------------



13.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12-09-2017 09:00 AM
Do you have the --save-mysql-access flag on the zenbackup  on your old Zenoss?  If so, do another backup using that.
Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------



14.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 13 days ago
Well, there's a "--no-save-mysql-access" option but no "save-mysql-access" so I tried that.

zenbackup and zenrestore seemed to have worked.  The device count on my new server matches the count on my old server and it looks like my devices are all there.

Unfortunately, there are no events showing on the new server (and there should be) and if I try to access a device from the Infrastructure window, it shows a getProductionState error (no details, just the option of sending the error to Zenoss).

Any ideas?

------------------------------
Clayton
Network Computer Technician
------------------------------



15.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 13 days ago
Ok, weird.  Under /opt/zenoss/perf, my new server has no Devices sub-folder but in the web interface under Infrastructure, there are 612 Devices listed.

I wonder what would happen if I zipped the Devices sub-folder from my old server and copied it to my new server.


------------------------------
Clayton
Network Computer Technician
------------------------------



16.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12 days ago
Edited by Jane Curry 11 days ago
If you did a full zenbackup then it should have included the performance data - there is a --no-perfdata flag but i hope you didn't use it?

Your backup file is just a big tar gz file so you can find a (large) piece of filesystem, outside of /opt/zenoss, and unpack it - you should find a packed file in there wth all your perf data.

On the getProductionState error, I have seen this with the latest ZUP.  If a device didn't have prodState set for some reason, then it barfs with the latest zup.  I fixed it using zendmd. - there's a post here - https://community.zenoss.com/forum/community-home/digestviewer/viewthread?GroupId=25&MID=3429&CommunityKey=f1fb2446-4ac2-40dd-8b4c-0d451d151f20&tab=digestviewer#bm4    that talks about changing prodState from zendmd.  I found it was only one or two devices with this problems so hand-tweaking them was a reasonable proposition.

Cheers,
Jane

NB.  Link above has been updated.

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------



17.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12 days ago
Yes, I did use the --no-perfdata flag.  How screwed am I?

And the link didn't work for me but I'll try to find it by searching.

------------------------------
Clayton
Network Computer Technician
------------------------------



18.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 12 days ago
I think I found the post.  Is this what you were referring to?

zendmd

d=find('zen42.class.example.org')

d.productionState responds with 300 ie Maintenance

d.setProdState(1000)

d.productionState responds with 1000 ie Production

commit()

If so, I don't have a FQDN for the new server yet so is the IP address good enough?



------------------------------
Clayton
Network Computer Technician
------------------------------



19.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 11 days ago
Edited by Jane Curry 11 days ago
Yup - you have the right link - sorry about that - I have updated it now.  Should work with the IP address too.

Re the perfdata, is it an option to go back and take another zenbackup including the perf data?

In practise, this is just a bunch of subdirectories and files so you ought to be able to transport them just with tar, provided you ensure the structure and the directory / file ownnership / permissions.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------



20.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 9 days ago
Well, I can definitely take another backup and include the perf data this time but what would restoring a second set of devices over the current set in the new server do to the database?  Is there a way to empty the devices from the new server first?  Is it as simple as deleting all the devices except localhost?

And just to be clear, I backed up using the no-perf-data parameter.  I wasn't sure if you thought I backed up everything and then tried to restore using no-perf-data.

I tried zipping the Devices+subfolders from the old server and restoring it on the new server.  The sub-folders group, owner and permissions all look the same but I still get the "Report to Zenoss" error if I try to pull up a particular device.  Also, the counts don't match up.  My old server Devices sub-folder has 2140 sub-folders while the one on the new server has 1688.

As always, thanks again for all your help.

------------------------------
Clayton
Network Computer Technician
------------------------------



21.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 8 days ago
Hmm - bit weird.  If you zipped up the old $ZENHOME/perf/Devices and restored on the new build then you ought to end up with the same set of stuff that you should have if you had done the backup including the perfdata.  Can you pinpoint what is missing?

From memory, if you do a zenbackup including the perfdata (and the zodb as before), then a restore over the top of what you have now should be OK (though I would definitely have a backed up / snapshotted recovery position.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------



22.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 8 days ago
I didn't look too close but I'm thinking I didn't zip the empty sub-folders.

I'm taking a new backup.  "zenbackup --no-eventsdb --no-save-mysql-access".

And it failed with:
2018-01-09 14:05:56,008 CRITICAL zenbackup: 'Performance Data backup failed.'
2018-01-09 14:05:56,008 CRITICAL zenbackup: 'Backup packaging failed.'

And now I remember that I left out the perf data before because of the 32-bit to 64-bit issue but I'm wondering what's the point of backing up without the perf data?  I mean, it looks like I got the list of devices moved from the old server to the new server but without the data?  Is there a way to get Zenoss to use the device list to go out and "discover" them?

Taking a new backup.  "zenbackup --no-eventsdb --no-perfdata --no-save-mysql-access".

I'm also zipping up the Devices folders from the old server.  "tar czvf devices.tgz /opt/zenoss/perf/*".  I wasn't sure if I should grab just Devices or should I include the other files/folders from perf.  Now I'm thinking just Devices.

I took a closer look at the two Devices folders and it just looks like there are folders missing from the new server.  Weirdly, I can't find the original tar command on the old server to see what I did wrong.

So I did the tar -czvf on on the old server and the tar -xzvf on the new server and the totals are closer.

old_server# ls -la /opt/zenoss/perf/Devices
total 1692

new_server# ls -la /opt/zenoss/perf/Devices
total 1684

And I'm still getting the Zenoss error on the new server when I try to access device details.







------------------------------
Clayton
Network Computer Technician
------------------------------



23.  RE: Zenoss 4.2.5 auto deploy script feedback

Posted 2 days ago
Not quite sure what is failing now?  Is it looking at performance data for a device?

You are correct that there is an issue if you are moving from a 32-bit to a 64-bit implementation.  Forgotten the details but fundamentally, as you say, you can't see the data - subtly different format so either tar'ing up the old files or using zenbackup with perfdata, will not work.  I vaguely remember a discussion about there being rrd export / import tools that might get around this but I'm afraid I never tried them.

Many people who go through the sort of rebuild process you are doing, decide that they will move the devices but abandon the old events and perf data.  I'm sorry, but if you have gone from 32-bit to 64-bit then I have a nasty feeling that you won't get your perf data.

Can you actually see the devices and their data (other than the perf data)?  I believe that the 32/64 bit issue was only with rrd data and that your ZODB database ought to be portable.  The other technique I have used with several customers is to use the zenbatchdump / zenbatchload facilities to move devices to a new system.  Use zenbatchdump -h for usage.  Basically you can export your ZODB database, either en masse, or in bits - I often use device class as a parameter to move a bit at a time.  It produces a text file  which you can then inspect, edit, and prune devices that you don't want to import.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------