Blog Marcina Bojko

„Doing backups is a must- restoring backups is a hell”
So how can we make this ‚unpleasant’ part more civilised? Let’s say – our VM is a couple hundred GB in size, maybe even a couple TB and for some reason, we have to do some magic: restore, migrate, transfer – whatever is desired a the moment.

But in oppositon to our will – physics is quite unforgiving in that matter – restoring takes time. Even if we have bunch of speed-friendly SSD arrays, 10G/40G network at our disposal, still few hours without their favourite data can be ‚no-go’ for our friends from „bussiness side”.
In this particular case, Veeam Instant Recovery comes into rescue.
How does it works?
It uses quite interesting quirk – in reality, the time you need for having your VM ready is to restore small, kb-sized VM configuration, create some sparse files for drives, and mount the drives itself over a network. This way, within 3-5 minutes your machine is up and ready

But, disks, you say, disks! „Where are my data, my precious data?”.
They are still safe, protected in your backup repository, mounted here as ReadOnly volumes.

So, after an initial deploy our VM is ready to serve requests. Also in Veeam B&R Console tasks is waiting for you to choose: begin migration from repositories into real server, or just to power it off.

Phase 1 during restore:

• IO Read is served from mounted (RO) disk from backup repisotory
• IO Write from client is server into new Drive snapshot (or written directly into sparse Disk_1 file.
• In background, data from backup repository are being moved into Disk_1_Sparse file.
  
  
  
  
  

Phase 2 to begins when there are no data left to move from backup repository – this initiates merging phase, when ‚snapshot’ (CHANGED DATA) are being merged with originally restored DISK_1.

As with everything, there are few PROS and CONS

PROS:

• machine is available withing a few minutes, regardles of its size
• during InstantRecovery restore, VM is alive and can (slowly) process all requests

CONS

• restoring can take a little longer then real ‚offline’ restore

• space needed during restore proces can be as much as twice your VM size. If you’ll ran out of space during restore, process will fail and you’ll loose all new data.
• your data are served with some delay – to read a needed block, VM have to reach it from repository, which means on-the-fly deduplication and decompression
• if your VM is under heavy use, especially on IO Write, restoring can take much longer then anticipated, as there will be no IO left to serve read and transfer from original disk.
• if your restore will fail from ANY reason – your data are inconsistent. You’ll have to repeat this process from point 0 – any data changed in previous restore attempt, will be discarded.

So which targets should be ideal for this process?

Any VM which doesn’t change data much but needs to be restored within few minutes:

• frontend servers

Any VM which offers small-to-medium usage of IO Read , not so many IO Write

• fileservers

Which targets should we avoid?

• database servers
• cluster role members/nodes

As we may know – Microsoft introduced new way of doing snapshots/checkpoints in Hyper-V 2016. However term „production” is misleading, implying Standard checkpoints are not production ready – which is simply not true.
The biggest difference is that Production checkpoints are mostly used with VSS-aware applications (like MS SQL/Exchange, MS Windows itself) allowing them to flush/sync/commit changes to filesystem.

As a major difference – production checkpoints don’t save memory or cpu state, starting always with machine powered off after restore.

You can choose which way you want to do your snapshots here:

👁 Selection_999(411)

Windows-based virtual machines have supported this since previous versions of integration services (2012 R2, 8/8.1) and from the start in case of Windows 2016/10. What about Linux-based, Centos 6/7 machines?

When installed out of the box, without any additional packages, trying to do a production snapshot of Centos 7 (with all updates) we got something like this:

👁 Selection_999(410)

Quick how-to.

1. If youre using external LIS (Linux Integration Services) from Microsoft, as an external package – remove it. It’s a piece of crap,  breaking kernels from time to time, packed with ‚latest’ errors and workaround rejected by linux kernel maintainers. It’s really not worth risk to have it installed: 

   yum remove microsoft-hyper-v kmod-microsoft-hyper-v

   or

   yum remove $(yum list installed|grep microsoft)

2. Check if your Hyper-V offers all Integration Services for this VM.👁 Selection_999(412)
   
3. Check  and install hyperv-daemons

    yum info hyperv-daemons

   Available Packages
   Name : hyperv-daemons
   Arch : x86_64
   Version : 0
   Release : 0.29.20160216git.el7
   Size : 4.5 k
   Repo : base/7/x86_64
   Summary : HyperV daemons suite
   URL : http://www.kernel.org
   Licence : GPLv2
   Description : Suite of daemons that are needed when Linux guest : is running on Windows Host with HyperV

   yum install hyperv-daemons -y

4. Enable and start services

   systemctl enable hypervfcopyd
   systemctl enable hypervkvpd
   systemctl enable hypervvssd
   
   systemctl start hypervkvpd 
   systemctl start hypervvssd 
   systemctl start hypervfcopyd

5. Check status

   [root@centos7 ~]# systemctl status hypervkvpd
   ● hypervkvpd.service - Hyper-V KVP daemon
    Loaded: loaded (/usr/lib/systemd/system/hypervkvpd.service; static; vendor preset: enabled)
    Active: active (running) since Wed 2017-07-26 02:37:30 CDT; 14s ago
    Main PID: 3478 (hypervkvpd)
    CGroup: /system.slice/hypervkvpd.service
    └─3478 /usr/sbin/hypervkvpd -n
   
   Jul 26 02:37:30 centos7 systemd[1]: Started Hyper-V KVP daemon.
   Jul 26 02:37:30 centos7 systemd[1]: Starting Hyper-V KVP daemon...
   Jul 26 02:37:30 centos7 KVP[3478]: KVP starting; pid is:3478
   Jul 26 02:37:30 centos7 KVP[3478]: KVP LIC Version: 3.1
   [root@centos7 ~]# systemctl status hypervvssd
   ● hypervvssd.service - Hyper-V VSS daemon
    Loaded: loaded (/usr/lib/systemd/system/hypervvssd.service; static; vendor preset: enabled)
    Active: active (running) since Wed 2017-07-26 02:37:30 CDT; 27s ago
    Main PID: 3485 (hypervvssd)
    CGroup: /system.slice/hypervvssd.service
    └─3485 /usr/sbin/hypervvssd -n
   
   Jul 26 02:37:30 centos7 systemd[1]: Started Hyper-V VSS daemon.
   Jul 26 02:37:30 centos7 systemd[1]: Starting Hyper-V VSS daemon...
   Jul 26 02:37:30 centos7 hypervvssd[3485]: Hyper-V VSS: VSS starting; pid is:3485
   Jul 26 02:37:30 centos7 hypervvssd[3485]: Hyper-V VSS: VSS: kernel module version: 129
   [root@centos7 ~]# systemctl status hypervfcopyd
   ● hypervfcopyd.service - Hyper-V FCOPY daemon
    Loaded: loaded (/usr/lib/systemd/system/hypervfcopyd.service; static; vendor preset: disabled)
    Active: active (running) since Wed 2017-07-26 02:37:30 CDT; 44s ago
    Main PID: 3492 (hypervfcopyd)
    CGroup: /system.slice/hypervfcopyd.service
    └─3492 /usr/sbin/hypervfcopyd -n
   
   Jul 26 02:37:30 centos7 systemd[1]: Started Hyper-V FCOPY daemon.
   Jul 26 02:37:30 centos7 systemd[1]: Starting Hyper-V FCOPY daemon...
   Jul 26 02:37:30 centos7 HV_FCOPY[3492]: starting; pid is:3492
   Jul 26 02:37:30 centos7 HV_FCOPY[3492]: kernel module version: 1

   As a result:
   👁 Selection_999(413)
   
   and in /var/log/messages

   Jul 26 02:43:27 centos7 journal: Hyper-V VSS: VSS: op=FREEZE: succeeded
   
   Jul 26 02:39:25 centos7 systemd: Time has been changed
   
   Jul 26 02:39:25 centos7 journal: Hyper-V VSS: VSS: op=THAW: succeeded

Yes, we all „love” the way Microsoft gives us new stuff (like forced updates in Windows 10). As we usually know better what and when should be applied, let’s say it is time to upgrade our guest integration services to the latest version.
I usually  express strong conviction that HV integration services should be kept up to date, the thing most sysadmins disregards. Those tools are responsible for keeping your VMs in shape, maintain communication between them and host services, so they SHOULD be always updates. Of course, as with almost everything TEST it before applying on any important environment.
As on 2016-12-31, the latest HV Integration Services version is 6.3.9600.18398

Let’s say – we would like to upgrade all our Windows machines (those BEFORE Windows 10 and Windows 2016) . We can do it manually (not cool, not cool at all), semi-automatic (no reboot) or almost … magical way – fully automatic 😉

Let’s start we have proper source added to Chocolatey:

choco source add -n=public -s"https://www.myget.org/F/public-choco" --priority=10

• semi automatic (RDP, psexec, powershell)

  choco install hvintegrationservices -y

• fully automatic (with reboot) using The Foreman and win_manage
  

  win_manage:
   chocolatey_packages:
   hvintegrationservices:
   ensure: 6.3.9600.18398
   dsc_reboot:
   dsc_reboot:
   message: Machine requested a reboot
   when: pending

On my lectures and meetings with both: IT and Management  I’ve had a pleasure to be a myth buster about Hyper-V. As much as I don’t appreciate Microsoft’s ‚way of life’ – Hyper-V is mostly feared due to: lack of proper knowledge and very low quality support from Microsoft. Few most common myths are:

1. It’s very expensive
   Let’s calculate:
   If you’re going to use standalone hosts with Microsoft Hyper-V Server your cost will be just zero coma zero.
   If you’re going to use a lots of Microsoft Windows virtual machines on them – you can rent them as SPLA licenses (per machine), or just rent Windows Datacenter edition for whole host.
   If you’re gonna to use Linux machines (assuming opensource, not paid edition) – again – zero coma zero
   If you’re going to use HA, all you need is just 1 (preferably more) OS for Domain Controller.
   If you’re gonna to manage standalone hosts, all you need (and rather as a suggestion) is a Microsoft Windows 10 Anniversary Edition machines. Just one 🙂
   You don’t have to pay extra for all fine features like: HA, Live Migration, Cluster Aware Updates. With W2k16 edition few extra features are available only in Datacenter edition (which I believe is a grave mistake) but that’s all.
2. It requires System Center to be managed by
   No. As a matter of fact, SC is only useful in situations when you have lots of VLANS, Logical Network, templates to be deployed or Services. In any other case like: have your VLANS’s accounted for, drop Services as nobody is using this part. System Center Virtual Machine Manager is nothing more than overgrown cancer on a top of Powershell scripts it runs. Since 2012 edition Microsoft couldn’t ever fix the simplest things like: responsive console and not having refreshed it manually after every operation.
3. It’s slower than VMWare or ‚any’ other competitor
   No. Overhead of Hyper-V is done mostly on storage level and most problems with it are created on a level of infrastructure design.
   For example: If you have a lot of hosts, and you do not require virtual machines there to be Highly Available – do not (i repeat) DO NOT connect them all as cluster nodes.
   If you’re using 1 or 2 iSCSI 1GB cards as you paths for low quality machines – expect nothing more than problems.
   Instead: use local storage, combined with low-end HW controllers. Even having 2 (mirror) or 4 (RAID5 or RAID10) disks for those machines is way better than having one underpowered ‚best of the world storage’. Plan this usage carefully – you still have things like Shared Nothing Live Migration (in a case of maintenance on specific host).
   Creating a lot of host, giving them all 1 or 2 ClusterSharedVolume to share is just asking for trouble.
4. It requires a lot’s of PowerShell knowledge
   No. And I am the best example here 😉 With few exceptions like script for installing Hyper-V hosts, maybe create few LACPs – that’s all I used Powershell for.
5. It doesn’t support Linux
   It does, it does it very well.
   Official document: https://technet.microsoft.com/en-us/windows-server-docs/compute/hyper-v/supported-linux-and-freebsd-virtual-machines-for-hyper-v-on-windows
   Few my lectures: https://marcinbojko.wordpress.com/2014/10/22/xxi-spotkanie-regionalnej-lubelskiej-grupy-microsoft-i-moj-wyklad-systemy-linux-na-platformie-hyper-v-2012/
   As a matter of fact – I use CentOS/RedHat, Ubuntu/Debian machines and appliances, and have to say: working with them on Hyper-V is just a simple pleasure.
   In 2016 with things like Hyper-V and Veeam, support for Linux machines on Hyper-V is very much alive. Even our beloved ‚System Center Virtual Machine Manager’ supports creating templates for Linux machines, with small agent to set a lots of things during and after deployment.
6. It’s complicated to install, run, maintain especially HA & Clusters
   No. It is just simple as click few times: next, next, next,finish.
   Using System Center or (better) FailoverCluster Manager from any Windows Server machine works perfectly out of the box. Rules are simple, wizard will tell you what you should do next.
   With maintenance mode, Live Storage Migration, Cluster Aware Updates you can have stable and secure environment for your machines. Even migrating machines between different clusters (Shared Nothing Migration) is secure and efficient.
7. It requires specific hardware 
   One of the biggest myths. Learn with a hard way with VMWare hosts , you do not require special NIC, special motherboards or any devices from very narrow VMWare HCL list. Requirements of Hyper-V are very small: VT enabled CPU, enough memory to fit VM’s and host OS itself, one HDD, one NIC. For small setups it almost equals in using desktops and other workstations as a hyper-v farm.
   After hearing this statement from one of my clients, I began to pursue the subject. It was someone from VMWare camp that told him: ‚you will need special hardware for SMB3 and SMB Direct’ – which is generally correct in a same matter like: ‚if you want a milk, you need a cow’ 😉
8. It doesn’t work with Azure
   Hyper-V 2016  is a light years ahead of Azure:) They still seems to be using Windows 2008 as a hosts with all of its negative aspects.
   But, jokes aside, using pre-build templates or products like Veeam and Windows Azure Pack, creating you own hybrid cloud is one of the best things you can do. Don’t trust sales guy from Microsoft forcing you to ‚move everything to a cloud, our cloud’. Don’t trust you IT guy saying ‚only on premise or death!’. Live in a both worlds.
9. I know NOTHING about Hyper-V.
   If you have ANY knowledge about Windows – you have knowledge about Hyper-V itself.
10. But migration from platform X/Y/Z is pain in the ….
    Take a deep breath. Calculate it. Find tools to do it manually, recreate all you machines using somekind of CM tool (like mentioned The Foreman/Puppet)- https://marcinbojko.wordpress.com/2016/10/04/puppet-the-foreman-powershell-dsc-your-system-center-in-a-box/. Calculate it again.
    Do it 😉