So, one of the things I preach in my talks about ransomware is the need for “Weapons-Grade” backups. I want to talk a bit about what that means, and why it’s so important. This is not meant to be a complete guide to backups, but it is meant to get you to think a bit about the risk you are at with respect to your data. Further more, I’ll tell you how many of these concepts can be applied at home as well.
Why all the worry?
We all have a lot of things in life that are competing for our limited amount of time. In order to understand why we should dedicate some of that time to making sure we are backed up, we need to understand the risks being faced today. The top 4 things that increase my grey hair count are:
- Ransomware/malware that destroys or holds data hostage
- Hardware failure that results in loss of data
- Intentional or unintentional destruction or changes to data
- Physical theft of the data
You might notice a pattern here. All of them result in losing data. Not a big surprise given the topic. This is not an exhaustive list of how data can be lost, but it covers enough for this article. You should also be familiar with the 3-2-1 Rule before we go on.
Common Backup Methods and Pros/Cons:
- Copy to tape – Not usually used at home and often not in small businesses. This involves a tape backup drive and special magnetic data storage tapes to keep your data safe. In some cases, you can use the software built-in to the operating system to back up to tape, but often you will want some 3rd party software to help. Accessing individual files from tape is pretty slow compared to other modern storage devices, so typically it is used for long-term backups, or even backups of other backups (remember the 2 media rule) that have been made to a disk. Backing up to tape is a method that has saved a lot of tears from falling. Like anything else though, restoring from tape can fail, so it is important to test these regularly. Finally, tape backups are pretty easy to move offsite compared to some other methods.
- Copy files to another device – A lot of organizations have turned to backing up data to another computer or a Network Attached Storage (NAS) device, across the company network. You can do this with individual files or in backup sets, like you usually do with tape. When accessing individual files, this is usually much faster than tape, but is typically not as easy to store offsite. You can use external hard drives to do this is well and they are easier to move and store offsite than a NAS. It is very important if you are doing this, that you keep these files isolated from your regular network and test the ability to restore often. This can save them from being encrypted by ransomware that is network aware. A lot of people have found themselves in a bad place when their backups are found to be encrypted as well.
- Synchronizing/Replicating files – There are a number of cloud solutions out there that allow you to synchronize files. These include services such as Dropbox and One Drive and can have some. You can also use tools such as Robocopy, SyncToy and rsync locally. The cloud solutions are a good way to get files offsite in case of physical theft or destruction, however it is not foolproof. For one thing, many newer types of ransomware will look for these services and try to attack them as well as the local machine. Similarly, replication between sites is not the same as backing up. In this case if the file is infected or encrypted by ransomware at “Site A” and is replicated to “Site B”, that means that both copies of your files are infected or encrypted. Take for example THIS STORY where the Police Chief says, “Our automatic backup started after the infection, so it just backed up infected files”. That is a sign of replication as opposed to actually running backups.
Pitfalls and Fails
- Not checking the logs – I see a lot of admins that set up the backups, monitor them for a little while, and then stop watching logs. This is a recipe for much wailing and gnashing of teeth. If something goes wrong with your backups, alarm bell should sound, lights should flash, and pagers/smartphones should be going nuts. It’s really that important. If you get a lot of false alerts, you need to tune your alerts, but don’t tune them out.
- Not reviewing what is being backed up – I also see where backup jobs are set up but when new folders are added or the architecture changes, the backup jobs aren’t updated to include the changes. The result here is a lot of files and folders don’t get backed up. You need to review your folders and compare them to what you expect is being backed up on a regular basis. The more critical the data, the more often this needs to happen.
- Failing to test the ability to restore -More than one sysadmin, including myself, have felt the sinking feeling when backups fail to restore. If you haven’t experienced it, this is something really don’t want to experience. Although it takes time, it is vitally important to test your ability to restore files. Sometimes you can pick critical folders to test on but on occasion, maybe even monthly, I recommend that you restore the full backup set and ensure all of the files you expected are there.
- Not having enough space to restore – Something that folks often forget to look at is, do they have enough space to restore their files without deleting the old ones. This can be important when it comes to retaining the forensic evidence. If you follow the previous step and test your restores, you should already know if you have the space for this. One option is to move the old files to inexpensive external drives or other non-enterprise storage, so this really doesn’t have to be a financial burden.
- Backups are network accessible – I’ve heard of this happening several times where they have good backups, however they’re accessible on the network. What happens is the ransomware encrypts the backups as well, leaving these folks in a pickle. Make sure that any backups you have are not accessible on the network. Isolate them however you need to, for example, on a VLAN that only the backup server has access to. This can really save your day if you get a particularly nasty strain of ransomware.
Commercial backup software can get pretty expensive pretty quickly. Well I can’t specifically recommend any single solution because your needs may vary, it doesn’t hurt to look at options such as Code 42’s Crashplan. I use the free version of Crashplan at home to keep all of my stuff backed up. I like the fact that I can back up to a friend or families house and they can back up to me, and it’s all encrypted prior to transmission. In addition, it’s hard to beat free. Don’t discount the use of tools such as rsync, Robocopy, and Synctoy as well for replication of files or backups to other destinations.
If you follow these tips and tricks and you give your backups the attention they deserve, this can make your life a lot easier in the case of a ransomware infection.
Stay safe out there!