BennyEast.Com/Blog The official blog of Kenny West

3-2-1 Rule

Never a dull moment in IT that's for sure.

So, this week we had a bit of a situation.  It's mostly resolved.  But things got a little tense for sure. Here's kind of a little overview...

A while back I wrote about the swiss cheese model.  Not a swiss, cheese model, that's a cheese model from Switzerland... Just the swiss cheese model.

It's when there's a catastrophic failure because of a perfect storm of small events on their own that wouldn't in any one event really cause any major disruption.  But put them all together?  You get what happens on this planet every once in a while.

Usually it makes headline news.  Usually it's all over the TV.  It's always the swiss cheese model.  If ONE event in that string of events had never occurred...

The event would have never happened.  Nothing would have been reported in the news because nothing would have happened.

Life is about a lot of events happening in a very specific order.  It's very complex.

Most of the time we want a lot of things to happen, to come together, in a specific chain of events, so that it produces something that we want to happen.  This could be anything really, from a social gathering, or making a movie, or watching a movie.  Or a work project, or a school project.

But from time to time lots of events come together to produce a catastrophic failure.

That's what we had this week at work.

So basically in short what happened was files on a shared network folder were encrypted by ransomware.  Then of course one of the backups backed up the new encrypted files, making a restore... impossible... well, at first.  This is where the 3-2-1 rule comes in.

So... How did it happen?  Well, the first thing to happen was that our enterprise antivirus and anti spam solution was up for it's renewal.  This happens, not every year, but the contract has a specific length time of a couple/few years, I think it's somewhere in the 3 to 5 year range but I could be wrong.  The contract was sent out and received and reviewed.  Some items weren't approved by the lawyers that looked it over on behalf of work.  So, they requested to change the contract.  Then, there was a back and forth making changes with the reseller of the antivirus and anti spam software.

While this was happening, the virus and spam software ran out of license.  It went into grace period mode.

Now, because the software is in a higher price range it requires many signatures for the purchase.

One of the approval signature people went on vacation.  Then another.  Then the paperwork got stuck in limbo somewhere between legal and financial and back to IT and back to purchasing... And it was in some kind of waiting/holding pattern.

The grace period ran out.

In comes spam email with attachment.  Doesn't get picked up by spam software.

User clicks on email and opens attachment.  Attachment then runs on machine as a macro word doc...  Doesn't get picked up by virus software.

So most users don't have admin rights.  Some users have them temporarily to run very specific tasks.  Some users have them all the time, but this is a VERY select few.

Programs can have complete access to a PC with admin rights, that's why most users should NOT have admin rights.  I don't think the user had admin rights at the time.  I think the attachment simply ran as a macro after asking the user to enable macros for it to run.

So it got through 2 lines of defense.  The third line of defense was that the user actually clicked on the attachment.  We tell users to be suspicious of attachments they don't recognize.  So third line of defense is social engineering.  Playing on human weakness to have the email worded so that it's something they might open.  This person opened it.

Well... any shares that user had access to... became ransomware files.  The program first changed local files before moving on to shares.  Now, not ALL files were affected.  We caught on fairly quick.  But all it takes is that one or two important files.

We cleaned up the machine after taking it offline.

But what about the files?  Still have to restore the files.

So the next part was when the backup ran and it copied all the files from the one share to the next. This is backup 1.  It's effectively a sync.  So if the first file share server goes down, there's an exact copy on a second server.

Restore from second server is no good.

Everyone is freaking out.  There's a sense that those encrypted files are basically lost.

Now it's time to check the next backup.  So, the next backup copies files, but doesn't delete.  This file server had both the encryption files and the old ones.

From there we were able to restore.  Just have to pick out the ones that aren't "good" and delete the rest.  Then make new backups.

Crazy stuff.  So what's 3-2-1?  Well you can read more here...

http://blog.trendmicro.com/trendlabs-security-intelligence/world-backup-day-the-3-2-1-rule/

At least 3 copies.  2 different formats.  1 of them off-site.

I recommend taking it further, make one of those where no files are overwritten.  It just keeps making new files but never deletes or overwrites.  As it piles up, every now and then just clear out old stuff.  Personally I say take one of the backups offline and move it to a new physical location, then bring a new backup online.  Cycle through backups through the year so that you have a new backup device every quarter.  Then you can do away with a backup after a year and reuse that device.  Or just buy a new one and destroy the old one.

It's up to you.  Some people still use tape, or disc media...

I mean you can use anything that stores data, network servers, cloud storage, portable hard drives, whatever.

Long story short, we survived the ransomware attack.

Well so we got our license renewed, pushed the paperwork through and restored all files.

The key to surviving a ransomware attack?  Don't have live syncs.  Don't have files that all replicate instantly.  Have something that lags.  Have multiple backups.  Have versioning.  Or have a backup that never replaces, it just keeps adding new anytime a change or file occurs.  If a file is deleted, it's still there on the backup because it's not syncing.

Back up the backup once a week.  Backup the backup backup once a month.  Keep one offsite.

Because there's nothing worse than losing your data for good.

An interesting side note though, now that everything is back... Everyone is like... SOOOO overjoyed to have it back.  It's interesting... So when you have something, you have it, and okay cool.  Then when you might lose it, and then get it back, somehow it becomes even more prized and valuable.

It's interesting.  When you thought something had the possibility of being lost and gone forever, and then you have it back.  It becomes like gold.

Personally I like the totally over the top backup routine.  Backup live on the fly to one drive.  Then back that up nightly.  Then back up the nightly one weekly.  Then backup the weekly one monthly.  Then backup the monthly one yearly.

Each new backup is a new drive.  So that's like 6 drives.  Okay, I'll admit I slack a little... I only have 4 backup devices.

It drives me nuts though how most people have things in one place.

"I have it all on this flash drive."  And that's the ONLY place.  No where else!  So if that flash drive fails, or is lost or stolen or broken.  That's it.

"I have all my photos on my phone."  And that's the ONLY place.  So, if your phone is... broken, lost, stolen... That's it.

"I keep it all in the cloud in one storage account."  So if that account gets hacked and all the stuff is deleted.  Most modern cloud storages now let you restore deleted items.  Dropbox has 30 days.  There is the possibility of that company going out of business or being bought.

"I have it on my laptop."  All it takes is one cup of coffee knocked over in just the right way, or wrong way.

Just be mindful.  Ask your IT department how they backup their servers.  Make sure they run tests and test the backups.

Well... That was my week at work.  Never a dull moment in IT.