Information Technology Solutions
Tel: +1-760-805-0409

Lessons from the Cyber-War Front

An ounce of prevention? It’s worth a great deal when it comes to your corporate information systems.

Old adages stick around for a reason; they are usually filled with deep common sense. So this one is particularly fitting for some experiences I had over the past few weeks, that I am compelled to share, but first some background, to set the context.

We are in the IT business, servicing our clients by providing experienced technical ninjas, at competitive pricing and featuring new and innovative ways to meet IT challenges. These include helping our clients increase revenue, avoiding and reducing costs and minimizing risk to achieve business goals. We offer security and data protection solutions from leading vendors, managed services and we integrate technologies the clients have already invested in.

One of the services we offer is backup of systems, and as such we have some experience in helping recover lost or stolen data from backup systems, which can be at the customer site or on cloud.

An ounce of prevention is worth a pound of cure, 16:1

Now, back to the story, in the last month we have spent a lot of human brain cycles trying to get 3 “new” customers back into operation. The reason why I say human brain cycles is that I don’t think AI is going to be able to solve this trouble shooting problem any time soon – AI has no gut feel. It can be a big help, but it will be a while till an AI can match the creative energy of a team of experienced people engaged in a collaborative effort working on a timeline. So, a quick tip of the hat and kudos to the  leadership of the CIO’s involved and all the client, vendor and rescue team engineers hard work. This effort literally saved three companies and I am sharing these experiences to help anyone who wants to continue reading.

The three cases have one thing in common: they were totally avoidable situations. Granted we have no control over such things as rogue actors on the cyber stage, or tech disasters caused by nature or human error. However, we can control some things, like we can have systems and procedures in a well thought out DR Plan (Disaster Recovery Plan) to mitigate risk, and decrease RPTO (Recovery Point and Time Objectives). Patch management is critical.  IMHO in this discipline, is that all IT leaders consider prevention as “way cheaper” than the cure and is best to state your position to management to ask for budget – and probably keep your job.

Company A, I got a call at 7am on Thursday morning, when the tired CIO informed me he needed immediate IT support, since his financial application was “down”.  He indicated he and his IT person had been up all night, since without a place to enter orders and create invoices the business was stopped. He had been on manufacturer tech support, escalated tickets and all the basic checklist had been exhausted. We arranged a quick web conference call, and had technicians trouble-shooting with this customer for most of the day and half of the next day. It was almost a RPE – resume producing event – for the CIO; we worked alongside this customer and were able to restore the financial system. The CIO lost 2 full days of production 2 nights of sleep and lost credibility with management and the user base – say good-bye to performance bonus. We are now designing an implementing a remote fail over configuration that replicates with the primary server, a modern backup protection system and a DR Plan.

Company B, similar call at 9pm on a Monday night, except this client had lost files and systems had degraded to be non-functional. Users complained about slow or no response from critical business systems, files were corrupted and so were backups. As the client was in close proximity we were able to have 2 technicians on site by 8am the next day. It was unclear at the initial triage meeting what had happened, when it happened and what the extent of the damage was. We did not know what we did not know.

We spent the week and the long weekend and a couple of sleepless nights with the client, determined there had been a security breach at some time combined with a configuration error had caused file corruption. Since the backups were not configured correctly, they were also corrupted. What a conundrum the CIO found himself in facing an RPE, management on his back, and users fuming. The team of engineers of which we had as many as four at a time at this client site was able to restore operations. We are now designing an implementing a modern data protection environment, which includes controlled fail over and fail back of all business operations to cloud, should systems on-site experience degradations, outages or security breaches.

Company C, was recently exposed to ransom ware in the movie industry, now looking to protect media assets. Client files were hacked and the “wanna cry crypto locker” had put a stranglehold on the business. Since this customer did not want publicity, they acquiesced and paid the ransom. This was in bit coin, which took them a few days to get, because not everyone has 3,600 in bit coin ready to go. Luckily for this customer, the hackers released the lock on the files and they were able to restore files.

There is a major cleanup of systems required since a virus can stay on hard drives or in memory for years without detection, and so it must be cleaned and any configurations to system need to be changed to prevent the virus from “calling home” to reactivate any new unknown crypto locking scheme. A fine mess for our tech team to reconfigure, but they do delight in keeping corporate networks clean and performing optimally.

This was not a fire drill emergency as the previous two examples, but was enough of wake-up call. The IT Director called me and asked us to move the backup project into this quarter. We will be implementing a zero-day threat management system. We are now creating a local backup copy of all data on a separate media, and arranging to have all media files into cloud object storage with erasure coding. The benefit here is that the hackers may get into one of three geo-striped locations, but the data is unreadable but still available for production purposes in the other two locations.

Computer files have become the lifeblood of commerce. Protecting and securing them makes absolute sense as insurance for your business, no matter the size of the business. There are technical teams from your current vendor who, like us, can come in and help you on a “rescue” basis – but I agree with the old school adage – as in matters of personal health – prevention is better than cure.

The early bird gets the worm.

Getting up early, usually means getting an early start, getting a good seat on the commute train or if you live in LA, beating the traffic into the office!

In our context I would like to address how we help our customers with zero day threats. Worms are bad actors on computer systems in that they propagate to other systems unbeknown to any basic monitoring systems that may be in place. Worms, virus, ransom ware….it seems our vocabulary is growing faster each millisecond. Bots rule the Internet or so it would seem. We can’t seem to patch fast enough, systems that are on-site, and attackers are persistent and smart. This is where AI helps and we provide this capability.

Detecting and stopping the threats that matter and containing these threats in real time is absolutely a requirement to play on the Internet today. This means monitoring, quarantining and eliminating worms and all malware threats as early as possible.

A bird in the hand is worth two in the bush.

Untested backups are the two in the bush. When we go in and ask customers about their DR Plan, the answer is usually not good. They have copies on a remote server, or some-one copies files on to tape, and then takes them home, or users keep copies on their hard drives of data files only, and system files don’t get backed up. This is the Humpy Dumpty scenario – and putting Humpty back together again after the fall takes time. He does not look quite the same with all those broken eggshells glued together.

The bird in the hand is verified backup of the entire system – virtual and physical machine – and the data, so you are able to quickly resume business operation. One copy of the VM should be kept locally on a separate media platform and one copy kept on cloud to spin up as needed. For good measure, we recommend a periodic full backup into inexpensive online secure object storage, on multiple clouds based on value of data.

You can lead the horse to water, but you can’t make the horse drink.

This is the ‘I told you so’ adage, just doesn’t sound as condescending. IT guys are supposed to have backups. However some IT teams are so caught up on keeping the day-to-day operations going that they may not have time or staffing to make time to solidify the infrastructure and operational processes. For these folks, and because of these three recent incidents, we have started DaaSle Rescue Services for IT. This is an emergency response team that will help you get your systems back in operation, should you have a system failure or cyber attack. We will provide this on a best effort basis, and the service is subject to resource availability.

Failure to plan is planning to fail.

If you have read down this far, and are still not convinced that data protection and security should be the first priority as CIO, then I have one last adage for you – “pay now or pay later “(>16X?), the cost for rescue service for these 3 companies was not as high as it could have been. Seriously, they came close to having their business shut down entirely.

The best laid plans of mice and men.

No matter how carefully a project is planned, something may still go wrong with it. The IT leadership at each of these companies was able to stay focused and leveraged all of the resources that were available and opened up shop the following business day.   We were all quite lucky and being prepared helped. CIO leadership, dedication, co-ordination and timely response by the rescue team combined for the positive result, this time. Time now to sharpen the saw, and stay vigilant.

Water off a ducks back.

When cyber-attacked, then for the prepared IT guy, it’s like “water off a ducks back.” In modern data protection schemes, systems can automatically fail-over to alternate data centers or to our cloud. Bad guys get nothing, and IT guy keeps systems running.

Living and learning by these timeless adages is just plain common sense and applicable for us in IT. Getting your backups and security strategy created and implemented is best done immediately since “hope is not a strategy”.   To get ahead of the game contact me at this link,  we can get your hybrid cloud replicating, and that will be as close as you can get to 100% uptime.  Link to DaaSle Service Request Form.

Ok, so that’s all I got today.  Thanks for reading, and I hope you are feeling more prepared for the inevitable downtime challenges. Now, from the songwriter Ian Anderson of Jethro Tull fame, it’s time to go “skating away on the thin ice of a new day”.

 

Written by: James Mal, CEO  Daasle, Inc.

 

 

 

Comments are closed.

Latest Posts

DaaSle IT Services

REQUEST SERVICE NOW CLICK HERE

PROFESSIONAL IT SERVICES: DaaSle offers IT Security Solutions from leading vendors like IBM, Microsoft, VMware, Veeam, Quantum, Cisco, Dell and HP. Our experienced staff can provide you with Security and Risk Assessments, Remediation & Remote Monitoring. Our Network & Systems Engineers can help you with design, implementation, support and consulting services. We excel at providing high ROI infrastructure that sustains business agility and delivers Business Value.

Shop DaaSle for IT Infrastructure

SHOP DAASLE CLICK HERE

Enter here for the DaaSle eCommerce site, which features products from Enterprise grade vendors. We have several FREE TRIALS of leading edge business solutions available. We configure, build and provide our customers with an integrated solution that provides a sustainable platform to execute on the business vision.