- Service: Technology
The Challenge: Figuring Out What Hit... and How to Restore Operations ASAP
The client—a mortgage company with hundreds of employees—called at 10 p.m. on a Monday night. Something was wrong. The company could not access critical systems and data, including on-premise email servers.
West Monroe immediately assessed the situation, with the initial analysis revealing a number of encrypted files and a ransom note requesting bitcoin payment in exchange for releasing the encrypted files.
Further investigation the following morning revealed widespread penetration, with significant file encryption, deleted back-up files, and critical server outages. In short, the company’s entire on-premise environment was unavailable, disrupting any operations not running on cloud-based services. The impact would affect not only the company’s own customers, who were in the process of settling real estate purchase transactions; it could also have downstream impacts for sellers whose own real estate transactions depended on those settlements to occur on schedule.
While initial concerns centered on “ransomware” style attacks, in which computers and their contents are compromised with harmful files, the investigation revealed a deeper issue. The initial compromise was not the direct result of ransomware; rather, the attackers first infiltrated the company’s network through an unsecured server that had remote desktop protocol (RDP) services exposed to the internet and then gained the credentials for a domain administrator account. From there, the attackers were able to delete backups, encrypt data, and pivot throughout the environment.
The Approach: Mobilizing the Right Blend of Business and Technology Expertise
West Monroe assembled an investigation and recovery team with expertise in key disciplines. In addition to cybersecurity proficiency, the team brought essential skills in infrastructure, desktop application management, cloud services, advanced analytics, and business continuity. This team worked with the company 24/7 to lock down and stabilize the environment and then restore it back to basic operation—and beyond. In all, more than 45 West Monroe professionals participated over a two-week period during the holidays to help the company regain full business services.
The Solution: Rejecting the Ransom and then Immediately Starting to Rebuild the Environment
The ransom request was simple: one bitcoin per affected machine or four bitcoins for all systems. The attackers even offered a concession: Pay them half (two bitcoins) and receive a random sampling of the decryption keys to prove they were serious. The company had seven days to complete the transaction. With the value of a bitcoin jumping from a little under $15,000 per unit on the day of the attack to its all-time high of just under $20,000 per unit during the ransom period, the company was looking at a cost of over $90,000, including exchange and brokerage fees.
While the amount of ransom was relatively trivial compared the cost of rebuilding the environment, a lack of centralized logging meant the organization could not identify all activities undertaken by the attackers prior to the day of the attack and, therefore, could not guarantee the future security of its IT environment. With advice from legal counsel and West Monroe, the company’s executive team chose not to give in to the demands and pay the ransom, concerned that once paid, the attackers would see the organization as an “easy mark” and use undiscovered backdoors to relaunch an attack.
Instead, West Monroe immediately began planning to rebuild the environment, working closely with the company’s leadership to determine priorities for restoring operations—for example, which data to scrub, documents to recover, and servers and workstations to restore first.
By Thursday morning, less than three days after the first notice, West Monroe had identified the source of the attack, isolated the systems, and begun planning the recovery process, which included:
- Segmenting network services to create a clean network and isolate potentially infected machines
- Restoring basic infrastructure services such as authentication through Active Directory
- Deploying tools to recover deleted back-up data
- Considering approaches for recovering 765 workstations that could no longer be trusted as unaffected by the attack
Executing a "Big-Bang" Rebuild
As Friday began, the team convened in the company’s boardroom and floated the idea of undertaking a “bigbang” rebuild effort over the weekend. The challenge was that the company’s workstation environment included a mix of Windows 7, 8, and 10, with three versions of Microsoft Office. Worse yet, desktop base images had been deleted as part of the attack.
Over the next eight hours, the team devised a plan to rebuild and standardize the platform on Windows 10 and Microsoft Office 365 with enhanced security configurations. West Monroe’s Microsoft Desktop team leader confirmed availability of a previously designed automated build environment (nicknamed “ABE”) solution, although it would require some adjustments. However, this solution would require sufficient hardware to stage any data downloaded from workstations as they were imaged, and the company did not have any such hardware to spare.
By mid-day Friday, with the technical aspects of the plan looking feasible, West Monroe sent out a call for resources. It was December 18—the office was set to shut down for the holidays, and people already were leaving for planned time off, yet 36 people from West Monroe’s Chicago technology team answered the call to assist with the weekend rebuilding activity. The mortgage company’s legal counsel also volunteered his college-aged sons to pitch in. They called it their “first internship.”
Late Friday evening, a West Monroe consultant collected hard disks and server platforms from retail stores across Chicago. Over the next 12 hours, a small team built two ABE platforms and tested and provisioned the environment. At 5:00 a.m. Saturday morning, they delivered the platforms to two company sites, ready for the rebuild effort to begin. West Monroe teams then backed up user data, re-imaged and then re-deployed nearly 450 high-priority workstations, and restored the company’s critical operating services.
The core team left the company’s offices at 5:00 a.m. Sunday morning. In a little under one week since the issue was identified, the team had delivered 25 man weeks of effort, restoring the organization’s critical services and limiting the fallout from the breach and ransomware attack.
The remaining workstations were spread across 25 sites, ranging from 1 to 35 users. With the holidays fast approaching, the team conducted strategic site visits during the following week to deploy additional machines. By the end of that week, it had completed 600 of 765 workstations and had a plan in place to reimage the remainder.
The Impact: Fast Recovery, Limited Revenue Loss, and an Upgraded Environment Going Forward
With West Monroe’s support, the company restored full operations within days, enabling it to avoid potential operating losses estimated in the hundreds of thousands of dollars per day.
Not only did the recovery effort restore operations quickly; it put the company ahead by completing planned upgrades and standardizing workstations with the latest technology. This will reduce future IT costs—an unexpected byproduct of the opportunity to upgrade all workstations and end-user devices in one pass. By standardizing workstations and platforms, the company has also improved its security posture.
With the initial rush to restore business operations complete, West Monroe’s Cybersecurity team continues to work with the company to focus investment and plans on avoiding this costly problem in the future. Key projects have included implementing multifactor authentication for remote access, deploying privileged account management for domain administrator accounts, and segmenting the network to limit traversal and support modular survivability— all previously planned projects that had been deferred in the company’s 2018 budget.