| Client Info | Local Newspaper | Daily newspaper production | +100k subscribers |
|---|---|
| Challenge | The printing press system unexpectedly failed, threatening the newspaper’s production during its peak selling days.
Note: They called more than five IT companies before coming to us, but none of them could figure out what was wrong with their system. |
| Solution | The build and print system for the newspaper used an older version of Microsoft SQL Server to process printing of the daily newspaper. The internal IT department did not have backups of the most recent database. They only had backups from a week prior which would have destroyed all the data entry. Acumen was able to go into the tables of the system and root out the corruption and thus get the newspaper back up and running. |
| Results | Newspaper production restored within few hours
Saved the company hundreds of thousands of dollars in losses in ad reimbursement and customer non-delivery reimbursement. Saved the newspapers brand which would have resulted in future losses of hundreds of thousands of dollars more. Saved the IT departments reputation and allowed them to overcome future similar issues. |
After five previous IT consulting firms failed to resolve their issue, a prominent journal manufacturing company reached out to us as their last hope.
Our team went to their facilities with a deep understanding of the urgency of their situation and our unwavering commitment to putting ourselves in their shoes. We arrived on site to assess the business software powering the newspaper’s printing presses and sprang into action. Our goal was clear: to restore stability and ensure their success. as soon as possible.
They already everything they needed to be produced for Friday, but they wouldn’t be able to print on Saturday or Sunday. They were especially concerned about Sunday because it was their busiest day of the week, with the highest sales peak.
The newspaper’s production and reputation were threatened, potentially resulting in significant financial losses and subscriber and advertiser losses. To avoid negative consequences, the situation required immediate attention and resolution.
Risk Assessment
During the assessment our disaster recovery team used data analysis tools to performed a risk assessment and determined that the database they were using was an older version of Microsoft SQL. We determined the patching and service pack level of the system. We assessed that there was no disaster recovery system in place. The backups of the database were over a week old, too old to restore with any significant savings of time and would not get the Saturday or Sunday paper printed.
We sat down with the internal IT team asking them over 50 questions. We went over how the system functions normally, what happened right before the incident and what had they tried after the incident. Using this information and our data analysis results we created a one time disaster recovery plan to get the data and system restored and created an ongoing disater recovery plan to prevent this from happening
in the future.
After we performed the risk assessment and the disaster recovery assesment we backed up the corrupt database and the entire system twice using special imaging and compression software to two different storage media and media types. We took one of the copies off site in case of a further disaster.
We then restored the latest backup to an alternative server system we brought over as a dummy test system. We rebuilt the entire newspaper printing system with the older data, service packs and patches and software.
Finally we painstakingly analyzed the latest data on the corrupt system picking out the corrupt records and withholding them whle moving good records, record by record to the dummy server.
After about 14 hours straight we were able to make the dummy system simulate a printing of the Saturday and Sunday paper without error with about 98% accuracy of ads and copy. 6 hours after that the Newspaper was able to figure out what was missing and recreate it from scratch typing in stories, uploading pictures and typesetting the missing 2%.
The newspaper printed their Saturday and Sunday paper from Acumen’s dummy server that weekend. Meanwhile we installed better new server subsystems with fresh hard drives and cloned the dummy server to the new printng press server. Furthermore we setup a disaster recovery plan with proactive testing and checkoff procedures that the Internal IT department learned and implemented quarterly.
This led to the Internal IT department understanding creating an recovery time objective (RTO) with an image based backup system that takes snapshots. They also increased their recovery point objective (RPO) from seven days to one hour thus vastly improving their ability to recovery from a point in time from a week ago to an hour ago.
Overcoming this disaster and investing to insulate themselves in the future, the local newspaper was lucky that Acumen had the Microsoft SQL database, triggers and transact SQL expertise and that we were available on such short notice after five other IT companies came up empty handed. Had it been another more obscure database it is possible the paper would have not printed that weekend. They are also fortunate that the damage to the database and its software patches was not more extreme.
Now that the company understands RTO, RPO and what it means to have lost data and how that affects the enitre organization, the leadership values running regular disaster recovery tests on a regular basis to ensure that the systems could recover from other possible events such as fire, flood, theft, terrorism and the like. They have invested in systems and training for the IT staff to be able to recover even worse scenarios than the one they had without the need for database and disaster recovery experts to intervene. They are much stronger and more resilient to all forms of attack and disaster in the future.
