Service Interruption Aug 24-25: What Happened, and What We Did About It

The short version: On Monday evening, August 24, the data center that houses one of our hosting servers began a major infrastructure migration — without telling us or anyone else first. Websites and email hosted on that server went down Monday night, and even after they came back, many visitors (depending on their internet provider) had trouble reaching sites through Tuesday midday. Everything has been fully restored, and no email or data was lost.

What you may have noticed

• Websites unavailable Monday evening (roughly 9:00 PM to 12:30 AM)
• Sites loading for some people but not others through Tuesday morning — working on one device, failing on another
• Email arriving in bunches or later than expected

What actually happened

The data center moved our server onto new equipment. Their new network had a hidden defect: it silently discarded any piece of internet traffic larger than a certain size. Small exchanges worked — which is why the problem hid from normal monitoring — but the bigger pieces that real web pages and emails are made of were being thrown away for much of the internet. That is why the trouble seemed random: it depended on how each visitor’s internet provider packaged its traffic.

What we did

• Monitored and stabilized the server through Monday night’s migration, confirming no data was harmed
• Traced the “works for some, not others” mystery to the data center’s network defect using packet-level analysis and testing from 10+ locations worldwide
• Applied a fix Tuesday afternoon that makes our server communicate in sizes their network handles — within one minute, testing went from failing in 10 of 10 world regions to passing in all 10
• Audited email health end to end afterward: no messages lost, delayed mail caught up automatically, and our sending reputation checks out clean
• Filed a formal incident report with the data center demanding a permanent fix and accountability for the unannounced work

Where things stand

All websites and email on the server are operating normally and are reachable worldwide. We continue to hold the data center accountable for a permanent network repair on their side, and we have monitoring in place to catch any recurrence immediately.

If you saw anything odd during this window — a message that seems to have never arrived, or anything still not behaving — reach out and we will chase it down: info@weldonpc.com or 888-7-WELDON.

— Weldon Hastings, WeldonPC.com