Site icon TwistedSifter

The Ghost in the System: How a Simple Code Choice Made Back in the Day Created a Multi-Million Dollar Crisis for a Modern IT Team.

man working on a computer

Shutterstock

If you’re old enough to remember the Y2K craze, you’ll probably recall how frantic people were about what might happen when the calendar flipped from December 31, 1999 to January 1, 2000.

Folks were convinced that all our systems were going to crash, planes would drop out of the sky, and the world would be thrown into chaos.

Well, that didn’t happen…but the person who wrote this story had to deal with a Y2K hiccup more than two decades later.

Check out what they had to say about this unique work challenge.

A Y2K bug surfaced 26 years late today.

“I work in IT for a small regional hospital chain.

We have a LIS (a Laboratory Information System, basically a smart database) that was custom-made for us by a tiny external vendor back in the late 1980s, back when HL7 was brand new.

Over the years, that vendor ported it from whatever it was originally running on, to HP-UX in the early 1990s, and then to Linux in the late ’90s where it has remained ever since without a recompile (thanks for the don’t-break-userspace policy!).

This is pretty interesting…

External vendor is legally still around, but it’s shrunk into basically just being a solo operation consisting of the one now-elderly woman who actually wrote the bulk of the code back in the day doing consulting for her ancient systems.

Earlier today, while chasing an unrelated issue, I went to put in a test order ten years in the future (to avoid confusing it with anything actually happening soon). It failed with a generic error message.

I try a couple more times, fails.

I ask if anyone else is having trouble putting in orders, works fine. I put in a fake order for tomorrow, it works.

2030, it fails.

2027, it works.

I quickly binary search it down to January 1, 2028.

Stop me if you know your calendar trivia…

I trudge over to the physical LIS machine and look through the local logs. The LIS is complaining about an invalid date. I check the system date, and, 1998?

Weird. I change the date to 2026, hoping for it to just start working.

This was getting to be frustrating.

It does not help at all; actually, no orders are working now. Out of curiosity, I turn the clock back to 1980, try to put in an order, and it goes through… but by the time it crosses the HL7 wire to the EMR, it comes through as being from 2008.

I try a few other dates. 1975 becomes 2003, 1990 becomes 2018, 1998 becomes 2026 as was working before, 1999 becomes 2027, and 2000… breaks.

Ohhhh no.

We call up programmer lady, who after some reading of the old code, confirms our suspicions.

A ha!

The LIS was storing years as two digits, because disk and memory were that precious in the 1980s, and 2000 felt like a long way off.

As 2000 approached and we were still using the LIS, the other people at her once-company decided that updating the system to properly handle 4-digit years was too expensive, and so instead, decided that the proper fix was setting the clock back 28 years (because the calendar repeats exactly every 28 years, and they’d only need to hack in 4-digit year handling at the places where it communicates with other systems, to increment/decrement the year by 28.)

How weird!

So from the LIS’s perspective, 2000-2027 was 1971-1999 and everything was dandy.

Ten points to whoever guesses the fix first:

Programmer lady changed the increment to 56, and we set the date to 1970 and recompiled the software for the first time this millennium.”

Trending and Popular

If you enjoyed this story, check out this post about an IT employee who refuses to change his “perfect” software install because the hardware was mistakenly installed upside-down.
Read The Drama

Now let’s see what readers said about this.

This person weighed in.

Another reader shared their thoughts.

This individual spoke up.

And another person weighed in.

Well, that was interesting.

You just never know what kind of computer bugs you’ll run into if you work in IT.

Good thing they got this figured out!

Trending and Popular

If you enjoyed this story, check out this post about a woman who is happy to find roles reversed after HR tells her to quit if she’s not happy.
Read The Drama
Exit mobile version