A few days ago I decided to do a fairly routine pre–New Year operation and upgrade the OS on the servers used for Fullyst. The support period for the current OS version ends in March, and it is better to prepare in advance. What could possibly go wrong? Turns out, a lot.

The upgrade completed successfully, all services started and were running, but today I discovered that one of Fullyst's core features, the automatic anti–spam, had stopped marking messages as spam. It turned out that along with the OS version, some Python modules used in this subsystem were also upgraded, and of course backward compatibility was once again forgotten.

So for several days the bot was diligently trying to figure out whether a message was spam, but the only thing it was getting back from the anti–spam system was NULL.

I fixed the problem by rewriting the entire subsystem to work with the new module versions, and once again I was reminded that it is not enough to check that a service starts. You also have to verify that it actually behaves as expected.