mood octupus has gone code red

I put this mood octopus behind me months ago. I can turn it inside out (red) when I'm deeply unhappy. That hasn't happened once. Maybe more people should get one.

This was a fun week. On Monday I deployed something I've been working on for a few weeks. It was moderately complex, and the environment limited testing. But it worked, and it's accomplishing something objectively valuable. It's exhiliarating.

The Florida weather I miss is finally back this year. The sky turns dark and it rains every day.

Dangerous thinking in software development: I tested it. It works. There was this one time I tested it and it didn't work. I don't know what that is. But I tested it again and it works. It's good.

I'm amazed by how many times a problem seems unsolveable and I don't find the answer until I either tell someone that I'm at a dead end or I'm about to. There's something about that moment of giving up that makes me look one more time and look differently.

I chased a number of red herrings, particularly code for message lock renewal that pottentially accessed disposed objects. It was right there in front of me all along.

Later when the CancellationToken was deserialized, it contained references to kernel objects stored at memory addresses which were no longer accessible. (I had to learn what a "handle" is because I never cared before.)

The command was sent using SendLocal, which kept it in-process. But if for any reason it failed repeatedly, delayed retry was invoked, which sent the command via Azure Service Bus to be retried later.

I spent days trying to figure out why a .NET app running in Kubernetes repeatedly crashed with 139 and no errors in the logs. The cause was a CancellationToken added as a property to an NServiceBus command.

So they duplicated it within the project. The thing is, they managed to leave the old one in the request pipeline. The result: Two nearly identical pieces of middleware, one that succeeded and one that failed.

The code in question was HTTP middleware to add an authentication token. It was referenced in a Nuget package. At some point a developer had a need to change that middleware, but they couldn't change the original package.

The method is simple: try { do the thing log a distinct success message. } catch(Exception ex) { log the exception with a distinct message } There is no way the success and exception messages can both be logged. But they were. Would you like to know how?

When I deployed and ran it, the new log messages unambiguously indicated that the method had succeeded. There's also a separate log entry for an exception from that method. That was pre-existing. I didn't add it.

Imagine a line of code that throws an exception, seemingly because of bad config values when deployed. As a last resort, I added logging around the method to log the values it was using. (That's a rare act of desperation, not how I normally work.)

I just experienced a Schrodinger scenario in which code simultaneously worked and didn't work. It's one of the strangest things I've seen in 25 years, and that's a lot of strange things.