28 September 2016

Make up of a legacy

Let’s assume that you know in and out of a system, all of its intricate dependencies and nuances, to the level that you can make any changes thrown at. When does this system become a legacy?

When seemingly simple changes take long time to implement, we often point our finger to the system and tell ourselves and others that we are dealing with legacy system. With the assumption that we made earlier we could safely discard our inability to implement all the necessary changes. We are descent developers and our customer knows that too. So why is that we all encounter such system, if it is nothing to do with our ability as a developer?.

It maybe something to do with our perception of complexity. What if those seemingly simple problems were in fact complex ones? Instead of reengineering legacy system, shouldn’t we simply tell ourselves and customer that we need to realign our view and understanding of the problem?. Could I convince everyone to have a go at hypnosis at work? Seriously though I think a legacy is made up of our inability to embody the perceived business process in our code as transparently as possible. When the perceived business process needs to be corrected then yes we might need to practice hypnosis to align our views between developer and customer.

When I first heard of domain driven design, it was at Guardian. I was working for Thoughtworks and I was told that Guardian codebase has been hugely influenced by domain driven design. I never heard of the term before and with the codebase I thought it’s mainly just an OO concept of object relationships and constraints. In fact I couldn’t really relate the editorial process that would have happened by team of journalists from the codebase. Almost 10 years later, I was at onthemarket.com, talking to one of agent supports after the first version of the rewrite. I realised that I’ve made a mistake. The perceived unit of processing were a batch of instructions in a branch rather than individual instructions. We often simply put away such difference as implementation detail and put ourselves and our users through an obscure abstraction which often leads us to be lost in complexity. This time I made the change and shortly afterward It turned out that by moving away from processing instruction independently to processing in batch of branch, the system could move away from a push model triggered by arrival of feeds to pull model from processing of a list of known branches. This also meant that contention from feeds that are repeatedly submitted in a short period of time were simply gone. The complexity of coordinating potentially concurrent modifications to a single instruction disappeared as well. It might have been a random sequence of events that lead into a realisation then to a simplification. However even if the simplification didn’t happen, having a common understanding of process would help our customers see complex problem as complex as it really is, preventing this mis-match of perceived complexity vs real complexity and accidental slippage into the world of legacy.

It maybe something to do with our tendency to ignore details behind an abstraction. What if those seemingly simple problems were made complex from hidden details ironically all done to deal with complex reality? Seemingly obvious facts to our customer aren’t taken advantage of by the system simply because of the level of abstraction that we decide or inherited to work with. It’s hard to get it right first time, and reorganising it to adjust takes a bit of effort. If we are simply shifting problem into another by ignoring the details then, maybe it’s a wrong abstraction.

A few more weeks gone by, I’ve noticed the system was lagging behind the volume of feeds. We were processing over 150,000 feeds per day each ranging from few kilobytes to 3 gigabytes. The partition key of feeds were the id of feed provider and because feed providers supply instructions for vastly different number of branches among them, the partition weren’t uniformly distributed. Basically bigger the feed provider the bigger the delay was. When I explained to one of our agent support why different branches had different amount of delay, I received a puzzled look. Surely we want to treat our big feed provider better? I told him I want to partition the feed a lot smaller, but random feed naming convention and vastly different feed upload patterns by different feed provider lead me to abstract it away instinctively and following process couldn’t take advantage of it. He gave me another puzzled look. Look he said, I can see this feed provider upload feeds like this and so on and so on. We often ignore such differences, even the one that’s blindly obvious to our customer to take advantage of, instead we limit ourselves by our own rule that our customer doesn’t understand. It turns out it was really simple problem and I made it complex by ironically trying to manage its complexity.

With enough time and thought exercise you will become a domain expert no doubt. Sometimes a system becomes a legacy when the knowledge embodied lags behind our own understanding of the domain. You wouldn’t call a system a legacy when its inner working is based on the understanding as good as you are aware of. It’s often that you will find those misalignments in the legacy system are from early decisions made in its development and never been updated to reflect new knowledge gained over time. We need to acknowledge that our development in the early stage will be different from the later hopefully stable and mature codebase. We will gain new knowledge that contradicts with earlier assumption, that might require rework of completed work which might feel like repeated regressions. However I believe the focus in the early days should be geared toward assisted learning of domain knowledge through iterative development, solidifying the understanding, not toward delivering features with high degree of certainty.

More often than preferred, I find myself working with developers whose sole focus is in technical excellence in absence of context. More often than not I find complete alien world, a sort of parallel universe in system that shows little resemblance of the world it is resident of. Sadly a lot more energy than necessary has been put into bridging the world of reality and world of bits. Over DRY to achieve nth level of reuse. Over abstraction to achieve obscured simplicity. We are so accustomed to this bizarre world of IT, that it’s obligatory to label our customer the one who doesn’t get it.

Sometimes we find this rare gem, stable team with intimate knowledge of a problem domain and everything just run smooth. Certainly I took it for granted in my previous places, and re learnt the significance only when it was gone. Our system can only be as good as our knowledge of the business. We need to look beyond our technical ability to code in vacuum, it will only lead us into yet another world of legacy. Worse still, I believe focusing purely on technical ability of an individual does disservice for our industry as whole, solidifying the image of IT resource and not a valued member of business.