The DRY principle

I am a big believer in the DRY principle. But that’s not a substitute for good designs. I still believe a perfect design is amenable to a completely DRY implementation, but of course in reality one needs to deal with well established languages/packages, that do not work well together necessarily.

One instance of my attempt to enforce the DRY principle at my current company is to refactor their yaml configuration files, through something called deep import, that I learned from another popular configuration language hocon.

Here is how yaml supports the notion of inheritance. When you concatenate two yaml files, the later one overrides the earlier one, but only for the top level keys. So essentially a dictionary update is performed, in the python parlance. In practice, however, we work with not just a flat dictionary, but a nested dictionary. Then there is no way to perform a nested/deep update, where you preserve all the information in the earlier configuration, but only update the value at nested key path.

I suppose this kind of features is not taught at schools for those fresh graduates at my current company: it’s probably too trivial to be mentioned in any cs class. However without the ability to override in a nested fashion (i.e., beyond top level keys), there is no clean way to enforce DRY in building an army of configuration files, which is what’s being done at my work.

Now granted, yaml does offer another important builtin feature, namely anchors. Using anchors, one can give any piece of configuration block a name, and reuse it later in the same file. This can be viewed as the global variable approach in the programming world. But I guess there is a reason we have OOP and inheritance.

After some initial resistance and doubt, the team finally assimilated the idea and now it’s an integral part of the configuration ecosystem. The implementation is pretty straightforward: build a stack of imported yaml files, and apply deep update in a depth first manner. There are several catches though.

• Sometimes we want to override the deep imports locally: for instance, for a top level key, we do want to override its value in its entirety. This can be called an atomic override. Indeed that is the solution I came up with, back in my previous company already in the context of hocon: support an atomic override special key.
• Anchors and deep imports do not work well together, since anchors refer to an explicit block of configuration. With deep import however, it becomes ambiguous whether the block should include parent information. With shallow import this was never a problem.
• Deep import followed by shallow import is a recipe for unexpected behavior. Shallow import is implemented simply through concatenation of yaml content, whereas deep import converts yaml content to a json dictionary first. Thus once we start using deep import, we we are in the json land and farewell from the yaml land.
• List (or repeated field in proto parlance) does not support nested update. This is in contrast to proto merge_from utility, which treats list append as the analogue of dictionary update. What i did in my previous company was to implement a + operator in that situation, and later I also added support for the – operator. But in hind-sight I think that was a bit of overkill.
• The speed of parsing is now O(n^2) instead of O(n), where n is the number of files in the import stack. This is because the shallow import is simply concatenation followed by libyaml parsing, which is all in c++, whereas the stack approach needs to call libyaml n times. But in theory if we write everything in c, the difference shouldn’t be that big.

The yaml specific catches above I learned at my current job, but the majority of the language independent consideration already matured before. One conclusion I had at the end of my previous tenure was that it’s probably a good idea to get rid of configuration language entirely and use python script directly in their place, since python is so flexible and bug-free (at least as a configuration language), I didn’t need to worry about cross-lingual compatibility, or the need to add additional support by modifying the parser itself.

Now back to the general topic of DRY. I hear complaints on various forums that people often abuse it at the expense of poor design choices. To re-iterate my earlier point, in the perfect world, DRY is flawless, just like the universe is probably made up from a few simple axiomatic principles, at least according to the theoretical physicists. So I almost never get mad at people who try to enforce DRY, maybe because people I worked with in the past are usually good enough with design strategies, or that the bad ones haven’t even bothered looking into DRY yet. But I can understand how in certain work environment, less capable people are incentivized somehow to enforce DRY before other more important aspect of engineering. I could be one of those people in certain people’s eyes.

But recently I had one incident with my manager, who manifested the vice of DRY-zealotry in one of the most unlikely situations. During our 1-1, I mentioned, perhaps for the 5th time, that I am working on some project that involves the use of faiss, a popular tool open sourced by facebook that allows efficient vector similarity retrieval. Really it’s just a tool, and my goal is to do some clustering, some nearest neighbor retrieval, as a way of data mining/exploration. To me the magic of ANN is the most nontrivial way I can think of to do NLP related data mining.

From the start, my manager felt I should not invest my time in this area. And somehow whenever he hears the word faiss, his immediate reaction is that I am wasting my salary on things that he doesn’t think is important, or that someone else on the team is already working on. In his mind, there should be only one person in the entire team working on faiss. Everyone else either shouldn’t touch it, or should take the output of that one person as input. Now this is starting to sound like the DRY principle in engineering. Being an engineer for most of his life, I can sympathize with his motivation. He wants the whole team to be like an intricate piece of software, where every engineer plays the role of some library, who can call each other, without any redundancy, and completely under his master design and control.

So I was taken to this other colleague, who recently used the work done by the other colleague who has spent much time understanding the faiss library. At that point I wish I had the courage/decisiveness to tell him that I am not re-investing all the time to learn about the faiss package and compare its various setting. To me it’s just a tool, similar to python, jupyter, or numpy. I don’t think he would be averse to the idea of multiple people working with numpy directly. But somehow faiss is a mysterious thing that made him uncomfortable that more than one person is using it directly. In reality, I am using faiss only as an off the shelf batch matmul library, since I am only using the brute force mode of its retrieval functionality.

My manager’s reaction to take me to this other colleague, and question my judgment of not leveraging existing team members work, felt like a gross mistrust in my ability and leadership. Perhaps he is so busy these days with all the firefighting operations at our small company, that he forgets about some basic courtesies. In a similar earlier storyline, I was almost forced to merge my work on integrating some deep learning pipeline I built into an existing system of another team, though not initiated by my manager, but a member of that other team. I understand small companies want to be lean. But forced merging isn’t necessarily the right way to reduce redundancy. People design things in very different ways. A forcefully merged product may be twice as bad as the components separately. What I find works better is to have a shared library of common utilities, that both sides try to polish and maintain.

I also don’t mind merging my work with others, provided the two have very similar design taste in mind. I am all for convenience and DRY, but the other party is all about bloated configuration/pipelines. Even if we merge successfully, my workflow would be 5x more time-consuming. There was simply no motivation for me to pursue that direction unless I am drunk (which I have never been).

But I don’t blame the other party either. I know they are relatively fresh graduates, with lots of energy and no time commitment other than work. They also follow their managers’ directions very closely, as they should in a startup. My general feeling about the managers is that they are very adept at avoiding the commonly documented pitfalls, and follow the general principle of management very well. I honestly don’t know how to be better managers. What I do know is that their vision for how employees should fit into the grand design of the company spells poor wlb for the employees, even the smartest ones. To me that’s not sustainable, not only for the employees, but the company as well.

To conclude, I am still a big fan of being DRY. I think it’s a word that should be more popularized. I never anticipated people would apply it to people management as well. So while shocked and slightly depressed after such treatment, I am also amused by the nerdy-ness of my manager. At the end of our brief 1-1, he said that I should keep him in the loop. I guess that’s an important signal I missed earlier. Such signals are difficult to convey in a neutral manner, so I applaud his courage to let it out. I should do better to keep him informed, though I had 2 concerns: a. he usually appears very busy and I don’t feel the need to add to his workload; b. he is generally not encouraging on directions that does not align with his vision, yet at the same time he wants me to come up with plans to save a big part of the company. I think I did my best to stay productive while taking the appropriate risk of self-isolation.