art of documentation and communication the most talked about and the most forgotten

Let’s face the truth, hard-core geeks can often have trouble in communicating their thoughts and ideas. Good software craftsman who can communicate effectively are indeed a rarity. You quickly observe this as soon as you start punching in code at your very first desk right after school. A “battle-worn veteran” such as Oracle’s Eric Lowe has written about this topic over and over. There is no China Wall between failed and successful projects, there is a thin line which is only defended by effective dissemination of key information amongst engineers. Eric Lowe writes:

“I’ve seen other projects flounder because the members of the team didn’t understand or buy-into the mission, major deliverables or objectives weren’t clearly defined, team members didn’t agree on the requirements, or team members grew differing perceptions of what the final product would look like!

The impacts of ineffective written communications within a project often are magnified by conspiring factors such as geographically dispersed teams, rushed schedules (i.e. we don’t have the TIME to define WHAT we’re going to deliver!), and in extreme cases, recalcitrant team members!”

Nowadays the requirements, and even the high level concepts, are more free to evolve over time as the project evolves. In this kind of project, it is even more important to keep in mind that the key concepts and requirements need to be thoroughly documented and for the community to agree upon them. Design decisions are the politics of software design and if others don’t understand the reasoning behind a decision, they will be more inclined to disagree with the outcome. The first nail in the coffin is when seemingly unimportant details - such as how the requirements were gathered, how the decisions were made, what was before and what happened after - are not recorded. My wife once told me in “nursing documentation” you record the nursing process: Data gathered and what tools used to gather it, the hypothesis you have made based on that data, what intervention you did based on the hypothesis, and what was the measured outcome. But in our field, sadly stories tend to get forgotten and history is lost.

Documenting early constraints will help a smart team leverage them and overcome them as early as possible. I have seen over and over development teams trying to design-hack around backward compatibility of old crappy library or support protocols that must be dead at this day an age. Die IE8, Die! Sometimes removing support is the best way to solve a problem. Documenting these constraints, if we have a proven documented case that we need them, will help getting rid of them in the future. The details, the assumptions and justifications must be kept safe. It is critical to document the precise reason for why constraints are being kept. It will then be easier for us to reassess once conditions change.

Cloud is a different beast altogether. Public facing APIs usually get very well documented but internal details get forgotten. But things move fast, a lot faster. I’m not referring to inline documentation in this article at all. But I believe it’s a good practice to always imagine that all that is offered are a bunch of libraries with no source code. The only possibility here is extending them or their configurations. How could it be described to an engineer who is going to these libraries? I’m pretty sure a good tutorial-like story in pages of a coherent book, with a clean, well-documented interface would make any user happy. Sometimes it’s an egg and chicken question. Not enough doc = very busy or too many developers, while very busy people = not enough docs. Strange isn’t it? Somehow it feels like you have to go through Limbo, Lust, Gluttony, Greed, Anger, Heresy, Violence, Fraud and Treachery to get it done with all the love and peace it deserves.

I think a specialized documentation team, having team members embedded in dev teams who cover not only the public facing APIs, but also every internal technical detail, is a tremendously helpful model. Preferably few non-technical individuals amongst them would be helpful to maintain the balance of control between civilian and military. If developers can tell their story to someone with minimal technical knowledge, even helping them get a general idea of how a library can get extended, then we have guaranteed ourselves a smooth path. Of course during this process management should be careful that no animosity rises between docs and devs and make clear it is engineers’ responsibility to produce documentation and help the documentation team to turn it into the final product. And we should be most careful that no one feels obliged to say “Yes, I understand”…

We all need storytellers, someone who passionately writes our story along the way. Budapest in its Golden Years produced a ridiculous number of amazing intellectuals during a period of exceptional brilliance in the middle of the 20th century. But if you jumble it up, it would all sum up in Von Neumann that tells the story of that time. It was Beckenbauer’s Germany that made history even when they lost to Italy. Some people are just naturally good in telling a story. I believe in the world of software, it’s only Chekhov’s passionate Russian Realism that can honestly and honorably tell a story. Maybe instead of Descartes, we could look for emotional Spinoza, especially when it comes to something like a computer program that is hard on the surface and all too soft at heart. The story has to be told like a book, a solid, well-kept and ever evolving book. It’s only then that we have tamed these computer programs. But

“…if you tame me, then we shall need each other. To me, you will be unique in all the world. To you, I shall be unique in all the world… if you tame me, it will be as if the sun came to shine on my life. I shall know the sound of a step that will be different from all the others. Other steps send me hurrying back underneath the ground. Yours will call me, like music, out of my burrow.” [1]

It is quite scary though. It makes us reflect. “When you gaze long into an abyss, the abyss also gazes into you.” [2]