Documenting Service Interactions

This post is published in December 2015 after I re-read much of Simon Brow's stuff again and figured that, with Structurizr extraction, he is so much on the right way I can stop pondering about that topic - this post is a snapshot of my thinking back in July, which I never really finished due to a unconvincing conclusion.

I’ve actually read "Documenting Software Architecture". ATAM, V&B, DITA and ALM at least ring a bell in my memory. I’ve attended Jazz/RTC, Sparx EA and UML trainings. I can safely say that, in the last 5 years there has barely been a work day I did not use the Atlassian toolchain. Yet, to me it seems there is no truly elegant architecture documentation, let alone knowledge retention, techniques out there in custom software development land.

So why is architecture documentation still so bad?

You have the integrated toolchains on one hand, the legacy of Smalltalk, Lisp Machines, Jazz/RTC/Eclipse/Mylyn and Visual Studio. On the other hand the self-documenting-code and markup world, the legacy of Cobol, like Python, TDD/BDD, Javadoc, Swagger and so on. There are a few attempts to bring the two together, XText, Lighttable, Structure101, and GitHub for that matter, and a few rather academic visual/structural coding systems like Dataflow (e.g. vvvv). Human-understandable documentation, though, still feels like a 2nd class citizen in any larger scale software development ecosystem I am aware of.

The reasons are pretty clear: most documentation systems are either tied to the hip of their very language ecosystem, often as afterthought, and hence not being able to fulfil Polyglot Programming. Or, they have been invented to enforce a given top-down structure, either for waterfall development or as operational support, which is by definition not agile and breaks traceability. Both of these reasons make it tedious to work with documentation. It breaks the flow of feature development, and forces everyone to externalise knowledge into the least common denominator. Hence people named “Architects” or “Business Analysts” still create static documents in more or less nice templates and models, which are outdated the moment they are published, because "the code is the ultimate truth" (unless you do MDD).

That’s a big problem in a digital world. Because software doesn’t just copy real-world business processes, and captures them in a box, anymore. It creates new processes, flows unseen before, that are constantly modified and updated through feedback, until no human understands them anymore. We turn our software into deep learning, minus the human understanding.

Documenting Service Interactions            

At Dockercon, Adrian Cockcroft gave a nice talk about Microservices and how to document the architectural implications. It sounded a little bit like Service/Process Mining, a backwards-looking approach to architecture, more feature/vertical slice/cell rather than layer-based. As such, for me it would fall in the markup bracket from above, a fancier version of Developer Portals in API Management. What I still miss, though, is structure, the levels of abstraction. Not in the UML sense of meta levels, but in the sense of a lean take on views. Without structure, I hate to say it, static documents will never die, and code and architectural documentation will stay detached.

In order to reduce the overhead and eventual detach between code and documentation, we can look at non-architectural documentation, and see how the problem is solved here:

• Interface and developer documentation, how to use a service or library, can, and should, easily be generated out of code itself (Clean Code meets Convention over Configuration), and versioned directly with it.

• Traceability to stories can be linked easily, e.g. per check-in. In a post-GitHub world, we must assume this is standard practice.

• Tests, as behaviour documentation, what is called a specification in BDD, should be versioned and documented with the code, that means aligned to stories and features. Personally speaking I don't think a test can ever cover a real requirement (see below), but it can to certain degree specify a behaviour.

• Requirements traceability, i.e. feature documentation, is a tricky one, because it typically is orthogonal to stories. This is a vast, separate topic, think MDA, DOORS and Sophist.

• End-User documentation should be driven by feature documentation, and hence falls in the same area.

• Integration architecture and dependencies are a bit harder, typically EIP e.g. in IBM Integration Toolkit are used. A bit more lightweight, we could use Adrian’s approach from above. But right now I am not aware of an integrated, structured way to feed back monitoring information into developer documentation (like Hystrix for instance).

• Operations and maintenance documentation is another tougher step. Even if monitoring on service level is achieved, the impact of infrastructure or database issues is rarely documented and formalised into post mortems that can be simulated. DevOps is a cultural solution, but requires very experienced engineers with tacit knowledge.

• Information architecture would be a next step, a third dimension to story and requirement, the data view. This is, today, the weakest link.

Aesthetics of Architecture

When documenting architecture, one dives into the aesthetics of code - why something in the world was perceived in a way it deemed necessary to break into into parts of a machine. Which immaterial (as in non-functional) values shaped the design of this machine over time. And why those values were agreed in the first place by the group of authors. Hence, architecture documentation has to become a part of the beauty of code. 

In building architecture, plans have evolved from models and experiments, towards guidelines and procedures. I have mentioned Alberti earlier, also in the first chapter of my book, the inventor of the modern plan as original artwork. In recent years, building architecture has gone back to a much more integrated practise, with plans showing values and ideas rather than processes and concepts. This style has been made popular by Atelier Bow Wow. It enriches architectural section drawings, pushing them almost in the direction of perspective competition drawings*.

Detail from "Graphic Anatomy", Atelier Bow-Wow 2007, Toto Publishers

The myth of the right proportion

Recently, I've been to the Venice Biennale. In one of the exhibition catalogues, Pro Portio, this remarkable passage about the usage of proportions caught my eye:

proportion is [...] investigation of how elements and patterns are connected and interconnected across disciplines [...] universal proportions guide our understanding of creation and the dynamic dance between order and chaos

The art of connecting the different views and perspectives of an architecture, understanding the values that make up our perception of a "good architecture", that's proportion. It's putting the elements of the machine into the right relation to each other. It's the grid in responsive design, the harmony of Pattern Languages, the Fibonacci sequence in planning, Amdahl's law in scalability and, after all, finding the right proportion between documentation and code.

After Venice, we took the chance and drove up to a monument of proportion: Palladio's Villa "La Rotunda", which Colin Rowe called "the ideal villa". It is an example of perfect harmony, as Rowe writes in Architectural Review:

It was not in fact suggested that architectural proportions derived from musical harmonies, but rather that the laws of proportion were established mathematically and universally diffused. The Platonic and Pythagorean universe was compounded of the simpler relationships of numbers, and such a world was formed within the triangle made by the square and cube of the numbers 1, 2, 3

While it stands as an ideal of renaissance proportion, there is an issue: Inside it's absolutely hideous. It's a monument that proves proportions, or patterns, or a grid, can never be the sole source of beauty.

Which might be the reason taking pictures inside is not allowed. For us in software architecture it means while we need to follow a plan, and maybe even make a plan, we need to use proportions as a tool to understand why we made decisions to have relationships established. A 3-tier architecture is not better because it adheres to Fibonacci's sequence or the triangle made by the square. Gregor Hohpe has called this "paying more attention to the lines than to the boxes", or Stefan Tilkov the "Cross-System properties" to care for.

In a wonderful talk, Alvaro Videla hunts down some common myths (a little bit in Bret style). He argues the reason why we still believe them is because they "are easy to repeat and to follow, than to actually learning the whole methodology". Grids, patterns, number laws and thumb rules are myths, as one easily takes ratios for silver bullets instead of guidelines. An example of a myth from the talk is "Miller's Law", the thumb rule of number of elements a human can put in context. Over time, humans have adapted, and the rules need to be changed. It's these changes, and the rules, which is the most important part of documentation. The tradeoffs and their interaction, the timeline, and the change in context and understanding.

I have written about Choice Architecture before. I firmly believe an Architect is a gardener, and as such she needs to work with the seasons, with time, with taste and the weather, to allow emergence (or evolution if you want) of beauty of the plot. Yet it's true, choice needs to be limited, and we require categories and guidelines to make decisions. Taking those decisions, and being able to explain them to others, so they can make them on their plot, is the art of our profession.

*) Architectural drawing style is a wide field, see e.g. Blomfield