Letter to my future self

This is a variation on a letter to my future self, just a public one. A reminder what I thought, written on the last day in my old job - exactly 6 months ago. I would write it differently now, so I’ve added many current links. Don’t read it as a manifesto, read it as a snapshot.

When I quit my first full-time employment in a start-up, after my own one, it was because I felt start-ups disguised the struggles for power under a hood of coolness and artificial problems.

I joined consulting to be outside of power structures, and tackle real problems (as real as it gets in corporations). Accenture Digital was inspiring, and t-shaped/widened my horizons - more than once. It allowed me to thrive in technology, strategy, product prototyping, Agile, design thinking and to trust liberal arts. I will always remember the learnings in those years: All the opportunities I had to meet great, diverse, truly interesting people across the globe, to learn about dozens of domains, to build digital-first, mobile-first, offline-first products, omni-chanel SPA, Microservices with vertical slices, API management and EAI, or CQRS systems. In short I am grateful to have been inspired and had the chance to inspire others.

Really building stuff around existing problems is what makes technology consulting so amazing. Seeing what you’ve built in the wild, changing the way an industry runs. Building a team around a problem, changing their perspective, and mine. Unlearning is just as important a skill as learning. We are constantly developing, iterating, behaviours, patterns and principles of good systems, including its human actors. These principles always come with a drawback, outweigh by the advantages. Microservices might manifest team separation and can lead to incoherent style in end-user product experience. Event systems enforce eventual consistency, and not every organization is ready for probabilistic thinking. Containers can degrade security (and with it privacy) to a part of commerical opt-in dependency-hell rather than reliable infrastructure; Design Thinking either glosses over faster horses or fakes ownership because agencies just rebrand; Patterns sometimes promote solutions that only look good on paper; CI, Lean and Feedback prefer simplistic, biased, measurable KPI’s over long term goals. OSS, Agile and DevOps can establish a status-quo burn-out treadmill of minor, over-hyped, pseudo-improvements, and holocracy could build a cast system; Opinionated often just means tribal or elitist (for lack of a better word); competing API, AI and Cloud standards with unexplainable, magic automation and integration can cement vendor lock-in; Cross-vendor services are still not solved, maybe due to academic REST and DDD being inherently anti-evolution with their focus on a Xanadu-ideal, fixed state. And simplicity does never illustrate success, while complexity does. Not everyone wants to hear those drawbacks, and few openly make the tradeoffs. Most want to hear the innovative, the disruptive, which sometimes simply excludes the best option for the largest group.

In addition, the problem with the typical project organization is that it takes a lot of effort to not disguise debt with another layer but actually change the underlying problem (a post on this is coming up). I always enjoyed this challenge, and I had the pleasure of being on many projects which achieved this, and turned into real, strong products. But kickstarting this "digital transformation" every so often started to nag on my belief in actual progress.

An unfortunate side effect of the project and OSS “publishing” treadmill seems communal wisdom is often discouraged in favour of new marketing slang. Or, like Maciej puts it:

"This is not the first time an enthusiastic group of nerds has decided to treat the rest of the world as a science experiment. Earlier attempts to create a rationalist Utopia failed for interesting reasons, and since we bought those lessons at a great price, it would be a shame not to learn them"

Sometimes it feels through lean and measurability the Peter Principle and pop-culture power struggles in the form of Deutungshoheit have eventually arrived in what used to be an idealistic geek world of IT, pushing practitioners from actual consulting into meta- and consult-speak. If software indeed eats the world, it seems to prefer junk food. To avoid this trap I took a step back on conference talks and wrote an obscure book. But I am interested in historical patterns, across disciplines, which is hard to communicate. The nagging feeling stayed.

So I decided it’s time to move away from a project into a product organization. I wanted to feel out of my comfort zone again, but still contribute to technology. A few of my considerations which company to join, in no specific order, were:

• It should be first class in building delightful products with end-user impact, where good design is a strategic asset. Not just gadgets. I see technology simply as a tool to materialize curiosity. 
• Many traditional enterprises consider themselves digital companies now and like to use “disruptive” technologies, but I haven’t seen many who actually care. Mostly it's a hedging strategy. I want to work in a real digital enterprise, maybe on intelligent and intuitive products (e.g. what I often miss in AI and technology as a whole is the capacity to trace the intent “why did you do that”?) 
• It should be first class in technology. I am more of an operational excellence person, so I don't care if it's boring technology, as long as there is a clear rationale and it makes a real impact. In short: An engineering mindset. 
• It should see information and technology as a narrative. Over the years I’ve got more and more interested in systems of systems, the polytechnic, involving people, and believe the role of an Architect is to enable strategic, uncertain, but fair, critique and decisions within the team. 
• It should have a strong and clear vision, not only building products but aligning those products to a long term goal. However, I would prefer that goal to be ethical, not to wipe out the middle class, centralize means of production and distribution, or make everyone dumber (Note: This was written before the US election, there would be much more to say on this in hindsight). 
• While I believe my strength is to be a good tech lead and manager, and I enjoy building teams, I’d prefer less hierarchical organization (ever wondered that squad is a military term?), rather a swarm - what I liked about consulting. I’d prefer a strongly influencing, not necessarily managing position and a slightly slower, maybe even less visionary, but more humane company. 
• It should be global, distributed and diverse. Living in many places, working with people from all kinds of backgrounds has enriched my life more than anything else. I have enough startup experience to not believe fast-moving (hierarchical) unicorns necessarily make better corporate citizens. Hence I prefer truly multinational, larger organizations. I like foxes, not hedgehogs. 
• And finally, for now I want to stay in Asia, with its multiplicity an ideal model for the ever more complex and uncertain systems in his world, the opposite of the power structures I encountered in the past. Someday I might want to leave corporate world, and work on the grass roots. Asia is the best place to keep this door open. 

Google’s 10 things always resonated with me. They maintain a hive-mind culture based on research while at least sometimes trying to improve the world (yes, I know Silicon Valley). I am continuously impressed which brilliant people work there, and the degree of freedom they have. Let’s see if I can meet a few and learn.

Towards a progressive REST API model

"Although REST interaction is two-way, the large-grain data flows of hypermedia interaction can each be processed like a data-flow network, with filter components selectively applied to the data stream in order to transform the content as it passes"

Roy Fielding's dissertation

This is the last post of my 5-piece API story "You don't manage the API, the API manages you". In my first real full-time job in a startup, founded around 2004, I had the role of both CIO and CMO. Back then, outsiders were not only laughing at this combination, but investors directly took is an example why the organization would be immature and not able to survive. It did, though, and prospered for a long time. Fast forward 10 years later and McKinsey, Deloitte, Accenture and Adobe are promoting the CIO/CMO model and more, design thinking as culture and blueprint. And CMO/CFO get closer with programmatic content and advertising. From my own experience in startups, but also consulting and service design, I know this model is not only about making the company customer-centric, and product-centric, but focusing the culture on most value [outcomes] instead of cost savings. It's about having a common goal, and allowing fast feedback for the product or service of the organization. This fast feedback requires fast, agile changes, a lean cycle, a risk culture and a real collaborative service, or product, design - I don't like singularity but it seems that cybernetics has won.

What does that mean for our API's? Can we really still revolve around a relatively static* hyperlink model that does not embody this feedback?


*) I am aware of solutions to that, like Gateway API's, Content Negotiation and Redirects but the core issue of no common ontology still exists and DDD alone cannot solve this

Caching means Retention and other Web API quirks

"Fun here becomes related to formal logic and repetition, to the question of where software starts and ends, to mental states, to what operations it can carry out on the world, to the cultures and usages of software, to its building upon itself, to its aesthetics"
Olga Goriunova

For everyone who the other 3 posts in this series were to philosophical, relax, this is a technical one.

Thanks Daniel for suggesting Vinay Sahni's "Best Practices for Designing a Pragmatic RESTful API"; I like that he also emphasizes on documentation, what works in the real world and still believes in some principles like links. Also, Pedro's "List of HTTP API Specs", thanks to Mike for this tweet. And of course the ubiquitous REST Cookbook and the always good Apigee guides (I would reference CA Layer 7 here as well but they hide their guides behind a spamwall), the pragmatic API blog and Steve's rant.

As the 2nd last piece in this series, let's have a look into some technical quirks of Web API's.

The API and the Stream

As Alexander mentions in pattern No. 110:

“The position of main entrances controls the layout of the building. It controls movement to and from the building, and all the other decisions about layout flow from this decision. […] the first step in placing the entrances is to consider the mainlines of approach to the site”

In my last post I have written about API's as portals into your systems, a kind of grey box that gives enough semantics away to understand your system, but abstracts it for the target user group. While there is a lot material about patterns within API's and when to use API's (in messages/services), about the good and the bad, I still miss a holistic picture of systems-of-systems architecture that focuses on the API as the service itself. A behaviour that goes beyond a contract. I hope this will soon evolve in an interesting discussion, given that a few really clever people have literally picked it up:

A reminder that things we have long treasured, can be new -- and exciting -- to someone else!https://t.co/oRJFFNVjxK

— VisArch (@ruthmalan) May 28, 2016

A calm API uses situational awareness

In “Now Urbanism” Jan 13, 2011 on Informal Urbanism Celine D’Cruz explains:

“It’s important to look at all these aspects … In 20 years what will the structure look like … what do we have to do to make it simple … it’s not so complex to repair”

This post is part 2 of the series "You don't manage the API, the API manages you", which is a story of my experiences with API's over the last years and by no means a guideline for others.

In my first post, I had argued that the the distributed model of stand-alone API gateways is interesting, because it has some semantics from within the system, but no tools on the market fully support it. While there are some standards, like RFC 5988 and JSON-LD, and some implementations like OData, if you like XML/Atom, JSON API (Katharsis) or HAL/ALPS (Spring Data Cloud), they are very limited and with it limit the use of REST itself.

You don't manage the API, the API manages you

We live lives that are waveforms constantly changing with time, now positive, now negative. Only at moments of great serenity is it possible to find the pure, the informationless state of signal zero.

Gravity's Rainbow

Kick-Off for a 5-part series on API's

Inspired by a good, simple, API best practices post from Yahoo, I figured it might help to collect all the stuff I've learned in different companies and projects over the last years on API's - anything non-trivial beyond 12 Factor App, Web API Design and Serverless Architecture. For me, API's are the uniform access point to one or multiple systems. For the sake of this article I exclude API's within the same binary here, so you could argue I speak about stand-alone API's if you want (I don't like the term Web API or Front-End API too much).

Stand-Alone API's, and with it Microservices, are being promoted as "SOA, the good parts". Here, an API as front end to multiple systems is lean approach to SOA with right-sized governance, less canonical data models, and a distributed, scalable, containerised, architecture. Quite simply though, they are just a natural evolution of open ecosystems with ubiquitous, uniform HTTP interfaces, propelled by contemporary service-based business models leveraging duck testing for continuous delivery. A good example of such an API is Firebase for instance, a portal to a PaaS.

Over the course of this week I will share a few ideas I stumbled across in the last years. Some of them might have worked, others not, on some I might have worked directly, on others not. This is not a tutorial, this is a story.

The ephemeral context

Over the holidays, I've read a nice outlook by FJORD and an awesome rant by Maciej. Though the two publications could not be any more different in any sense, they touch on a similar topic:

The re-emergence of context

Back when I was studying mobile application development, it was all about context - changing texts and pictures, structure information, optimize information to location and usage, cram information on a small screen, those things. With responsive, atomic and material design, the focus shifted more and more into grid systems and ordering components efficiently, under the assumption that screens converge so much that all information is equal.

While responsive design works beautifully in many situations, the technical difficulties and trade-offs** superseded the initial idea of contextual design. However, with the "disappearance of apps", context becomes more important than efficient design. With smarter notifications, smart watches, the aggregation of information into other streams and more participatory and collaborative software service usage patterns, giving the right information at the right time becomes the major value proposition. Uber and Airbnb (with new Google Maps ads), Amazon (with Echo), Tinder but really anything that shapes an environment into a service, like IKEA, Google Now (as in Her) or Nest, are on the forefront of this movement. They don't respond to context, they create their own, ephemeral, context.

Reverse CQRS

This means mobility and analytics, in the form of deep learning, merge more and more. Mobility brings its knowledge of context into analytics, turning the flow of applications around. This makes event sourcing the new user interaction, separating the query (as in CQRS) from the command - the event becomes the query, really, and the command, as in the interaction, is executed on an interesting result.

From a service design perspective, this means we need to build stronger mental models, but as concept are harder to reason than to visualize, we need to find a way to explain them better first. We can't just respond to a demand, like opening an app, but must provide different entry points (not just API's) to services, that provide their own feedback loops. Not only, as mentioned above, as entry points via operating system hooks, like notifications. Rather with completely new, independent behaviour, starting from where we see it today for secure logins on a risk basis (e.g. the new Captcha) or proximity-based suggestions. A service design flow looks very different if there is more inputs than outputs. There is a reason SAP looks into interactive programming, and I hope Tesla's ideas for carsharing and using cars for autonomous tasks will soon be reality.

As usual, I believe communication can solve this - call it patterns, guidelines, or I've written about wrong proportion recently, but you need to establish loosely coupled "atomic" interactions, or agents, that can react to "reverse CQRS", like Redux advertises it. The analytics is actually rather straight forward, it could even start with a simple subscription model, but splitting your application flow down will be hard work - especially for all the responsive web apps that used plain old MVC and hard-wired their controllers with the overall flow.

*) Disclaimer: I work for Accenture, which is the parent company of FJORD.
**) I have myself just been on a very service/product-driven software project where we had many of those