tag:blogger.com,1999:blog-68498069028382159262024-03-13T22:29:26.860+01:00unarchitected systemsA blog about architectural emergence, specifically in IT systems by @janpeukerUnknownnoreply@blogger.comBlogger54125tag:blogger.com,1999:blog-6849806902838215926.post-64554426467375146502022-10-30T15:06:00.004+01:002023-01-03T03:31:48.782+01:00Modern Professional Services - Part 1 - Types of Consulting Organizations<p><span style="font-size: x-small;"><span style="font-family: inherit;">I am giving a <a href="https://www.usenix.org/conference/srecon22apac/presentation/peuker">talk at SRECon '22 APAC "Deploying Humans at the Edge of SRE"</a>. It's meant as inspirational, so I'm starting a 3-part series on Professional Services. </span><span style="font-family: inherit;">The bigger context is the discourse on </span><a href="https://twitter.com/kelseyhightower/status/1561412003364646914" style="font-family: inherit;">interesting career paths into tech</a><span style="font-family: inherit;">, which I've been working on joining Google, </span>particularly<span style="font-family: inherit;"> with my </span><a href="https://github.com/janpeuker/awesome-tech-roles" style="font-family: inherit;">Awsome Tech Roles</a><span style="font-family: inherit;"> compilation (please contribute!).</span></span></p><h3 style="text-align: left;">A series on Professional Services</h3><div>In this 3-part series I'd like to introduce "Professional Service" organizations in tech product companies. In times of standard convergence and consolidation, recession and and "cash is king", it's easy to observe product companies pivoting to offer services, usually paid, as a way to ensure success of implementation and provide additional value-add and thought leadership if products differences are minimal. But implementations can vary widely, and it's hard to compare how they work.</div><div><br /></div><div>I'm excluding pure managed services, outsourcing and consulting firms and the traditional industry (in-house consulting, ops) here, same as in my <a href="https://github.com/janpeuker/awesome-tech-roles">Tech Job Titles List</a>. Product <i>implementation</i> may be outsourced in some tech firms and therefore be relevant, but specifically in tech product firms is either internal effort based ("what can I get done with the SWE cycles I have") or covered by a "partner" organization who scopes projects in a similar way to what I am about to explain.</div><div><br /></div><div>This series will be split in 3 parts:</div><div><ul style="text-align: left;"><li>Part 1 on Types of Consulting Organizations</li><li>Part 2 on Estimation, and Fordian vs Bayesian Timebooking</li><li>Part 3 on Impact Metrics, including Cost</li></ul><div>To be clear, this is neither driven by Stripe or Google, not represents Stripe's or Google approach. It is purely based on my observations when talking to my peers - and given I am not in Silicon Valley likely highly biased on not canonical at all.</div></div><h3 style="text-align: left;">Types of Consulting / Professional Services Organizations within Product Companies</h3><p>When I say Professional Services I generally mean paid post-sales (post go-live / post-commit) services provided from the same company that built a product to the buyer or user of the product. That services goes beyond standard case-based support and troubleshooting or continuous customer success plans, and is defined by a <i>project</i> <i>scope and business impact</i>, not a product pipeline or effort SLA, and not billed outsourcing-style "time and materials" (like, say, an adoption-focused Customer Success Manager or Resident Architect or extended workbench).</p><p>Here is how I see the 4 extreme types of professional services organizations in tech product firms - this may sound negative, but the goal here is to show extreme cases, ymmv:</p><ol style="text-align: left;"><li><b>Separate organization or profit center</b> with completely separate P&L contribution, often even a different brand or subsidiary with different contracts and benefits. Usually these are product firms with very strong partner ecosystem, making sure their consulting arm stays "independent", often due to a strong license lock-in ("stickiness") and fixed-support-fee business that does not depend on a SasS / usage pricing model. In other words customer's success is not <i>actually</i> relevant post-sale, customers may even be forced to pay a maintenance fee which puts the professional services organization close to "customer operations". As separate organization, they may often do real implementation and delivery work. Therefore, individuals are often incentivized in their performance process purely based on P&L, e.g. chargeability metrics or thought leadership generating leads and opportunities. Due to this it is not uncommon this is actually a different entity with different contracts and employment plans.<br /><br />IBM Global Services used to be the classic example, SAP Advisory Services and, until recently, Microsoft are others, but there is a general trend to blur the lines with the shift to SaaS and customer success metrics. Not even outsources prefer this model anymore as capacity planning is hard (more on this in the next post on estimation).<br /><br /></li><li><b>Separate team within a larger organization</b>, usually either the Sales /
Go-to-Market or Support / Customer Success organization, aligned to
their goals but different methods. The lucky invariant is a prototyping, solution engineering or deployment engineering team, embedded in the rollout motion but connected to initial support. The less lucky invariant is an intermediate stage to (1), as the KPIs / metrics / OKRs need to be hacked to fit a fundamentally different culture e.g. via profit or revenue share in a <a href="https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/saas-and-the-rule-of-40-keys-to-the-critical-value-creation-metric">value-selling approach</a>. Therefore, individuals are often defined by roles rather than reporting lines, and incentivized in their performance process on metrics they have little control over e.g. product upsell or utilization. This may lead to frequent re-organization, strategy and leadership changes and associated employee retention issues.<br /><br />Usually these are SaaS usage pricing companies that at the same time have lock-in potential and therefore see professional services as a retention channel. The classic examples used to be VMware, Salesforce and ServiceNow. Cloud providers may fall back into this as default or crisis mode (as they often don't have natural "stickiness" e.g. license leverage, at best commit contracts), and while aspiring to profit centers in reality often turn it into a cost center by discounting or waiving cost for services e.g. for migration.<br /><br /></li><li><b>Horizontal enabling team</b>, role, community of practice or speciality organizations with fluidity and some fuzzy lines to presales / solution engineering, developer relations, evangelism or support. Usually these are SaaS / usage pricing companies with real long-term customer success vision beyond quarterly goals. Often for more experienced / senior individual contributors like principal architects; individual contributors who are ok with swarm-style loosely defined reporting lines and define their own specialization e.g. <a href="https://www.holacracy.org/constitution">holocracy</a>. In the extreme case a catchall for all work not otherwise clearly defined even to the extent of building unofficial tools and integrations. Therefore, individuals are often incentivized in their performance
process on metrics that represent this long-term vision like customer or partner happiness, public engagement, IP / asset generation or product enablement (e.g. sublinear support growth). In this model professional services may be close to developer relations or field engineering.<br /><br />Often de-facto operated "cost neutral" as cost or investment center and therefore smaller than in comparable firms. Amazon with its very independent team, customer obsession and embedded roles is the classic example. And every provider claiming to do "Digital Transformation" pretends to be here (e.g. Microsoft is moving from 1 to 3).<br /><br /></li><li>A rare hybrid between type 2 and 3 are professional services organizations <b>aligned to
the product feature engineering org</b>, for instance combined with Product
Ops for <a href="https://twitter.com/kelseyhightower/status/1005185318125834240">customer empathy</a> (<a href="https://skamille.medium.com/the-product-culture-shift-441c31a3fdf1">also here</a>) strategic customer advisory and lighthouse customer launch support,
or with DevRel to <a href="https://github.com/GoogleCloudPlatform/professional-services">cover OSS tools and integrations</a>. In the best cases these are actually rotation-based, starting with a <a href="https://buildyourfuture.withgoogle.com/programs/cloud-technical-residency">residency program.</a> Often they are smaller organizations where "everything customer facing" is grouped e.g. into Field or Deployment engineering.<br /><br />Often these are data companies who build very specific use cases and can monetize their reuse, e.g. Palantir, Databricks and Looker, also Google used to be like this. A recent thread shows <a href="https://twitter.com/clairevo/status/1575161631524343810">Color seems to do Solution Engineering this way</a> (the right way). Seen as an investment or profit center but via product usage or adoption, not direct payment for services. Unsurprisingly all of these sound familiar to <a href="https://teamtopologies.com/key-concepts">Team Topologies</a> - this type would be the Platform Team, the closest to how SRE works, just on the other end of spectrum as maximum customer empathy team (often Professional Services sees itself as "spearhead" or the product).<br /></li></ol>A fifth type would be an organization that <b>does not have in-house professional services</b>. The classic example here are <a href="https://tomtunguz.com/five-pillars-of-plg/">Atlassian</a> and <a href="https://stratechery.com/2014/big-blue-apples-soul/">Apple</a>, arguably even Oracle, who always put self-services and partner network first, with their field engineers primarily being a glue between partners and internal engineering teams. In Apple's case there is also simply the product decision to <i>not</i> be customizable, so naturally field engineering is more like DevRel, evangelizing the ideas which are famously executed <i>without</i> considering the customer. Microsoft and SAP over the aeons oscillated between this model and type 1 above.<div><br /></div><div>In part 2, I will talk about estimation, and how work is prioritized in Professional Services teams across those 4 models.</div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-75390790659994388742022-01-15T08:46:00.006+01:002022-04-13T06:00:45.314+02:00Letter to my future self 2022 editionReading my earlier “<a href="https://unarchitectedsystems.blogspot.com/2016/11/letter-to-my-future-self.html">letter to my future self</a>” from 2016 is interesting, because it still holds true, irrespective of how the pandemic changed the workplace: I wanted to work on delightful products with end-user impact, intelligent and intuitive products with a strong and clear vision. First class in technology, with information and technology as a “narrative”, meaning culture, and with a strong focus on operational excellence. Back then <a href="https://sre.google/books/">the SRE book</a> had just not been released so it was hard to explain to classic “architects” what my role would be, yet I was keen to learn how to run things at scale. I wanted to be part of a global, distributed and diverse organization, a “swarm” with a focus on Asia, what I liked about consulting, not a manager.<br /><br />All of that, and more, came true and I am grateful to Google for giving me this opportunity. I learned more about product development and excellence, customer empathy, tech leadership, reliability and eventually SRE than I ever expected - from the most brilliant and humble people I ever met. I was lucky to work on one the largest scalable, concurrent and low latency systems in the industry, diving deep into <a href="https://www.montecarlodata.com/data-observability-the-next-frontier-of-data-engineering/">(data) observability</a> and machine learning. Helping migrate some of them to Spanner I built up experience to help <a href="https://cloud.google.com/blog/products/gcp/gcp-opens-a-third-zone-in-the-singapore-region">launch Cloud Spanner in Asia</a>, and with that move from product technology management to strategic cloud engineering. I was again lucky to help integrate and migrate <a href="https://cloud.google.com/customers/go-jek">real-time streaming systems</a> and <a href="https://www.youtube.com/watch?v=jx9NozDp2XY">data warehouses</a>, and improve products like BigQuery, Dataflow / Beam and Kubernetes / Cloud Run. The only thing I couldn’t avoid was becoming a tech lead manager again - but I love building up teams and so I focused a lot on creating coaching, scaling (security and scoping), hiring, onboarding and culture programs, with a special focus on inclusion of diverse backgrounds and community and student outreach to <a href="https://unarchitectedsystems.blogspot.com/2021/11/tech-job-titles.html">make tech legible</a>.<div><br /><span><a name='more'></a></span><div><br /></div><div>But much of tech is turning to a <a href="https://locusmag.com/2021/01/cory-doctorow-neofeudalism-and-the-digital-manor/">state of feudal subscription-ication</a> - yet worse <a href="https://medium.com/humane-tech/there-is-no-technology-industry-44774dfb3ed7">every company is a tech company</a>, and <a href="https://future.a16z.com/every-company-fintech-company/">every tech company a fintech company</a>. Apple probably being the best example of become a <a href="https://stratechery.com/2020/apples-shifting-differentiation/">platform</a> and eventually <a href="https://500ish.com/apple-prime-and-the-iphone-pro-40d970060c8b">a bank</a> by <a href="https://stratechery.com/2013/clayton-christensen-got-wrong/">forcing</a> users with iron fist over their and their <a href="https://www.nytimes.com/2021/12/30/technology/apple-airtags-tracking-stalking.html">families</a> “<a href="https://www.ben-evans.com/benedictevans/2022/1/2/2022-questions">private</a>” data into <a href="https://calpaterson.com/printers.html">subscriptions, and reduced interoperability</a> across services. Other examples of this are Amazon also <a href="https://fortune.com/2017/02/14/andreessen-horowitz-fintech-alex-rampell-amazon/amp/">becoming a bank</a> for instance with <a href="https://mjtsai.com/blog/2020/06/01/unhelpful-amazon-order-confirmation-e-mails/">empty order emails</a>, or generally restricted access to <a href="https://www.eff.org/deeplinks/2021/10/internet-archive-transforms-access-books-digital-world">books</a> and scientific <a href="https://www.theverge.com/2018/2/8/16985666/alexandra-elbakyan-sci-hub-open-access-science-papers-lawsuit">papers</a> only to per-publisher subscription, iPhone’s and Google Drive’s <a href="https://superuser.com/questions/1415845/can-you-back-up-your-google-drive-folder-on-my-mac-with-time-machine">intransparent</a> and <a href="https://discussions.apple.com/thread/252732394">broken</a> file sync, headphones that <a href="https://discussions.apple.com/thread/251375367">automatically open Apple Music</a> or the general unavailability of non-blockbuster movies, <a href="https://www.economist.com/finance-and-economics/2019/10/05/the-economics-of-streaming-is-changing-pop-songs">music</a> and podcasts unless you subscribe to “plus” streaming services. Having said that, this reliance on lock-in subscriptions is only a side effect of the positive trend of easier direct payments and sponsoring, of the democratization of platforms. But the lock-in side is a slippery slope to <a href="https://www.patrick-breyer.de/en/posts/messaging-and-chat-control/">censorship</a> or <a href="https://www.reuters.com/article/us-china-apple-icloud-insight/apple-moves-to-store-icloud-keys-in-china-raising-human-rights-fears-idUSKCN1G8060">worse</a>, given many services also reserve the right to <a href="https://workspaceupdates.googleblog.com/2021/12/abuse-notification-emails-google-drive.html">remove</a> or <a href="https://blog.quintarelli.it/2021/08/apples-child-new-abuse-prevention-system-an-antritustcompetition-point-of-view/">forward</a> data without appeal. 10 years after the financial crisis some of tech turned into a <a href="https://blog.pragmaticengineer.com/advice-for-tech-workers-to-navigate-a-heated-job-market/amp/">gold rush machine</a>, partially because <a href="https://digitalrightswatch.org.au/2021/09/02/australias-new-mass-surveillance-mandate/">governments</a> and <a href="https://www.nytimes.com/2021/02/17/business/media/australia-google-pay-for-news.html">media</a> learned to use it to <a href="https://www.patrick-breyer.de/en/posts/messaging-and-chat-control/">their advantage</a> <a href="https://en.wikipedia.org/wiki/Manufacturing_Consent">(reference)</a> - and push the <a href="http://powazek.com/posts/3229">subscription = privacy</a> fallacy in an attempt to preserve <a href="http://powazek.com/posts/3229">failing business models</a> and power structures, <i>against</i> democratization. </div><div><br /></div><div><br />Mirroring the consumer market and the trend to become banks (after all <a href="https://www.forentrepreneurs.com/saas-metrics-2/">SaaS negative churn</a> aka "ramp" is the same as modelling a fixed income structured product), many cloud providers (except e.g. <a href="https://www.swyx.io/cloudflare-go/">Cloudflare</a>) are entering a new phase of “moving up the value chain”: Specific, <a href="https://www.linkedin.com/pulse/working-microsoft-consulting-services-michiel-van-vliet-/">packaged industry solutions and digital transformation</a> (“marketing slang” as I called it in my old post, but also in the transformation to <a href="https://techdatacloud.eu/blog/how-a-little-known-financial-model-can-finance-migration-to-the-cloud/">subscription sense</a>) are the new spearheads, not technology, or technological products and platforms. The platforms are not enablers anymore but outsourcers - and to turn this around <a href="https://guykawasaki.com/how_to_prevent_/">quickly hire and acquire outsourcers</a>. Enabling or disrupting technology is often left to the new, fast, product startups, <a href="https://ottertune.com/blog/2021-databases-retrospective/">especially in the data</a> and <a href="https://kwokchain.com/2021/02/05/atomic-concepts/">collaboration</a> space. Each cloud provider still has their traditional strength, like developer and user integration, ease of use, hybrid scalability, security, data analytics, regional resiliency or legacy integration through flexibility and versatility in paradigms - but the overlap has become so large that <a href="https://www.airplane.dev/blog/i-started-a-saas-company-in-2013-and-2021-heres-how-its-changed">enterprise SaaS deals</a> are decided on a strategic level and <a href="https://architectelevator.com/architecture/cloud-oss-lockin/">lock-in is considered a feature</a> of <a href="https://www.infoq.com/articles/avoiding-lockin-switching-costs/">shared fate</a> and long customer lifetime value. Not even <a href="https://www.cisa.gov/emergency-directive-22-02">security</a>, <a href="https://news.ycombinator.com/item?id=21103683">trust</a>, cost or <a href="https://www.theverge.com/2021/12/22/22849780/amazon-aws-is-down-outage-slack-imgur-hulu-asana-epic">reliability</a> incidents can break those long-term financial, often business strategy, ties.</div><div><br /></div><div>Yet despite all the FinOps, procurement departments, especially the ones of the failed business models from above, still don't know how to play this smartly - which leads to large, fixed deals. Like a virus those reduce the agile and experimental culture of the affected tech firms and introduce a zero-sum-game mindset. And with larger deals larger teams have to be constantly fed. Some providers start <a href="https://twitter.com/QuinnyPig/status/1461860631838019584">taking shortcuts</a>, others in the market place consider to follow pace, away from technological progress. <a href="https://unarchitectedsystems.blogspot.com/2016/11/letter-to-my-future-self.html">Almost 6 years ago that’s why I left Accenture Digital</a>, who were early, innovative and still strong in this market. Now I decided to leave Google for the same reason, to <a href="https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/">once again move back</a> from manager to “swarm” and from strategy to product and technology with a clear user focus vision and narrative.<br /><br /><br />So, nothing changed - really. Well, the pandemic did its part and so I became more interested in real, <a href="https://stripe.com/blog/remote-hub-one-year">multi-national remote work</a>, meaning a culture around strong operating principles that fosters global collaboration around a basket of enabling products, not only platforms, and has a non-zero-sum mindset. That <a href="https://medium.com/@emdashry/engineering-in-asia-3386dcc26861">takes Asia seriously</a>, doesn't have a California-centered image of the world and doesn't only have the best opportunities in select physical locations. It so happens that <a href="https://www.readthegeneralist.com/briefing/stripe">Stripe</a> was long on my radar for that.</div><div> </div><div>I like the spirit of making an inherently messy and complex world legible and democratized. In particular real <a href="https://stripe.com/en-sg/jobs/life-at-stripe">user focus</a>, and a <a href="https://slab.com/blog/stripe-writing-culture/">writing culture</a> specifically without that <a href="https://www.youtube.com/watch?v=IEe-5VOv0Js">marketing slang</a> but instead rigorous, long-term thinking. Incidentally, I actually worked on quite a few payment systems before, from FX to mobile terminals. If the world and tech already changes to payments and subscriptions, at least I can try to make them better, more democratized and more open for everyone. I also want them less invasive (No, I don’t want to set up a PayPal account <a href="https://quolum.com/blog/saas/i-analyzed-saas-billing-dark-patterns/">dear dark pattern</a>, and why does Apple share my Apple e-mail with merchants on Pay?), more interoperable and resilient, but most importantly <i>inclusive</i> of people with less money than tech elites, and people outside the US, especially Asia. This user focus combined with strong developer architecture is what I find interesting. I am happy to admit that I bought almost all issues of <a href="https://increment.com/">Increment</a> and almost all <a href="https://press.stripe.com/">Stripe press</a> books anyways before.</div><div><div><br /></div><div><br /></div><div>Joining Stripe Singapore as an integration engineer I’d be delighted to <a href="https://unarchitectedsystems.blogspot.com/2021/11/tech-job-titles.html">make this role more visible</a> and publish more on it - who knows, maybe it’s the next SRE?<br /></div></div></div><div><br /></div><div><br /></div><div><span style="font-size: x-small;"><span style="background-color: #fcff01;">Note</span>: As I am starting a new job I will publish 2 <i>backdated</i> articles (<a href="https://unarchitectedsystems.blogspot.com/2021/12/my-tech-interview-style-and-integration.html">first one here</a> and <a href="https://unarchitectedsystems.blogspot.com/2022/01/a-view-into-state-wip.html">second one here</a>) in a rough state so I don't get too biased / informed by my new role.</span></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-2068012328069264682021-12-27T08:43:00.664+01:002022-01-29T07:01:09.077+01:00My Tech Interview style (and the Integration Engineer role) [backdated draft]<p><i><span style="background-color: #fcff01;">Note</span>: This post is backdated to the date of the last draft (27 Dec 2021) as I <a href="https://unarchitectedsystems.blogspot.com/2022/01/letter-to-my-future-self-2022-edition.html">changed my job and role</a> and didn't want to bias / inform myself by that. It's an unfinished fragment of my thinking at that point in time that I just cleaned up a little and added references where necessary, but it's still rough and incomplete.</i></p><p>I've never been happy with the Tech interview process and burned by it many times - being under-levelled, in role I barely understood (a reason I launched the <a href="https://unarchitectedsystems.blogspot.com/2021/11/tech-job-titles.html">Tech Job Titles</a> list), rejected for algorithm questions ("write a solver for Tetris but in n dimensions"), or simply not even screened for "no product experience". This form of gatekeeping in the tech industry is one of my pet peeves, but it's also simply unproductive and inefficient. It only works because of survivorship bias for people from top universities who are being prepped with special courses, books, test interviews and special coaching by tech firms - as such it functions more as a ritual than actual role fit and/or <a href="https://business.linkedin.com/talent-solutions/resources/interviewing-talent/how-to-assess-skills/culture-add">culture add</a>. It is basically a code (e.g. speaking out loud during programming), and by knowing that code the interviewee signals the interviewer to be part of the same group ("likeability").</p><h3 style="text-align: left;">My interview style</h3><p>I've done about ~250 tech interviews at Google and ~250-500 at Accenture, plus quite few in non-for-profit volunteering and my own startups - regardless whether the role was called <a href="https://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-programmer/">programmer</a>, engineer, consultant or architect. Instead of going into depth on what the current interview process is or what's broken with it (<a href="https://github.com/janpeuker/awesome-tech-roles#interviews">the list has a few pointers</a> and there is <a href="https://neverworkintheory.org/2021/09/13/whats-wrong-with-tech-hiring.html">great research by others</a>), let me highlight what is important for me in tech interviews (leaving <a href="https://www.writethedocs.org/hiring-guide/">behavioural</a> and hypothetical aside):</p><p><br /></p><span><a name='more'></a></span><h4 style="text-align: left;"><br /></h4><h3 style="text-align: left;">Standardized questions and structured answer guides</h3><div>My going-in position is that interviews need to be fair, equitable, check for culture-add and avoid tech industry bias. But that's hard to bring across in interviewer trainings (or takes a long time), so for me <a href="https://hbr.org/2017/06/7-practical-ways-to-reduce-bias-in-your-hiring-process">fixed interview question and answer guides</a> with clear rubrics across work best. With standardized I don't mean <a href="https://twitter.com/carnage4life/status/1462510561141215240">same-for-all (that's also biased)</a>, but usually those questions are pluggable into the whole end-to-end interview experience.</div><div><br /></div><div>With "pluggable" I mean a question should be <a href="https://bradfrost.com/blog/post/atomic-web-design/">atomic</a>. It should be independent from the round / stage, experience (level but also for each required skill separately), role / function and special requirements such as local language. Composing which questions to ask for a role and stage is a separate process, ideally supported by tools that also allow for permutations based on interviewer (e.g. "<i>This interviewer is fully calibrated, let them choose to test this new beta question</i>"), while keeping a balance of areas asked in questions (e.g. "<i>Ask first a 5-min specific tech question from the candidate's specialization, second a 5min specific tech question from the role's specialization, followed by 1-2 10min tech coding questions etc"</i>) and therefore allow faster feedback because the rubric is in the tool itself (which at some point I am planning to build).</div><div><br /></div><div>Feedback not only to the candidate but the recruiter for <a href="https://hbr.org/2019/05/your-approach-to-hiring-is-all-wrong">good pre-screening and for long-term metrics</a> and passive or internal candidates. Such <a href="https://rework.withgoogle.com/print/guides/6487203966353408/">structured feedback</a> also helps candidates being routed to other roles that fit their skills, potentially even during the interview. Especially for <a href="https://acloudguru.com/blog/business/why-you-should-invest-in-undervalued-people">undervalued</a> people <a href="https://engineering.linkedin.com/blog/2018/06/coding-conversations--finding-the-right-fit">non-traditional backgrounds</a> some roles might be better to build up skills - and should be planned like that in the career path and company culture (Accenture was <i>excellent</i> in this).</div><div><br /></div><div>For me a good question and its answer guide needs to have most of these characteristics:</div><div><ul style="text-align: left;"><li>Short, easy to understand and free of bias or misunderstanding - a 45min interview should focus on solving, not understanding the problem. In particular do not use jargon ("<i>reverse a linked list</i>"), slang ("<i>let's solve this IRL</i>"), in particular not tech slang that excludes non-traditional backgrounds ("<i>how would you disrupt Netflix?</i>") language that's hard to understand for non-native speakers on a call ("<i>given a parallelogram with dimension a 15 inches and h 50 inches...</i>"), US-focused cultural references ("<i>build a system to optimize the order of the NBA draft</i>") or brainteasers because they require a certain education (e.g. a western-style MBA). A good starting point are <a href="https://www.quora.com/Whats-the-logic-behind-Google-rejecting-Max-Howell-the-author-of-Homebrew-for-not-being-able-to-invert-a-binary-tree/answer/Max-Howell?share=100e0bb6&srid=unBJ9">real world problems</a> and common things around us globally.</li><li>Open and helpful - the goal of an interview is not to stun a candidate or impress them with the smartness of the interviewer. Questions should "fan out" an possible narrative through a question like a tree and allow to go into different directions. A good answer guide has nudges and hints in different degrees, and links to the rubric whether or how a hint does influence the final rating. Building a reasonable hypothesis in ambiguity (not knowing the right answer immediately) and validating it is itself an important skill.</li><li>Versatile and flexible - the question should cover a broad range of <a href="https://en.wikipedia.org/wiki/Dreyfus_model_of_skill_acquisition">skill / experience levels</a> and can be used with slight modifications in different rounds or different roles (see below). In particular the question should allow for drill-downs or follow-ups that probe deeper yet still giving the candidate a good experience and getting some signal even if the original answer or direction was not optimal. The answer guide should rank answers by skill / experience level, not just give an ideal answer. An ideal question is extremely simple and gets quickly more complicated with each drilldown (the easiest drilldown is "Why?"). Facts should be the nodes of the answer search tree not the root, e.g. if the first question is "<i>How would you build a messaging app like WhatsApp or iMessage?</i>" at the end if could be "<i>What's a reasonable SLO for a message to show up at the receiver?</i>". The only exception are some easy fact questions in the beginning to verify a common basis and ease into the interview.</li><li>Able to add more <a href="https://rework.withgoogle.com/print/guides/6487203966353408/">attributes e.g. cognitive ability</a> - for some roles or levels not only the "what" but the "how" becomes important, if the candidate takes clear assumptions or decisions and scopes the answer properly for instance. The answer guide should allow drill-downs for that, easy ones are role games e.g. "<i>How would you explain this approach to a less experienced peer to implement?</i>"</li><li>Clear grading rubric that shows expected skill and level and outcome with example answers e.g. in a <a href="https://blog.pragmaticengineer.com/system-design-interview-an-insiders-guide-review/">system design question</a> one particular sub-skill or attribute might be "concurrency" and another "reliability". If the role expects competent in concurrency and proficient in reliability architecture it should be easy for the interview to gauge the level out of the answer guide grading rubric in the context of the question e.g. "<i>Concurrency: Competent if speaking about (lightweight) threads or processes and distributed systems without going into consistency models e.g. synchronization, consensus</i>". Usually this requires some general definition of those skills or experiences across all roles and a <a href="https://css-tricks.com/the-importance-of-career-laddering/">role ladder</a>. However, innovative and original answers should be encouraged and feed back into the answer guide.<br /></li></ul></div><h4 style="text-align: left;"><br /></h4><h3 style="text-align: left;">Managing Complexity</h3><div>Interviews are <a href="https://www.quora.com/Whats-the-logic-behind-Google-rejecting-Max-Howell-the-author-of-Homebrew-for-not-being-able-to-invert-a-binary-tree/answer/Max-Howell?share=100e0bb6&srid=unBJ9">not the real job</a> and there is no real way to test for that while also <a href="https://www.bbc.com/worklife/article/20210727-the-rise-of-never-ending-job-interviews">being respectful</a> of a candidate's time. I like David Farley's foundations of software engineering: An engineer has to be (a) an expert in the art of <a href="https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD316.html">managing complexity (from Dijkstra)</a> and (b) an expert in <a href="https://www.lambdabytes.io/posts/selearning/">iterative learning</a> and feedback. First and foremost I have to assess these two factors in an interview before testing further for specific skills that help me (hopefully, to a degree) fill skill gaps in my team. Hiring externally is expensive, and external hiring changes the dynamics of the team (potentially churn or attrition especially in a <a href="https://blog.pragmaticengineer.com/advice-for-tech-workers-to-navigate-a-heated-job-market/">heated market</a>) and the whole company culture in the long term - no short-term skill gap for a certain feature or project can justify sacrificing that.</div><div><br /></div><div>Managing complexity and learning in an interview means the candidate needs to be able to <i>develop</i> an approach and <i>rationalize</i> it with strong intuition and experience (see below). Knowing important facts helps in rationalizing faster, e.g. mentioning a space/time complexity correctly for an algorithm simply leaves more time for interesting detail discussion, but the fact alone is rarely relevant.</div><p>For example, algorithms and data structures are important but I usually look for the why, not pattern recognition - I don't care how easily a candidate can <a href="https://www.crackingthecodinginterview.com/">"crack"</a> to identify whether the problem can be solved with a boilerplate tree, hash table or dynamic programming solution and hack that down in 10 minutes in a coding interview. I do fully recognize research shows knowing algorithms is a good predictor of job performance because of the ability to learn, but I apply that insight to <i>types of problems</i>, not memorization. For instance, realizing a problem is complex (e.g. NP complete) and discussing options to approximate or simplify a solution with good real-world assumptions is more impressive than coding the standard solution. For that reason I prefer code reading and improvement questions - more below.</p><h4 style="text-align: left;"><br /></h4><h3 style="text-align: left;">The Scope - Constraint - Focus model</h3><p>I quite like <a href="https://philcalcado.com/2016/03/15/on_asking_job_candidates_to_code.html">take-home</a> or <a href="https://jacobian.org/2021/nov/23/wst-homework/">homework</a> or pair programming questions but they can introduce bias for certain languages, frameworks of paradigms. Worse though, especially for Google there is an <i>industry</i> of coaching that goes to the extent of fraud, and questions are frequently leaked which requires constant monitoring and adjustment. I personally had <i>a lot </i>of first hand experiences with other people in the room, electronic help, memorized answers and trying to write down sessions or recording of the interview. This challenge has to be balanced with the need for standardized questions - writing good answer guides and rating rubrics is hard and having to throw them away is frustrating.</p><p>A good middle ground between homework and live coding / whiteboarding (which can be <a href="https://medium.com/@Johnmont_67962/rethinking-how-we-interview-in-microsofts-developer-division-8f404cfd075a">quite stressful and simply not realistic</a> but may be necessary to validate an answer sadly) is coding reading and improvement. You could provide a piece or code, design or data upfront, usually 24 hours, and the candidate has to come up with improvements, usually some on their own and some based on system properties (e.g. "<i>How would you make this 2x faster</i>"). I usually prefer homework that needs a good discussion and explanation and a little bit of live improvement in a shared document. That can be adding / fixing code, drawing a diagram differently or doing an analysis on data (e.g. "<i>what data would have to be cleaned up first?</i>"). If necessary the code reading can also be live, however properly understanding a piece of code or system can easily take 10 minutes which you won't get back. Needless to say coding interviews should allow any common language, never test syntax or conventions and generally not allow helper libraries (rather simplify the problem than require libraries that exclude some candidates).</p><p>Going deeper into "versatile and flexible" from above: Over time I've developed some frameworks to create new questions that can share portions of answer guides and rubrics, I call that the Scope - Constraint - Focus model. I know frameworks are biased too (the <a href="https://www.ddiworld.com/solutions/behavioral-interviewing/star-method">STAR model</a> has lead to highly ritualized behaviour questions for instance, which reinforces the "code" problem) but I feel this is flexible enough. A good example are <a href="https://sre.google/workbook/non-abstract-design/">NALSD questions</a>:</p><p></p><ul style="text-align: left;"><li>Scope is the problem itself, the story. While the question should define a rough context / boundary, managing the scope within the timeframe for the answer is also a great way for a candidate to show skill and employ nudges, so scope management (e.g. assumptions, preparation) should be possible. For instance "<i>Design an air traffic control system</i>".</li><li>Constraint adds a (varying) degree of <a href="https://www.infoworld.com/article/3639050/complexity-is-killing-software-developers.html">complexity</a>, from <a href="https://analytic-storytelling.com/scqa-what-is-it-how-does-it-work-and-how-can-it-help-me/">SCQA</a>. It tests for application of experience, intuition, risk approach, tradeoffs - as we say <a href="https://www.bredemeyer.com/whatis.htm">architecture are the significant decisions</a>. Constraints are the easiest to tweak for different skill or role levels and can make a question easier or harder. For instance "<i>Design an air traffic control system for a regional airport so that information is never older than 200ms</i>".</li><li>Focus allows for better time management and specific terms for the role. It's optional if the goal is to actually test scope and time management, but important if facts or drill-downs are important. For instance "<i>Design the high-level components of an air traffic control system backend for a regional airport with a data freshness SLO of 99.99% consistent within 200ms</i>". However it's important not to get into <a href="http://steve-yegge.blogspot.com/2008/03/get-that-job-at-google.html">random details</a> that <a href="https://aphyr.com/posts/341-hexing-the-technical-interview">add noise</a> instead of reducing it.</li></ul><h4 style="text-align: left;"><br /></h4><h3 style="text-align: left;">Testing for Experience</h3><div>What I want to check for in a candidate is <a href="https://cloudirregular.substack.com/p/the-greatest-resume-ive-ever-seen">hands-on experience</a> not necessarily in exactly the same technology or domain but the pattern of thinking, and the growth potential in that direction. I find checking if a candidate is (and <a href="https://www.verywellmind.com/what-is-self-determination-theory-2795387">wants to be or can grow to be</a>) rather a hacker or a fixer, a troubleshooter or an architect, an optimizing engineer or boundary-breaking scientist, more important than whether someone has Python2 or Python3 experience. Of course there are limits to this: I can't assume someone can learn machine learning in a short time, but given onboarding usually takes 3 months every technology learnable to a degree that the person understands the product and can make simple changes within that time should be acceptable. Anyways technology changes often faster than that - 6-month old Kubernetes knowledge is as good as none, which I have learned myself the hard way. An Angular developer can learn React in that time, and a Python developer Ruby. But a data scientist might not be able to learn distributed systems, or a Bash-wizard might not be able to learn Go concurrency in that time. The goal of an interview is to estimate this gap. I prefer to ask for vision, simply "<i>How would you make this better if you could, what about this should be automated and why?</i>".</div><div><br /></div><div>An additional advantage of this approach is that existing "hard" skills are less relevant - CVs are not screened out too early based on missing keywords. Personally, I never read a CV or resume or look someone up before an interview - I only ask recruiters for a specialization (e.g. to choose a question I should know if someone has has used SQL in general, but not which database).</div><div><br /></div><div>A good example of this problem is when an experienced interviewer hires less experienced candidates, in particular if there is a long time difference between the time each party spent in university (or ramping up on skills) - the interviewer may assume "<a href="https://blog.royalsloth.eu/posts/they-dont-even-know-the-fundamentals/">fundamentals</a>" that are not fundamental anymore. For instance, many infrastructure questions assume a level of familiarity with networking that simply is not needed anymore, and sometimes even hard to learn. Students work with "magic" Wifi on serverless Cloud platforms like GitHub, AWS Lambda or Glitch, they don't deploy services onto a VM environment. Asking "<i>What happens when I type google.com in the browser?</i>" is not only an obvious and too easily memorizable question with not much signal - it's also irrelevant nowadays apart from from infra ops.</div><h4 style="text-align: left;"><br /></h4><h3 style="text-align: left;">The Integration Engineer role</h3><p>As highlighted in my list of <a href="https://unarchitectedsystems.blogspot.com/2021/11/tech-job-titles.html">Tech Job Titles</a> (<a href="https://github.com/janpeuker/awesome-tech-roles">GitHub</a>), some roles are especially ambiguous because they depend a lot on the org structure the role - I had mentioned the Solution Architect. Here I want to give one more example, the "Integration Engineer". It may also be called Deployment Engineer, Migration Engineer, Field Engineer, Delivery Architect, Technology Architect, Customer Solution Engineer, Implementation Services, System Engineer, Solutions Engineer or basically anything in Professional Services. In my last role in <a href="https://unarchitectedsystems.blogspot.com/2022/01/letter-to-my-future-self-2022-edition.html">Google (see Note on my role change above)</a> it was called "Strategic Cloud Engineer" (SCE) or simply "Cloud Data Engineer" or "Cloud Infrastructure Engineer".</p><p>The speciality about the Integration Engineer role is that it's an engineering role, but it requires more empathy, <a href="https://www.inc.com/justin-bariso/there-are-actually-3-types-of-empathy-heres-how-they-differ-and-how-you-can-develop-them-all.html">particularly cognitive empathy</a>, because it's <i>someone else's system</i>. And that system exists - it's not legacy, it's heritage, and changes might have surprising, unknown side effects. In that regard Integration Engineers are very similar to SREs, just that SREs focus on internal customers, whereas integration engineers focus on external ones. SREs therefore have a clearer framework and metrics, SLOs for instance, whereas integration engineers also need to be flexible with processes and communication standards, and often <a href="https://bellmar.medium.com/hunting-tech-debt-via-org-charts-92df0b253145">literally translate or bridge cultures</a>. Both deeply care about the system, want to improve it and may often be "on the hook" for it, yet only have influence over it, not control, power or authority. Both SRE and integration engineers are masters in observability and <a href="https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-called-legibility/">legibility</a> of sociotechnological systems.</p><p>So how would I hire Integration Engineers ideally (this is not a hiring guide for the SCE role, and the process is very different)? Trying to combine as <a href="https://blog.interviewing.io/technical-interview-performance-is-kind-of-arbitrary-heres-the-data/">many independent factors</a> as possible:</p><p></p><ul style="text-align: left;"><li>Greate pre-screening by recruiters that identifies strengths, options and relevant focus areas for drill-down, especially for a first conversation (e.g. tell me the hardest bug you solved which made you happy)</li><li>Design homework: Drawing up a basic integration with a clear constraint e.g. single-threaded single system. Then changing the constraint in the interview so the context / scope is clear and doesn't change e.g. to a serverless system in the cloud.</li><li>Data analysis homework: Coming up with certain insights and discussing them in the interview, drawing conclusions out of those, for instance how certain features are used or what values are present in an API or integration.</li><li>Coding homework with an existing integration changing it "roughly" - for instance from an existing version 1 of an API to a provided version 2, that requires some data wrangling. It could be followed by live pair programming to make the change production ready.</li><li>Live system design questions around rolling out such changes. In particular what questions to ask the other side and how they influence the solution, how to prioritze requirements and estimate integration patterns - potentially even a role play with a customer who has certain requirements.</li><li>Finally behavioural and hypothetical questions about past experiences. Campfire war stories are always great insights but also relaxing and fun and close a set of interview rounds niceley.</li></ul><p></p><p><br /></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-91622462879696515932021-11-12T16:55:00.005+01:002022-01-24T07:29:09.417+01:00Tech Job TitlesOne of the most common questions I get on career panels, mentoring or coaching sessions and after talks is to explain what my job, my role actually is. There is no easy answer, because it requires an understanding of the complex dependencies of roles in "Tech" (for lack of a better word, information technology product) firms.<br /><br />Information Systems, Computer Science or Design Thinking courses or books for career starters typically focus on explaining the Software Engineer (SWE) job, sometimes the Product Manager (PM) and partially process-specific rituals and roles (e.g. in Scrum the Product Owner). But rarely do they mention the difference between performing the SWE role in non-tech industry (<a href="https://medium.com/humane-tech/there-is-no-technology-industry-44774dfb3ed7#.c9xi6vier">for lack of a better word</a> - firms or public entities that primarily exist due to non-tech “hardware” products even though they <a href="https://www.wsj.com/articles/every-company-is-now-a-tech-company-1543901207">might claim to "digitalize"</a> afraid of <a href="https://future.a16z.com/software-is-eating-the-world/">disruption</a>), in consulting (firms that provide services, usually to the former, to "digitalize") and tech (firms that sell “software” products and services to gain an advantage with speeding up "digitalization"), and within tech <a href="https://medium.com/swlh/https-medium-com-tomaspueyo-the-secret-to-picking-your-next-tech-employer-7869a5a9367e">between startups and established firms</a>. I've never seen any mention tech-adjacent roles that might have "engineer' in their title but are not SWEs.<br /><br /><a href="https://blog.pragmaticengineer.com/advice-for-tech-workers-to-navigate-a-heated-job-market/">With recent hypergrowth</a>, a lot has been published on "upwards"* career management and manager roles in Tech. But I haven’t seen a good resource on the continued blending of these archetypes and roles and responsibilities - with a <a href="https://github.com/janpeuker/awesome-tech-roles">list of observations</a> and this article as background reading I'm trying to do so. Maybe later I’ll do a variation for non-tech industry companies and consulting firms.<br /><br /><span style="background-color: #fcff01;"><b>tl;dr </b>I've <a href="https://github.com/janpeuker/awesome-tech-roles">created a GitHub repo "awesome-tech-roles"</a> modeled after the "Awesome Lists" with the goal of showing typical clusters of tech-adjacent roles and references to good articles that illustrate the variety within the roles - for career starters in the tech industry. The links are examples, observations, not endorsements of my personal view. Given the bias towards SWE and PM roles I've tried to keep roughly the same number of references per role cluster and focus on established tech firms.</span><div><br /><span><a name='more'></a></span><p>Let me explain how I structured the <a href="https://github.com/janpeuker/awesome-tech-roles">observations going into the list</a>, and by doing so trying to build a framework for career starters to navigate the different roles they might see - or get offered. To make this easy to read I'm going to break it down into a few principles / assumptions / axioms / catchphrases.</p><p><br /></p><h3 style="text-align: left;">1. The product is less important than how people use it</h3>Design Thinking courses often get very handwavey when it comes to turning a prototype into an evolving product. And engineering courses often pretend the best engineering makes the best product. But adoption is not guaranteed by perfection or brilliance in either, rather the interplay of them and user empathy..<br /><br />Good engineering allows flexibility and fast learning (iterations, experimentations) not only on solutions but on <a href="https://hbr.org/2012/09/are-you-solving-the-right-problem">problems</a> and the <a href="https://mcfunley.com/choose-boring-technology">mapping</a> between. And a consistent, easy to grasp model <a href="https://domainstorytelling.org/">telling a story</a> that makes complexity understandable - to the point of appearing like simplicity (not magic). A good example are the <a href="https://kwokchain.com/2021/02/05/atomic-concepts/">swath of new products</a> making common tools simpler yet more collaborative, for instance Slack, <a href="https://kwokchain.com/2020/06/19/why-figma-wins/">Figma</a> or Notion to name a few. For that you need analysts, designers, <a href="https://www.businessinsider.com/larry-page-the-untold-story-2014-4">even project managers</a>, <a href="https://cloudirregular.substack.com/p/every-engineer-should-do-a-stint">and yes, consultants</a> and engineers <a href="https://twitter.com/kelseyhightower/status/1088521771085574144?lang=en">working directly with customers</a>. Even if it's just for the sake of keeping engineers (SWEs) <a href="https://www.blogger.com/blog/post/edit/6849806902838215926/9162246287969651593#">happy and focused</a> on <a href="https://www.twosigma.com/articles/admire-a-product-driven-approach-to-automating-migrations/">core work</a>, yet everyone feeling included in the product success. However, that’s also why the list limits articles on PM and SWE resources, as engineering culture is a vast, different area with <a href="https://github.com/sindresorhus/awesome/blob/main/readme.md#work">many Awesome lists</a>, <a href="http://www.engineeringladders.com/">frameworks</a>, <a href="https://www.levels.fyi/?compare=Google,Facebook,Microsoft&track=Software%20Engineer">levels</a>, university courses and <a href="https://blog.pragmaticengineer.com/reverse-interviewing/">career advisory</a>.<br /><br />Hence, the list only separates 3 main areas: Product, Operations and Customer-facing. There is no separation between “business” or “design” and “IT” <a href="https://esilva.net/articles/prodtech">like in most non-tech enterprises</a>. For sake of simplicity we assume it’s mainly tech where the <a href="https://www.mckinsey.com/business-functions/operations/our-insights/when-toyota-met-e-commerce-lean-at-amazon">product is lean</a>, with vertically integrated feature- or service teams. One could even argue operations should be part of Product, which it absolutely should, I just listed it separately because the functions listed are consolidated across many services and products.<div><br /><h3 style="text-align: left;">2. User success is your success</h3>For the longest time, IT has been treated by non-tech companies as a <a href="https://architectelevator.com/cloud/enterprise-non-cloud/">fixed asset (capital expenditure) and cost center</a>, typically outsourced to services companies. Recently this has changed to an enabler view - where IT is seen as <a href="https://lifesciences-resources.awscloud.com/aws-cloud-enterprise-strategy-blog/cloud-capex-and-opex-reframing-the-conversation">operating expenditure</a> to create <a href="https://aws.amazon.com/blogs/enterprise-strategy/revisiting-buy-vs-build-drawing-the-line/">more or faster value</a>. Often this is referred to as “digitalization”. <a href="https://architectelevator.com/cloud/enterprise-non-cloud/">Ideally</a> this would have changed the default commercial model to “pay-as-you-go” services, checking for <a href="https://blog.dataiku.com/intro-to-finops-for-ai">return on invest continuously</a>. In the old world, tech firms were driven by strict, fixed, annual licenses, basically still throwing a CD over the fence to the customer. Today, customers <a href="https://blog.airplane.dev/i-started-a-saas-company-in-2013-and-2021-heres-how-its-changed/">understand services and expect IT to grow</a> and move with their “lean” business dynamically, offering a <a href="https://www.sussna-associates.com/book">portfolio of digital services</a> with pricing on consumption. Tech companies with a consumer background are used to this flexibility and therefore thrive <a href="https://sloanreview.mit.edu/article/leadership-at-a-time-when-every-company-is-a-tech-company/">while traditional companies want to learn from them</a> (<a href="https://www.ben-evans.com/benedictevans/2021/3/18/outgrowing-software">until very recently</a>).<br /><br />This “agile” model can be a blessing or a curse (see No. 4 below), but it changes the role of services and tech firms from vendors or outsourcers to closer integrated co-innovators. Hence a surge in “value add” consulting services, often based on open source software. The trick is to avoid the <a href="https://www.blogger.com/blog/post/edit/6849806902838215926/9162246287969651593#">IBM trap</a>, turning a product company into a no-asset pure transaction business, which could become a <a href="https://www.theregister.com/2021/11/05/red_hat_jobs/">race to the bottom</a>. Contrast this with the <a href="https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/introducing-customer-success-2-0-the-new-growth-engine">Customer Success role</a> in pay-as-you-go tech firms which helps non-tech companies use the best and most efficient services for their business goal, almost like a management consulting function - these days sometimes sales account managers are called success managers and incentivized via long term consumption as well. On the other hand, consulting and design companies try to build product focus and <a href="https://hackernoon.com/6-examples-of-outsourcing-failure-yv223y4u">avoid becoming outsourcers</a> by aligning to the business vision and actually building (prototypes).Hence, the list doesn’t talk about “sales” of a fixed asset but assumes the product drives consumption, often starting with a free tier. Deeper commitments to the products (see No. 4 below) <a href="https://martinfowler.com/articles/oss-lockin.html">without lock-in</a> require a variety of customer-facing expert roles along the lifecycle of service usage.</div><div><br /><h3 style="text-align: left;">3. Teams learn through iteration, expertise comes from experimentation, and manager is not a promotion</h3><a href="https://en.wikipedia.org/wiki/Eating_your_own_dog_food">Dogfood</a> is a common technique in tech firms, using their own products to gain fast insights from experiments and improve accordingly. That’s a common differentiator to non-tech firms with long development cycles or simply products not everyone can use, so they have to rely on market research alone. In other words: <a href="https://blog.pragmaticengineer.com/project-management-at-big-tech/">Tech companies have iteration</a> and expertise from <a href="https://www.reforge.com/blog/scaling-product-delivery">experimentation with uncertainty</a> a.k.a <a href="https://hbr.org/2014/11/how-companies-can-profit-from-a-growth-mindset">growth built into their company culture</a>, which makes it hard to compare roles from <a href="https://hbr.org/2021/08/is-tech-the-right-career-for-you">tech and non-tech firms</a>. Outsourcing of core functions for instance is ideally for startups in incubation but sadly more often a sign a tech company is <a href="https://medium.com/swlh/https-medium-com-tomaspueyo-the-secret-to-picking-your-next-tech-employer-7869a5a9367e">not product focused anymore</a> or wants to <a href="https://twitter.com/GergelyOrosz/status/1457446667523866637">drive down cost</a> at the expense of innovation. That’s why the list doesn’t include industry (non-tech) enterprises, consulting firms, managed service providers, service delivery firms or startups.<br /><br />Iterations don’t only increase innovation, we also know that fast, safe, <a href="https://www.blogger.com/blog/post/edit/6849806902838215926/9162246287969651593#">iterations increase reliability</a> - your career should be the same. Building up a team as a manager is an exercise in reaching a product goal better or faster by bringing people together, not a promotion to a different rank of decision making. In Tech we often see a <a href="https://www.blogger.com/blog/post/edit/6849806902838215926/9162246287969651593#">pendulum between manager and individual contributor</a>, in startups <a href="https://www.hashicorp.com/blog/mitchell-s-new-role-at-hashicorp">even from CEO’s</a>. We also know <a href="https://www.infoworld.com/article/3639050/complexity-is-killing-software-developers.html">complexity</a> in systems keeps increasing. And the only way to deal with complexity is by <a href="https://dzone.com/articles/how-to-cope-with-complexity">observing</a> it, for which we create <a href="https://www.infoq.com/articles/open-sociotechnical-systems-design/">socio-technical systems</a> specialized on identifying patterns while state is emerging. For managers this means <a href="https://www.pearson.com/uk/educators/higher-education-educators/program/Poppendieck-Leading-Lean-Software-Development-Results-Are-not-the-Point/PGM973248.html">being adaptive and change context</a>, <a href="https://skamille.medium.com/the-management-flywheel-c076f398969b">identifying and changing the small things</a> iteratively, is the real art. It also means titles like “lead” or “manager” are often meaningless or, in the case of “senior” or “principal” vanity and don’t add value to a list of roles in my opinion.<br /><br />Take for instance various “ops” roles, starting from DevOps or SRE to <a href="https://www.finops.org/introduction/what-is-finops/">FinOps</a>, DevSecOps and <a href="https://dataopsmanifesto.org/en/">DataOps</a>. All of those value horizontal, end to end knowledge and <a href="https://www.amazon.com/Thinking-Systems-Donella-H-Meadows/dp/1603580557">systems thinking</a> instead of fixed <a href="https://www.rubick.com/how-to-build-silos-and-decrease-collaboration/">silos</a> (see point 2), <a href="https://www.nature.com/articles/s41562-021-01196-4?error=cookies_not_supported&code=95ed1634-db83-47f4-be4f-390930a08cff">especially after the pandemic</a>. Getting more experienced in those roles, to be “senior” for lack of a better term means <a href="https://letterstoanewdeveloper.com/2021/08/02/take-jobs-that-help-you-grow/">increasing breadth by changing roles laterally</a>, <a href="https://daedtech.com/how-developers-stop-learning-rise-of-the-expert-beginner/">while building out expertise and covering more skills</a>. Instead of looking for a role title when looking for a role, understanding the interactions and influence in the particular tech company and team is key. Developer Relations is a great example, it can range from evangelism and community work to product research, <a href="https://hackernoon.com/nerds-dont-respond-to-marketing-try-technical-documentation-instead">documentation</a> and even owning <a href="https://slab.com/blog/stripe-writing-culture/">company-wide engineering culture</a>.<br /><br />Apart from the vanity title problem there are simply a lot of <a href="https://neilkakkar.com/things-I-learned-to-become-a-senior-software-engineer.html">good lists</a> on <a href="https://skamille.medium.com/an-incomplete-list-of-skills-senior-engineers-need-beyond-coding-8ed4a521b29f">skills for “senior” engineers</a> and a lot of <a href="https://www.oreilly.com/library/view/the-staff-engineers/9781098118723/">literature</a> on <a href="https://lethain.com/finding-the-right-company/">“staff” roles</a>, even an <a href="https://github.com/kdeldycke/awesome-engineering-team-management/blob/main/readme.md">Awesome list on management</a> or for <a href="https://github.com/LappleApple/awesome-leading-and-managing#readme">leadership</a>. No need to create another.</div><div><br /><h3 style="text-align: left;">4. Cost is an SLA like any other, and you optimize for it</h3>As mentioned in point 2, ideally Pay-as-you-go is the standard commercial model in times of rapid change. But in reality it's usually combined with a subscription or other “commitment”. On the one hand because non-tech corporate accounting and procurement are slow to change and still think it’s better to “have” a software than use a service (which comes with free expert consulting) - a striking example of the Dunning-Kruger effect. On the other hand because tech companies also need to <a href="https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/unlocking-value-four-lessons-in-cloud-sourcing-and-consumption">plan their own cashflow to build future capacity</a>, and commitments keep customers close in an industry of often interchangeable products (e.g. <a href="https://www.swyx.io/cloudflare-go/">S3</a> and <a href="https://cloud.google.com/blog/topics/developers-practitioners/postgresql-interface-adds-familiarity-and-portability-cloud-spanner">Postgres</a> have by now become de facto standard interfaces for competitors). So they offer discounts for recurring use and payments ("Pay less for our 1-year plan!"). These days even Apple products are <a href="https://calpaterson.com/printers.html">barely usable without attached subscriptions</a> and full of dark patterns (“Pay a subscription instead of buying this song!”). Behind that is <a href="https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap-cloud-lifecycle-scale-growth-repatriation-optimization/">cost planning</a>, not anymore quarterly but based on customer or product lifetime value - the practice behind <a href="https://www.finops.org/introduction/what-is-finops/">FinOps</a>. In other words, <a href="https://martinfowler.com/articles/is-quality-worth-cost.html">cost</a>, like <a href="https://sre.google/workbook/reaching-beyond/#reliability-is-the-most-important-feature">reliability</a> or <a href="https://blog.nelhage.com/post/reflections-on-performance/">performance</a>, or <a href="https://cloud.google.com/blog/topics/sustainability/new-tools-to-measure-and-reduce-your-environmental-impact">sustainability</a>, is a feature, with an SLA, and <a href="http://www.cloudbombe.com/blog/finding-tracking-and-holding-teams-accountable-for-savings-opportunities-aka-cloud-waste/">one can’t hide from it anymore</a>.<br /><br />Hence in my list I've grouped the roles roughly in pre-commit, which is basically people trying to increase a customer's confidence into the value proposition of the product in order to attach a commitment to their pay-as-you-go model. And post-commit, which are people making sure the chosen “solution” (the word is a good indicator it’s a customer-facing role), that can be a product, technology or transformation (consulting), actually delivers that value so that customers expand the use of product across the ecosystem. You could argue the product build teams also fall in that category but for readability's sake I've taken them out.<br /><br />Take for instance the Solution Architect role. "Architect" in the enterprise sense sadly often means someone drawing hopelessly outdated <a href="http://weblog.tetradian.com/2016/03/30/architecture-as-boxes-lines-and-glue/">“boxes and arrows”</a> of technology (and organizational) components, in discussion with other "Architects" that define a business problem - <a href="https://www.mckinsey.com/business-functions/marketing-and-sales/our-insights/whats-wrong-with-solutions-selling-and-how-to-put-it-right">hence the "solution"</a>. Tech companies have “<a href="http://www.engineeringladders.com/TechLead-EngineeringManager.html">Tech Leads</a>” for internal products. Depending on the tech company's philosophy the Solution Architect may either be a pre-commit role, often as part of the sales team, especially if the product is "sticky" or expensive upfront, meaning hard to change, and customers want a "fixed" solution before committing (<a href="https://www.thoughtworks.com/en-sg/insights/articles/cloud-beyond-infrastructure-thinking">an illusion and fallacy of bad planning</a>). Or it can be a post-commit “<a href="https://databricks.com/blog/2020/04/28/new-study-databricks-delivers-nearly-29-million-in-economic-benefits-and-pays-for-itself-in-less-than-six-months.html">resident</a>” role helping customers to continuously improve their use of the product, especially if it's a highly flexible and fast moving platform, similar to a <a href="https://about.gitlab.com/handbook/customer-success/tam/)">Technical Account Manager (TAM) or Customer Success Manager (CSM)</a>. Depending on the target market audience it can be focused on a single customer, similar to a consulting role, in workshops, or more focused on mass-market consumer outreach, via publications and talks, more like developer relations. When you are offered such a role, try to find out how it is incentivized and what you can control or influence - and what not.<p><br /></p><p><br /></p><p><span style="font-size: x-small;">*) While it's good to give opportunities to everyone and not bias early, the focus on "upwards" leadership can also be problematic. Aside from the obvious problem with a hierarchical perception and rejection of hands-on expertise, it may simply lead to no one wanting to do <i>creating</i> instead of just <i>creative</i> work outside of startups - for instance I've had students ask me how long it takes to be a director "as I don't really like code" or not interested in network infrastructure "as it doesn't add value to my CV" or only finish a prototype "because outsourcing the coding is really easy". I feel this may as well add to imposter syndrome rather than reduce it.</span></p></div></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-29721759034891300182021-07-31T07:18:00.006+02:002022-01-24T08:44:57.848+01:00A view into state [backdated draft]<p></p><div style="text-align: left;"><p><i><span style="background-color: #fcff01;">Note</span>: This post is backdated to the date of the last draft (31 Jul 2021) as I <a href="https://unarchitectedsystems.blogspot.com/2022/01/letter-to-my-future-self-2022-edition.html">changed my job and role</a> and didn't want to bias / inform myself by that. It's an unfinished fragment of my thinking at that point in time that I just cleaned up a little and added references where necessary, but it's still rough and incomplete.</i></p></div><div style="text-align: right;"><i>In principle, any stream processor could be used for materialized views</i></div><div style="text-align: right;">Kleppmann, 2017, p. 467</div><p></p><p>Martin Kleppmann recently gave a <a href="https://martin.kleppmann.com/2021/07/02/debs-keynote-thinking-in-events.html">great keynote</a> in which he summarized his ideas under a new framework of unordered vs. ordered events and mutable vs immutable state. In the <a href="https://dl.acm.org/doi/pdf/10.1145/3465480.3467835">accompanying paper</a> he also gave an outlook on a new wave of tools that "abstract away" these consistency decisions in event-driven systems, in particular "Materialized Views" e.g. Materialize - a great overview on the different patterns is <a href="https://www.scattered-thoughts.net/writing/an-opinionated-map-of-incremental-and-streaming-systems">Jamie's article</a>.</p><p>I've been a fan of this idea not only since the famous "database turned inside out" (1) quote but actually when I first worked with <a href="https://research.google/pubs/pub45855/">Spanner</a> and saw firsthand how it turned <a href="https://research.google/pubs/pub46103/">into an SQL system</a> which required a surprising amount of abstraction of internal (consistency) concepts. Most surprising was that fundamentally Spanner is a messaging system (in other words it is truly distributed and not built on replication paradigms), and could even be used as a queue by observing changes "inside out". In most databases those changes are row-based and <a href="https://debezium.io/blog/2020/02/10/event-sourcing-vs-cdc/">often used (with CDC) as basis for event sourcing</a> architectures, sadly mixing physical considerations with <a href="https://openpracticelibrary.com/practice/event-storming/">domain events.</a> But I am interested in one level lower, of the changes <i>within</i> a domain event entity, and lineage, the reason why. This interest comes from Dataflow / Beam and its <a href="https://beam.apache.org/blog/timely-processing/">consistent view of temporal locality</a>, something I've been keen on since agent-based simulation using Erlang's actor inbox (2). Beam doesn't use "wall clock" time but time in the sense of <a href="https://dl.acm.org/doi/abs/10.1145/3401025.3404088">event-time</a>, <a href="https://medium.com/digital-anatomy/why-is-flow-important-909f2d9fc9bf">flow</a> or an <a href="https://dl.acm.org/doi/10.1145/359545.359563">ordered sequence of events</a> or the spacetime / <a href="https://arxiv.org/abs/1907.05636">promise-theory goal</a> "<i>to explain a process ... we need to be able to ... tell a story about its behaviours</i>". I'm interested in exposing this state and making it legible and explainable.</p><p><br /></p><p>Wouldn't it be great to have a database supporting per row state (consistency) metadata instead of state hidden in a stream processing system?</p><p><br /></p><span><a name='more'></a></span><p><br /></p><p>What I like would be <a href="https://github.com/arkhipov/temporal_tables">temporal tables</a> but <a href="https://queue.acm.org/detail.cfm?id=3480470">causal consistency</a> -ordered (e.g. Lamport or Vector Clocks), not only the data but the state of the row as metadata. The standard answer to this are Saga's in Microservice architecture. However usually Saga's are more seen like a <a href="https://jbossts.blogspot.com/2017/06/sagas-and-how-they-differ-from-two.html">variant of a two-phase commit</a>, often in a strict workflow engine, a way to <a href="https://queue.acm.org/detail.cfm?id=3321612">achieve consistency</a> as in the proverbial shopping cart or payment example. How the internal state of the workflow of a saga is managed usually stays opaque or a UI gimmick like <a href="https://aws.amazon.com/step-functions/">Lambda's Step Functions</a> or <a href="https://blog.cloudflare.com/introducing-workers-durable-objects/">Cloudflare Stateful Workers</a>. Usually consistency here refers to a workflow, and the domain object is created as an <i>output</i> of that workflow.</p><p>A better example than sagas for what I mean is the <a href="https://lmax-exchange.github.io/disruptor/disruptor.html">LMAX Disruptor</a> (which is also actor-inspired) or in general trading systems where one object (the transaction) needs to be enriched. For instance risk scores, party data or cost factors are loaded into the object to make it auditable, stateless and independently processable. This way the object can be sent to different engines <a href="https://en.wikipedia.org/wiki/Embarrassingly_parallel">embarrassingly parallel</a> and "reassembled" by a <a href="https://en.wikipedia.org/wiki/Reactor_pattern">Reactor</a> (the reactive version of Balking or more general <a href="https://blog.frankel.ch/teeing-java-api/">Teeing</a>). In trading or bidding systems the reactor may also decide to throw away the object to save resources before all subprocesses return, e.g. load shedding if it's unlikely generate revenue or likely to be fraud (the object may be sent to a fast machine learning model in parallel). This "concurrent <a href="https://en.wikipedia.org/wiki/Chain-of-responsibility_pattern">chain of responsibility</a>" is also a common pattern in Go's channels where it's called the <a href="https://www.oreilly.com/library/view/concurrency-in-go/9781491941294/ch04.html">"Fan Out, Fan In" pattern</a>.</p><p>But the use case for this are much wider, basically any reactive user interface, including APIs that hide state transitions - what's called the <a href="https://microservices.io/patterns/apigateway.html">Gateway API</a>, Aggregator or Gateway Aggregation pattern pattern. For instance to show a user profile page in the frontend dozens of services may need to be called - yet the profile itself should appear consistent to the user when shown (who doesn't hate when lists change while elements are being loaded for instance). Or an API might allow to update the same profile, yet behind the scenes change many services (which would be a saga). This is essentially the same problem as <a href="https://jepsen.io/consistency">external serializability</a> and <a href="https://martin.kleppmann.com/2020/07/06/crdt-hard-parts-hydra.html">CRDT's converging to it</a>, which is Kleppmann's research area. At the same time, I want to avoid reinventing distributed object (CORBA...) and transaction managers, state transitions should not be enforced but by itself an optional, observable event. Most importantly, I want a trigger (not in the database sense) that <i>independently</i> moves the state based on rules ("all required fields have values now"), for instance sending a message to another service (like an actor).</p><p>Currently most solutions I've seen basically use key/value stores with some kind of <a href="https://brandur.org/idempotency-keys">idempotency key</a> (<a href="https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header-00">RFC</a>) queried with every transition and the moving forward in a lock-free read-modify-write instruction, for instance RocksDB has a <a href="https://github.com/facebook/rocksdb/wiki/Merge-Operator">merge operation</a>. In particular Redis because they offer hashes and sets with field-level mutations, but they also actively participate in <a href="https://redis.com/blog/diving-into-crdts/">CRDT research</a> and similarly to Spanner have <a href="https://redis.io/topics/pubsub">messaging support</a>. But these databases either hold state in memory or don't allow granular time travel - in particular they don't have a way to attach an external state to row metadata, e.g. "this row is consistent as of X" or "this row has all mandatory fields set" or "the uncommited read version of this row is 60% dirty" or "this row is archived".</p><p>The other solution is to rely on event sourcing (usually Kafka) and use materialization to create a <i>view into state</i>. Modern, fast query engines like <a href="https://www.dremio.com/architectural-analysis-why-dremio-is-faster-than-any-presto">Dremio</a> (Apache Arrow is nice) in the Hadoop world, or the opposite, in-memory real-time Streaming databases like <a href="https://www.singlestore.com/">SingleStore MemSQL</a> offer fast insights but only in small, optimized use cases - think materialized views in OLAP, either because the Stream processing (e.g. Spark SQL) is microbatched or in-memory not scaleable. Data warehouses like <a href="https://cloud.google.com/blog/products/data-analytics/bigquery-materialized-views-now-ga">BigQuery</a>, <a href=" https://aws.amazon.com/glue/features/elastic-views/">AWS Elastic Views</a> and <a href="https://blog.cloudera.com/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/">Impala</a> added such functionality similar to <a href="https://github.com/MaterializeInc/materialize">Materialize</a> which <a href="https://news.ycombinator.com/item?id=22359769">adds some consistency</a> with a unified model by using <a href="https://github.com/TimelyDataflow/timely-dataflow">timely dataflow</a> (<a href="https://blog.acolyer.org/2015/06/12/naiad-a-timely-dataflow-system/">Naiad</a>) instead of shuffling and thereby offers snapshot consistency. But it abstracts away time, again relying on <a href="https://materialize.com/docs/overview/architecture/">CDC</a> from sources (relying on the physical data model) and <a href="https://news.ycombinator.com/item?id=25867693">ignoring late events</a>. Fundamentally that's still (in-memory) OLAP but at least with control without nightly ETL / ELT jobs. BigQuery had <a href="https://cloud.google.com/bigquery/docs/table-snapshots-intro">time travel and snapshots</a>, dbt (which works <a href="https://materialize.com/introducing-dbt-materialize/">with Materialize</a>) can write a consistent <a href="https://docs.getdbt.com/reference/dbt-jinja-functions/invocation_id">Invocation ID</a> and CosmosDB allows <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/consistency-levels">choosing consistency models</a>, it's just a different way of hiding that ETL / ELT. <a href="https://towardsdatascience.com/data-observability-the-next-frontier-of-data-engineering-f780feb874b">Data Observability</a> defines freshness SLOs for these kinds of approaches but <a href="https://lironshapira.medium.com/data-denormalization-is-broken-7b697352f405">(de)normalization</a>, especially in microservice architectures / <a href="https://martinfowler.com/articles/data-mesh-principles.html#DataAsAProduct">data meshes</a>, is the tricky part because (incremental) recomputation becomes NP-complete. I guess that's why fast In-Memory Databases (IMDB) and In-Memory Data Grids (IMDG) like HANA don't even really have <a href="https://www.sap.com/sea/products/hana.html">Streaming ingestion</a>, instead they favour (distributed) transaction semantics where locks are explicit.</p><p>Another options are continuous query models, maybe with <a href="https://link.springer.com/article/10.1007/s00778-003-0095-z">Aurora</a> as the first one - not the AWS Aurora, when it comes to streams academia had a headstart. Basically the argument that the <a href="https://nchammas.com/writing/data-pipeline-materialized-view">pipeline (with snapshots) <i>is</i> the view</a>. Maybe that's why Kafka (or better <a href="https://vectorized.io/redpanda">Vectorized Redpanda</a>) offers real-time querying over streams by using Kafka itself as database, which is why they call it <a href="https://www.jesse-anderson.com/2019/10/why-i-recommend-my-clients-not-use-ksql-and-kafka-streams/">ksqlDB</a>, but still just uses shuffling on whatever data is available and doesn't really expose a relational model or time in the sense of a consistent snapshot. But it still doesn't <i>explicitly</i> expose time in the way <a href="https://cloud.google.com/spanner/docs/true-time-external-consistency">Spanner</a> does or with the flexibility of reasoning about late events like <a href="https://beam.apache.org/documentation/programming-guide/#watermarks-and-late-data">Beam</a> (and Dataflow) allows you to. I guess another option would be to simply to expose time is use a <a href="https://docs.timescale.com/latest/tutorials/continuous-aggs-tutorial">timeseries database and aggregate in there</a>, or the opposite, <a href="https://medium.com/@vlapiner/spreadsheet-is-a-software-development-paradigm-70c871ff5f49">calculate upon every change, in Spreadsheet-style</a>, or to <a href="http://charap.co/reading-group-strong-and-efficient-consistency-with-consistency-aware-durability/">change consistency guarantees for reads </a>and timestamp them (like Spanner). But it's not time alone (and time itself can be a problem) rather the state of the value that's interesting.</p><p>I'd love to write such a database, and given Go has strong patterns I think <a href="https://nakabonne.dev/posts/write-tsdb-from-scratch/">Ryo Nakao's approach of a time-series database</a> would be a great starting point.</p><p><br /></p><p><span style="font-size: x-small;">1) Martin Kleppmann published the legendary "<a href="https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/">turning the database inside out</a>" in March 2015, followed by his <a href="https://martin.kleppmann.com/2015/05/27/logs-for-data-infrastructure.html">seminal posts on logs</a> in May, in June Google <a href="https://research.google/pubs/pub43864/">released the Dataflow paper</a> and in August Tyler widely published <a href="https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/">the core concepts of Dataflow</a> which defined now standard terms like "unbounded" and various windowing strategies, and a year later <a href="https://beam.apache.org/">Apache Beam </a>has its first release. The respective books followed <a href="https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/">2017</a> and <a href="https://www.oreilly.com/library/view/streaming-systems/9781491983867/">2018</a>.</span></p><p><span style="font-size: x-small;">2) Reminds me of Software-Transactional Memory (STM), a <a href="http://joeduffyblog.com/2010/01/03/a-brief-retrospective-on-transactional-memory/">failed experiment</a> of the early 21st century. Some of the ideas of the Actor Model like chronological encapsulation to facilitate time travel and transactions and <a href="https://dl.acm.org/doi/10.1145/960116.54016">promise</a>-based dataflow pipelines made it into concepts such as <a href="https://docs.racket-lang.org/goblins/intro.html">Goblins</a> and <a href="https://casmodeling.springeropen.com/articles/10.1186/s40294-019-0067-9">Agent-based Modelling</a> for simulations (fun fact OOP was invented for simulations).</span></p>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-18005489310618166182020-05-31T12:26:00.004+02:002020-06-01T08:12:00.575+02:00Observability, Debt or the Bret-Victor-Ization of distributed systems<div><font size="2">I've been thinking how the different way of conceptualizing cost (in the broader sense of <i>investment</i>) in Cloud changes the tech debt <a href="https://www.youtube.com/watch?v=JrjbX-KX6wQ">metaphor</a>. It never was a <a href="https://ronjeffries.com/articles/015-11/tech-debt/">good metaphor</a> to start with and allowed too many excuses, but I like the idea of expressing a suboptimal, incomplete or leaky level of abstraction somehow, as a dialectical, critical tool. I like declarative systems because they allow comparison of state over time, but they do limit expressiveness of our mental models, and omissions, when writing down code. Debt, with its pseudo-quantifiable touch is such a mental model limit. No one wants to keep ADR's for each of these*. How to solve this?</font></div><div><font size="2"><br /></font></div><h3 style="text-align: left;"><font size="2">Product-as-code</font></h3><div><font size="2"><a href="https://twitter.com/janpeuker/status/1264120465318019073">It's exciting to see</a> how the next step in polyglot programming is taking stacks apart and designating a layer to developer experience for <a href="http://signalvnoise.com/posts/3567-a-refresher-course-in-empathy">humans</a> based on “<a href="https://www.youtube.com/watch?v=HBqCpWldPII">progressive disclosure of complexity</a>”, and how we argument for this is feedback time or, in other words, <a href="https://itrevolution.com/book/accelerate/">Software Delivery Performance</a>. What a16z recently called <a href="https://a16z.com/2020/04/30/figma/">The Decade of Design</a> (but <a href="https://howtospeakmachine.com/2019/11/17/the-temple-of-design-doesnt-rule-this-century/">combined with craft and lean</a>), the best example probably being Stripe which shows that <a href="https://stripe.com/en-sg/payments/elements">beautiful API's</a> and <a href="https://medium.com/the-oxford-comma/stripes-new-api-reference-documentation-e6b1fc46896d">documentation</a> and a <a href="https://leerob.io/blog/how-stripe-designs-beautiful-websites">beautiful website</a> (and <a href="https://press.stripe.com/">beautiful books</a>) might after all correlate, and maybe because they take empathy to their heart (no surprises here for anyone who has seen or used PayPal). </font></div><font size="2"><br /><a href="https://twitter.com/janpeuker/status/1264120465318019073">When I first saw this</a> at Google working with <a href="https://research.google/pubs/pub36632/">Dremel</a> / <a href="https://research.google/pubs/pub41344/">F1</a> and all the <a href="https://martinfowler.com/articles/data-monolith-to-mesh.html">Data Mesh</a> <a href="https://research.google/pubs/pub47845/">tools</a> around it, an ontological, rhizomatic approach to data - without oversight, yet with structure, as side effect of, essentially <a href="https://en.wikipedia.org/wiki/Domain-driven_design">DDD</a> (or <a href="https://en.wikipedia.org/wiki/Alain_Badiou#Mathematics_as_ontology">set / category theory, referring to ontology</a> below). And the same when seeing Borg and the <a href="https://linkerd.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-one/">Service Mesh</a> around it. Both were built as <a href="https://www.thoughtworks.com/radar/techniques/applying-product-management-to-internal-platforms">products with a builder focus</a>, with "a builder" meaning <a href="https://www.google.com/about/philosophy.html"><i>universally</i> <i>anyone</i></a> who wants to <a href="https://gist.github.com/onlurking/fc5c81d18cfce9ff81bc968a7f342fb1">build something, meaning to contribute to a shared idea</a>. When we say everything-as-code we need to go beyond engineering components, we, and that means everyone and the maximum of diverse perspectives, need to look at the product and all of its users. Similar to medical doctors who moved away from seeing "man as machine", or architects seeing the city or the house as a machine, we are slowly moving away from seeing a software system as a machine, perfectly controllable. <a href="http://www.appropriatingtechnology.org/?q=node/296">Technology is not neutral</a>, and a constant process and struggle that goes far beyond engineering.</font><div><font size="2"><br /></font></div><div><span><a name='more'></a></span><div><h3 style="text-align: left;"><font size="2"><br /></font></h3><h3 style="text-align: left;"><font size="2">Understanding the product</font></h3><font size="2">
With <a href="https://charity.wtf/2020/03/03/observability-is-a-many-splendored-thing/">observability</a> our systems become not only monitorable or measurable but tangible to every part of the ever-emergent (just as <a href="https://axispraxis.wordpress.com/2020/05/27/what-is-emergence-and-why-should-we-care-about-it/">ontological)</a> socio-technological system, and they give us better feeling of time, to avoid bias like Reification and Goodhart's law</font><span style="font-size: small;">. In other words, it establishes a way to talk about the <a href="https://terenceblake.wordpress.com/2014/10/16/badiou-on-objects-and-relations-1-object-oriented-vs-relation-oriented/">ontological difference</a> between what we things our systems are and what they really are by leveraging <a href="https://blog.usejournal.com/monoids-to-groupoids-492c35105113?gi=1d1c4b826d5d">category / set theory</a> (<a href="https://twitter.com/markburgess_osl/status/1257967611943571456?s=20">maybe?</a>) or <a href="https://unarchitectedsystems.blogspot.com/2019/10/operating-manual-for-ship-of-theseus.html">OOO</a>.</span></div><div><span style="font-size: small;"><br /></span></div><div><span style="font-size: small;">We can check those biases by seeing the effect of our ideas in the real world (the production system) and iterating on the real effect of systems: The </span><a href="https://www.youtube.com/watch?v=PUv66718DII&feature=emb_logo" style="font-size: small;">Bret-Victor-Ization</a><span style="font-size: small;"> of distributed systems.</span></div><div><span style="font-size: small;"><br /></span></div><div><span style="font-size: small;">Very much in the spirit of SRE's </span><a href="https://cloud.google.com/blog/products/management-tools/tune-up-your-sli-metrics-cre-life-lessons" style="font-size: small;">focus on user happiness</a><span style="font-size: small;"> or </span><a href="https://twitter.com/mipsytipsy/status/1226182853148340224" style="font-size: small;">user pain</a><span style="font-size: small;"> - in other words empathy. Systems are for everybody and need to involve everybody, all of its <a href="https://xkcd.com/1172/">unexpected</a> users (something <a href="https://youtu.be/MbX_6MsjsIE?t=387">real world architecture finally starts to do</a>). It's worded a little technical but "</span><a href="https://www.infoq.com/articles/slos-engineering-team-API/" style="font-size: small;"><i>SLOs are the API for Your Engineering Team</i></a><span style="font-size: small;">", </span><font size="2">where API essentially means <a href="https://en.wikipedia.org/wiki/OKR">OKR</a> and SLO means promise to all users, is a catch phrase I like.</font></div><div><font size="2">
<br />But what I really like about observability is, that is also shows us there is not only one system - the one in the flowchart, in the architecture PowerPoint, but there is many, constantly emerging ones that are in the heads of all users. "<a href="https://medium.com/lightstephq/how-deep-systems-broke-observability-and-what-we-can-do-about-it-7e8b2ad6a16f">Deep</a>" Systems are <a href="https://www.usenix.org/conference/nsdi20/presentation/hauer">never fully up or down</a> and us humans merely <a href="https://twitter.com/davewallsworth/status/1224592024772534272">hint intents</a> at the complex, speculative, <a href="https://unarchitectedsystems.blogspot.com/2018/04/meshup.html">declarative</a> system, which turns our relationship with the system from a fixed state SDLC into, essentially, <a href="https://twitter.com/markburgess_osl/status/1176005392377270272">promise theory</a>, read Mark Burgess 2019:<br />
</font><blockquote class="tr_bq">
<i><font size="2">It's shown that the concepts of statefulness or statelessness are artifacts of observational scale and causal bias towards functional evaluation. If we include feedback loops, recursion, and process convergence, which appear acausal to external observers, the arguments about (im)mutable state need to be modified in a scale-dependent way. In most cases the intended focus of such remarks is not terms like `statelessness' but process predictability. </font></i></blockquote><font size="2">
The good news is that we've learned a lot of concepts of deep systems from <a href="https://www.newyorker.com/science/elements/how-the-artificial-intelligence-program-alphazero-mastered-its-games">recent advances in Machine Learning</a> (ML). In ML, Goodhart's law is called <i>overfitting</i>. ML's dropout regularization we call canary or A/B testing, embeddings can be understood as modularization or bounded context (let's not discuss whether that means sets / categories), and so forth. I'm not proposing to use ML to 'learn' the architecture of a product, but understand the research in those techniques to come up with our own tools to </font><a href="https://en.wikipedia.org/wiki/Explainable_artificial_intelligence" style="font-size: small;">explain</a><font size="2"> the real structure of systems as a discursive, critical tool, not as ideology. Coming back to my experience with <a href="https://research.google/pubs/pub36632/">Dremel</a>, I was stunned how normal and easy it was to <a href="https://cloud.google.com/bigquery-ml/docs/bigqueryml-intro?hl=en">embed ML into it</a> - for instance to test a hypothesis against a large body of logs by training an explainable <a href="https://en.wikipedia.org/wiki/Random_forest">random forest</a>.</font></div><div><font size="2"><br /></font></div><div><font size="2">In that sense, a <b>product is just promise</b> (an old marketing adage, like a product is a story that people personally want to be fulfilled). Like quality is defined in the ISO sense as the sum of expected properties, and the extent to which they are fulfilled in reality (and maybe how close marketing comes to that, or goes beyond). When we decide for a product, we instinctively trust it (with some bias), and we accept the promise of its quality and value. Internal understanding of a product is therefore like quality management. Unsurprisingly, <a href="https://cloud.google.com/blog/products/management-tools/meeting-reliability-challenges-with-sre-principles">SRE very much acts like the steward of quality expectations</a> and promises of its value, the SLO. Which, I guess, is why ITIL recently introduced a <a href="https://www.axelos.com/itil-4-concepts">Service Value System</a>, too (I'm skipping the "Value Chain" here, seems they couldn't completely detach from Taylorism / Fordism internally). Given that a product is a promise, and so are the relationships in our system, we can conclude that an ontological analysis of the whole system is possible - as long as everyone is included, the socio-technical system is open for participation, and we aim to reduce bias a much as possible.<br /></font><h3 style="text-align: left;"><font size="2"><br /></font></h3><h3 style="text-align: left;"><font size="2">Debt as promise</font></h3><div><font size="2">Recently I stumbled over a fascinating analogy when reading 3 books in parallel which surprisingly touched on different subjects in a similar way, the topic of debt - or the topic of promises:</font></div><div><font size="2"><br /></font></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgthlCBaM39eChkN_c-DZmQZM5-aKDxbJMdV0xwmPKxQOPIbVenbNoWfXfQLu4XHQ8jFWprA4Vtf4tqYnwML8jRQxiHlBxBvYswmdyFn7kFeHefAFqWALEZFZbVXGZMMFPJamlZJRdiIlT9/" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="161" data-original-width="303" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgthlCBaM39eChkN_c-DZmQZM5-aKDxbJMdV0xwmPKxQOPIbVenbNoWfXfQLu4XHQ8jFWprA4Vtf4tqYnwML8jRQxiHlBxBvYswmdyFn7kFeHefAFqWALEZFZbVXGZMMFPJamlZJRdiIlT9/" /></a></div><font size="2"><br /></font></div><div><font size="2"><br /></font></div><div><font size="2">The first book is <a href="https://en.wikipedia.org/wiki/Technological_Revolutions_and_Financial_Capital">Carlota Perez' "Technological Revolutions and Financial Capital</a><a href="https://en.wikipedia.org/wiki/Technological_Revolutions_and_Financial_Capital">"</a>, which I <a href="http://reactionwheel.net/2015/10/the-deployment-age.html">had read about</a> and was recommended a few times, but was afraid it would be too schumpeter-hype-cycle-idealistic-technological-deterministic. I picked it up again after a <a href="https://alexdanco.com/2020/02/07/debt-is-coming/">post on </a><a href="https://alexdanco.com/2020/02/07/debt-is-coming/">debt in technology</a><span style="font-size: small;">. Yes, it is clearly pre-financial-crisis, pre-precarious-employment and western, but with the open source world being predominantly western, it provides an interesting insight how governments and societal changes drive consolidation, in particular when she talks about capital, debt and technology dualism. The second book, "</span><a href="https://cup.columbia.edu/book/resistance-and-the-politics-of-truth/9783837639070">Resistance and the politics of truth" by Iain MacKenzie</a><span style="font-size: small;">, a random Google find when searching for how Deleuze's and Focault's concepts of violence and power relate and how that influences bias in socio-technological systems, in particular how to transform systems from within using <i>resistance</i> instead of revolution or reform. The concept of bias and how it affects </span><a href="https://hbr.org/2003/07/delusions-of-success-how-optimism-undermines-executives-decisions">risk</a><span style="font-size: small;"> started with </span><a href="https://en.wikipedia.org/wiki/The_Death_of_the_Author">Barthes</a><span style="font-size: small;"> and </span><a href="https://en.wikipedia.org/wiki/What_Is_an_Author%3F">Focault</a><span style="font-size: small;">, popularised in our software craft by the Agile movement and </span><a href="https://www.methodsandtools.com/archive/agilesoftwarearchitecture.php">Fairbanks</a>, MacKenzie talks about how embracing uncertainty helps to counter systems based on hegemonic truth of the "<i>algorithmic control society</i>"</font><span style="font-size: small;">. And finally the third book, <a href="https://www.sup.org/books/title/?id=30800">Giorgio Agamben's "Creation and Anarchy"</a> which I found when stumbling over Benjamin again in a Critical Theory book and wondered how he would relate to Max Weber**. Agamben explains how debt withholds anarchy and limits resistance, what </span><span style="font-size: small;">Benjamin originally calls a "</span><i style="font-size: small;">verschuldeten Kultus</i><span style="font-size: small;">", meaning not only owing to but </span><i style="font-size: small;">indebted</i><span style="font-size: small;"> to a cult of dependencies - the debt is the cult. Agamben writes that "</span><i style="font-size: small;">capitalism has no beginning or foundation</i><span style="font-size: small;">" because it is "</span><i style="font-size: small;">the anarchy of power</i><span style="font-size: small;">". <b>If debt, or non-debt, becomes a cult, it </b></span><b><span style="font-size: small;">tranquillizes</span><span style="font-size: small;">, while freedom lies in "</span><i style="font-size: small;">chance and uncertainty</i></b><span style="font-size: small;"><b>"</b>.</span></div><div><font size="2"><br /></font></div><div><font size="2">When reading <a href="https://en.wikipedia.org/wiki/Debt:_The_First_5000_Years">Graeber's "Debt"</a> long ago I remember myself nodding and agreeing that indeed it was debt, not money, that was created first, because debt is an <i>obligation to reciprocity</i>. In other words, debt is also just a promise - an asymmetric, slow one, with risk (cost) attached. So in an ontological sense we need to look at categories, not just sets, to <a href="https://terenceblake.wordpress.com/2014/10/16/badiou-on-objects-and-relations-1-object-oriented-vs-relation-oriented/">model relations between events (like in OOO)</a>. The relation here is trust and risk, or happiness and pain in SRE teams. Debt is why <a href="https://en.wikipedia.org/wiki/Tit_for_tat#Game_theory">tit-for-tat</a> is a surprisingly efficient strategy in the Prisoner's dilemma. Especially when compared to <a href="https://en.wikipedia.org/wiki/The_Gift_(essay)">Maus' Gift</a> as a form of community violence or power, which Graeber of course extended to the idea that the <a href="https://en.wikipedia.org/wiki/Seeing_Like_a_State">state is essentially enforcement</a> of debt.</font></div><div><font size="2"><br /></font></div><div><font size="2">Coming back to Perez' model, that's when "<i>unfulfilled promises had been piling up</i>" after the hype cycle, which leads to speculation, a debt crisis, "power seeking", and finally regulation stepping in. Sounds to me like limiting emergence in software systems by creating an "Architecture" institution enforcing rules, which then cause less resilience, and worse user experience. In our case, that means our socio-technical systems have to optimize for user trust and reduce risk, ideally without enforcement from institutions, to avoid the "<i>anarchy of power</i>" of Agamben or the "<i>power seeking</i>" of Perez or MacKenzie's / Badiou's / Focault's "<i>politics of truth</i>"</font><span style="font-size: small;">, which are the first steps towards a </span><a href="https://www.theonion.com/protestors-criticized-for-looting-businesses-without-fo-1843735351" style="font-size: small;">complete loss of faith</a><span style="font-size: small;">. That's exactly what SRE's do and good open source projects do, help making the system more resilient from within. SRE introduces the chance for <a href="https://landing.google.com/sre/sre-book/chapters/embracing-risk/">resistance using error budgets</a>. SRE embrace uncertainty by not getting trapped by queuing. There is not one truth in an SRE system, there is only the art of influence, of user empathy. Systems decay, but engineers are not blamed for debt. Instead of a debt/credit metaphor, toil is used to express a relationship between people. <b>Following SRE is <i>architecting</i> without architects</b>.</span></div><div><span style="font-size: small;"><br /></span></div><div><span style="font-size: small;">What does that mean for tech debt in general? </span><span style="font-size: small;">Is tech debt a good thing, a bad thing? Does it always have to be resolved, or only </span><a href="https://blog.pragmaticengineer.com/tech-debt/" style="font-size: small;">inventorized</a><span style="font-size: small;">? Who gets to define what counts as debt and what as quality, and why does that person or institution have power over this decision?</span></div><div><span style="font-size: small;"><br /></span></div><div><span style="font-size: small;">Given my learnings above, I am careful to enforce resolution of tech debt. Especially when there is no clear user happiness or pain impact. User trust debt needs to be resolved first, and that includes resilience debt, supportability debt or brand debt. Actually I don't like the term debt, because it implies it 'belongs' to someone, it reinforces the hegemonic power. A decision, a tradeoff, toil can be documented without implying a positive or negative value. The point has to be to resist this power position, and establish a culture of caring for the user and for the system. This requires a degree of uncertainty to allow creativity. A system that lacks trust, by its users, including operations, for instance in failing or malfunctioning releases or features, enforces wrong incentives. It will lead to power grab and survival of the fittest, to focus on meaningless metrics to hold onto positions of power, it loses track of the original story and credibility that your users wanted to be fulfilled. Therefore everyone has to be able to observe the system from all of its angles, and that observability has to be transparent and democratic for all users, not just operators or developers. Only this way, a consensus in what elements constitute</span><span style="font-size: small;"> risking user trust can be formed, at which level and where uncertainty is too much.</span></div><div><font size="2"><br /></font></div><div><font size="2">That's why I am careful to advise Design Thinking as a general remedy. This instance of Lean favours very fast feedback cycle and is therefore well suited for the fast <a href="https://en.wikipedia.org/wiki/Attention_economy">attention economy</a>. User empathy, and empathy for each other, and understanding are not fast, they are not as quantifiable as debt, but more importantly they are not <i>engineered. </i>They require non-engineering users to have a say. There are, at best, proxy metrics for such feedback.<i> A</i>n assumption Design Thinking, with its legacy from Cybernetics and Control Theory barely hides. Tech debt fits Design Thinking because it's an output, it gives the impression a prototype is fine as long as revenue is higher than debt - but that's not always true. Trust, sustainability and empathy cannot be offset by debt. I like feedback loops, but prefer to investigate <a href="https://en.wikipedia.org/wiki/Seven_generation_sustainability">larger systemic feedback</a> and <a href="https://fs.blog/2016/04/second-order-thinking/">second over thinking</a> over direct control loops - in other words <i>observing</i> and <i>resisting</i> rather than exerting power. <b>Resilience comes from resisting the temptation to move fast and break things</b>, see what <a href="http://leanmagazine.net/issues/issue-6/getting-flow-into-your-product-development/">flow the </a></font><font color="#0000ee" size="2"><u>socio-technological</u></font><a href="http://leanmagazine.net/issues/issue-6/getting-flow-into-your-product-development/" style="font-size: small;"> system can sustain</a><span style="font-size: small;">, and do what's best for the user, what fulfils</span><span style="font-size: small;"> the product promise.</span></div><font size="1"><br /><br />*) I've recently been building a hacky prototype how pairing conversation could be stored alongside code. I got the idea from <a href="https://www.blogger.com/#">speech-to-text</a> combined with an <a href="https://www.blogger.com/#">Eliza</a> like interface to drill down based on questions rather than free association, where the questions are based on NLP <a href="https://www.blogger.com/#">Keyphrase extraction</a>.<br /><br /><br />**) Since reading Max Weber, I have been fascinated by the similarities of <a href="https://www.blogger.com/#">Puritan</a> to Folk Chinese / Taoist idealist promises of a <a href="https://www.blogger.com/#">luck and bias-free</a> meritocracy and the <a href="https://www.blogger.com/#">equivalence of prosperity, fortune and status</a> which continuously biases the system towards the more powerful. In particular the <a href="https://www.blogger.com/#">similarity</a> in manicured meritocracy between the <a href="https://www.blogger.com/#">Anglo-Saxon University System</a> and the <a href="https://www.blogger.com/#">Imperial Exams</a> which still cast a mythological shadow on the <a href="https://www.blogger.com/#">NCEE</a>.</font><br /></div></div>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-19244821071631171582019-10-19T16:40:00.002+02:002019-10-19T16:40:56.488+02:00Operating Manual for the Ship of Theseus<span id="docs-internal-guid-818bc9b5-7fff-0e35-4966-d33b7c4994a8"></span><br />
<div style="text-align: right;">
<a href="https://frieze.com/article/too-much-too-fast"><i>science has always been jealous of art</i></a> </div>
<br />Over the last 5 or more years I've had kind of an abstinence from conferences and software architecture books. Industry focus was on Cloud, Serverless and ML, leaving system design stalling, with the occasional, rare exception (<a href="https://github.com/knative/serving">KNative</a>, <a href="https://dl.acm.org/citation.cfm?id=3196909">Learned Structures</a>, <a href="https://isa-principles.org/">ISA</a> and <a href="http://www.codingthearchitecture.com/authors/sbrown/">Simon's ongoing quest for explainability</a> come to mind).<br /><br />Conference speakers <i>still</i> explain Agile, DevOps, <a href="http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions">ADR's</a>, <a href="https://unarchitectedsystems.blogspot.com/2011/05/event-driven-web-applications.html">EDA</a> and <a href="http://erlang.org/download/armstrong_thesis_2003.pdf">Resilience</a>, while people <i>still</i> pile up tech debt and <a href="http://www.laputan.org/mud/">big balls of mud</a>, just now using Serverless or Kubernetes. The <a href="https://pragprog.com/book/tpp20/the-pragmatic-programmer-20th-anniversary-edition">20 year anniversary edition of The Pragmatic Programmer</a>, which I hold in my hands, says it is “<i>just as relevant today as it was back then</i>”. Given that I <a href="https://www.quora.com/Why-do-some-developers-at-strong-companies-like-Google-consider-Agile-development-to-be-nonsense/answer/David-Jeske">saw the limits of Agile</a>, I became more <a href="https://unarchitectedsystems.blogspot.com/2016/11/letter-to-my-future-self.html">interested</a> in the product operations, SRE side of systems and how observability, explainability, human collaboration and supportability spins system evolution to converge towards simplicity (or not) and builds a community-centric narrative that hopefully <a href="https://www.wired.com/story/why-we-love-tech-defense-difficult-industry/">enables</a>, in the long term, better <a href="https://www.jofreeman.com/joreen/tyranny.htm">socio-technological structures</a>.<br /><br />Over the last year or so, though, I found myself surprised by an influx of interesting material, in particular in regards to <a href="https://developers.google.com/machine-learning/crash-course/fairness/video-lecture">bias</a>, <a href="https://twitter.com/kentbeck/status/566255102067871744">culture</a>, <a href="https://medium.com/@duretti/no-flex-zone-empathy-driven-development-aebf4d8cf7cf">empathy</a>, <a href="https://www.ryn.works/blog/2019/03/29/on-failure">failure</a>, <a href="https://bravenewgeek.com/microservice-observability-part-1-disambiguating-observability-and-monitoring/">discovery</a>, <a href="https://www.youtube.com/watch?v=obB2IvCv-K0">resilience</a> (<a href="https://landing.google.com/sre/sre-book/chapters/accelerating-sre-on-call/">SRE</a>) and risk awareness in <a href="https://www.thoughtworks.com/insights/blog/macro-trends-tech-industry-april-2019?">orchestration</a> <a href="https://jobs.netflix.com/jobs/869465">complexity of socio-technical systems</a>.<br /><br />A welcome change from a long overdue self correction of <a href="https://www.reuters.com/article/us-softbank-group-visionfund/softbanks-plans-for-second-mega-fund-hit-by-wework-debacle-idUSKBN1WJ0AA">VC-fueled</a> <a href="https://www.nytimes.com/2019/08/02/technology/brex-start-up.html">get-rich-quick</a> <a href="https://mattstoller.substack.com/p/wework-and-counterfeit-capitalism">startups culture</a>. Concrete examples are my favourite book <a href="https://itrevolution.com/book/accelerate/">Accelerate</a>, which inspired a great <a href="https://www.youtube.com/watch?v=MZnrxjw602E&t=172s">Böckeler / Fowler talk</a> at Craft Conf and a <a href="https://www.youtube.com/watch?v=AkYDsiRVqno">very entertaining self reflection by Tilkov</a>, and other nice perspectives like George Fairbanks <a href="https://www.georgefairbanks.com/saturn-2019-continuous-design-of-it-systems">Continuous Design Talk</a> and <a href="https://www.michaelnygard.com/blog/2019/03/shared-mutable-team-state/">Nygard thinking</a> about <a href="https://unarchitectedsystems.blogspot.com/2014/10/denying-state.html">state</a>, <a href="http://alvaro-videla.com/2018/09/programming-languages-are-not-languages.html">Videla's</a>, <a href="https://medium.com/@mwichary/the-ghost-tech-in-todays-language-c3becc9ae30b">Wichary's</a>, <a href="https://mitpress.mit.edu/books/architectural-intelligence">Steenson's</a> and <a href="https://lithub.com/ellen-ullman-we-have-to-demystify-code/">Ullman's</a> thoughts about <a href="https://twitter.com/hillelogram/status/1116882063242727424">weird languages</a>, <a href="https://pragprog.com/book/mkdsa/design-it">Design It!</a> (and the <a href="https://pragprog.com/book/mnee2/release-it-second-edition">2nd edition of Release It!</a>), but also some really great distributed systems research by <a href="https://www.youtube.com/watch?v=KTHOwgpMIiU">Howard</a> and <a href="https://www.inkandswitch.com/local-first.html">Kleppmann</a> challenging our <a href="https://twitter.com/kelseyhightower/status/1032266094252158976">perspective on concurrency</a>, and new, <a href="https://twitter.com/fchollet/status/1178738075188350985?s=20">humble ways to create frameworks</a>.<div>
<br /><h3>
Empathy for Entropy</h3>
It's not simply that Microservices have made microblogging-driven startups suddenly realize the value of <a href="https://en.wikipedia.org/wiki/Big_Design_Up_Front">BDUF</a>. It rather is the agile move away from Enforcement towards Observation with better <a href="https://medium.com/@copyconstruct/distributed-tracing-weve-been-doing-it-wrong-39fc92a857df">tools, empathetic user-centric techniques</a> and ways of thinking about consistency and concurrency. It reminds me of my <a href="https://jaxenter.de/der-anwendungsserver-eine-zukunft-in-filmszenen-4253">W-Jax talk 7 years ago</a>, when I first read about <a href="https://cloud.google.com/spanner/">Spanner</a> and its formalization of time. When <a href="https://queue.acm.org/detail.cfm?id=3321612">Kleppmann argues for OLEP</a> and <a href="https://www.inkandswitch.com/local-first.html">Local First</a>, that the queue is the database, it reminds me how Spanner argues that <a href="https://cloud.google.com/blog/products/databases/build-your-own-event-sourced-system-using-cloud-spanner">the database is the queue</a>. Both arrive at the same insight: That information and time are entangled, and that entropy or consistency are derived from that. As <a href="https://blog.acolyer.org/2019/03/08/a-generalised-solution-to-distributed-consensus/">Howard</a> beautifully analyzed, for our systems it’s easy to replace the concept of time with state transitions (Lamport clocks). This allows us to step away from seeing the system as uncontrollable, and us reactive, to focus on the real, <a href="https://pragprog.com/the-pragmatic-programmer/extracts/software-entropy">human cause for entropy</a>.<div>
<br /><div>
<a name='more'></a><h3>
Flat Ontology</h3>
This has interesting implications for consistency in the domain-driven design sense, because it means that changes in the model (from a business- or user-perspective, designed and desired or not) have just the same, equal impact on the system architecture as-is as production releases, technology choices, bugs, company culture, team composition and societal trends. All of those need to be observed and reacted to, ideally anticipated, in order to keep external consistency to the user, just like gardening over the seasons or wine blending over the years. <br /><br />There is a group of thought experiments called <a href="https://en.wikipedia.org/wiki/List_of_Ship_of_Theseus_examples">Theseus’ Ship</a> which this art sometimes refers to. The basic idea is that a concept can stay the same even though everything material of if changes - even if every wood plank is replaced, it’s still Theseus’ ship.I was reminded of this when reading the example of the "<i>Dutch East India Trading Company</i>" (<a href="https://www.theguardian.com/books/2019/sep/23/the-anarchy-the-relentless-rise-of-the-east-india-company-william-dalrymple-review">follow-up reading on Anarchy</a>) staying the same despite everything around and inside it changing, in a book on <a href="https://en.wikipedia.org/wiki/Object-oriented_ontology">Object-Oriented Ontology</a> (OOO), a hip philosophical framework especially in art circles. And just today, reading the <a href="https://pragprog.com/book/tpp20/the-pragmatic-programmer-20th-anniversary-edition">20 year anniversary edition of The Pragmatic Programmer</a>, they look at their own techniques through the lens of Theseus’ Ship.<br /><br />While I don't agree with the OOO ideology on an everyday basis, particularly because it is highly subjective, <a href="https://terenceblake.wordpress.com/2016/06/17/zizek-and-object-oriented-ontology-a-non-scientistic-critique/">while claiming to not be</a>, and often unnecessarily <a href="https://youtu.be/6GHiV4tuRt8?t=3433">binary</a>, tribal and even hostile* ... so, while I don't agree with that, I do think it is a useful tool to think about Architecture in the System Design sense, because it equalizes objects, as an isolated concept, with events or changes, without a human perspective of time. It reminds me of why I love <a href="http://erlang.org/doc/design_principles/des_princ.html#behaviours">Erlang</a>, because it is explicit about the observer outside and the actor inside the system, it embraces <a href="https://bravenewgeek.com/microservice-observability-part-1-disambiguating-observability-and-monitoring/">speculation about potential realities and risks</a>. It opens a conversation everyone, not only programmers, can participate. In that sense, the 20th anniversary of PragProg is in itself an iteration, it is an actual learning on Architecture, not just a new experiment. I tried to explain that feeling clumsily in my book, 6 years ago, using <a href="https://en.wikipedia.org/wiki/Actor%E2%80%93network_theory">ANT</a> and <a href="https://en.wikipedia.org/wiki/Conway%27s_law">Conway's Law</a>, but OOO’s idea of a "flat ontology" is just a lot more elegant, and fits into more recent publications like adaptive <a href="https://teamtopologies.com/book">Team Topologies</a>.</div>
<div>
<br /><h3>
Architecture Astronauts </h3>
OOO reminds me of a fierce discussion I had 5 years ago or so with a Domain Architect for an <a href="https://martinfowler.com/eaaDev/EventSourcing.html">event-sourced system</a>. They had designed an event bus with a <a href="https://www.innoq.com/en/blog/thoughts-on-a-canonical-data-model/">canonical data model</a> verified in a metadata master system. Their USP was a mandatory, human multi-stage process to get any metadata change approved, new events required at least one presentation before the architecture board - a monthly committee in a different time zone. Needless to say my approach was the complete opposite, back then <a href="https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/">Spotify-oriented</a>, defining a set of accepted formats (eg JSON) and a few standard header fields, observing what goes on the bus and reaching out to teams with, for instance, long dead letter queues. One approach was ideological, prone to bias, focused on preserving hierarchy. The other observing, open, user-focused. In other words one was tall, steep (an <a href="https://architectelevator.com/">elevator</a>), the other flat. The ivory tower was what gave architects a bad name, the extreme end of the ivory tower was called <a href="https://www.joelonsoftware.com/2001/04/21/dont-let-architecture-astronauts-scare-you/">Astronauts</a>, who are more excited about defining abstract patterns than making lives of the end user easier.<br /><br />But actual Astronauts see changes in the earth earlier and clearer. They seem above, but actually their world is flat, a whole. They <i>fundamentally</i> care about impact. They leave to conduct experiments, and come back to the ground with new insights. They risk everything using the engineering they’ve helped to build - an engineering that inspires others. Similar to Theseus’ Ship, the stations are constantly changing, constantly accommodating new experiments. Astronauts and missions change, there is no ultimate goal but <a href="http://philosophizethis.org/consequences-of-reason/">constantly rebuilding our understanding</a> of the system from a slightly different vantage point, yet still within. 20 years later, I’m on the side of Astronauts, and I think we’d be better engineers if all of us sometimes think about the big impact, the world, the planet.<br /><br />Seeing things as a whole is not hierarchical, it’s empathy. Looking at emergent patterns of microservice or event communication, network latency, error types, user behaviour, machine learning models or support cases and feedback has nothing to do with building a house or a bridge, but looking at the landscape, the environment, and society. Which is a way of looking at causality similar to what academic history often does, as <a href="https://youtu.be/bRcu-ysocX4?t=400">hierarchies of culturally observable, overlapping effects</a>.<br /><br />In that sense, as Buckminster-Fuller said, "<a href="https://howwegettonext.com/we-are-all-astronauts-f3dec00f9911"><i>We are all astronauts</i></a>". Pragmatic ones. <br /><br /><br /><br /><span style="font-size: x-small;">*) As can be seen in the numerous attacks against others in the book while praising Sci-Arch even though they only use OOO to claim a new star architecture (“Killing simplicity”) that ignores community context </span></div>
</div>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-18174794829992751852018-11-29T15:30:00.001+01:002018-11-29T15:30:22.561+01:00An annotated Philosophy of Software Design<blockquote class="tr_bq" style="text-align: right;">
<i>We build our computer (systems) the way we build our cities:<br />over time, without a plan, on top of ruins<br /><a href="https://en.wikiquote.org/wiki/Ellen_Ullman">Ellen Ullman</a></i></blockquote>
<div style="text-align: right;">
<br /></div>
<div style="text-align: start;">
After much hype I've read John Ousterhout's <i><a href="https://www.amazon.de/Philosophy-Software-Design-John-Ousterhout/dp/1732102201">A Philosophy of Software Design</a></i> which he uses to teach his <a href="https://web.stanford.edu/~ouster/">course at Stanford</a> and presents as a personal experience "opinion piece". Its basic goal seems to be to develop an <a href="https://youtu.be/bmSAYlu0NcY?t=160">awareness</a> and intuition about how and when to manage complexity emerging out of proper problem decomposition in Software Design.</div>
<div style="text-align: start;">
<br /></div>
<div style="text-align: start;">
Undergraduate students thus seem to be the main audience, which explains why, despite the title, architecture or <i><a href="https://www.oreilly.com/library/view/the-site-reliability/9781492029496/ch12.html">non-abstract large system design</a></i>, distributed systems, developer workflow and design thinking are not covered much, and why a prosaic, almost aphorismic writing style were chosen. My assumption is this is also why barely any references or annotations are given. It's supposed to be the beginning of an iterative journey, similar to language learning. I figured it might help to share mine, though, to go the next step in that journey (Github would feel too official, so blog):<br />
<br />
<b>Preface</b>, Iterative process to this book. <a href="https://www.amazon.com/Exercises-Programming-Style-Cristina-Videira/dp/1482227371/">Exercises in Programming Style 2014</a>, <a href="http://shop.oreilly.com/product/9780596510046.do">Beautiful Code, 2008</a>, <a href="https://books.google.com.sg/books/about/Software_Craftsmanship.html?id=C9vvHV1lIawC&source=kp_cover&redir_esc=y">Software Craftsmanship 2001</a> (<a href="https://medium.com/@alandevlin7/software-craftsmanship-is-sexist-and-must-be-scrapped-1d62d79e6ad6">Craftsperson</a>) and before that <a href="https://en.wikipedia.org/wiki/The_Pragmatic_Programmer">The Pragmatic Programmer 1999</a> and even <a href="https://www.goodreads.com/book/show/52084.Programming_Pearls">Programming Pearls 1986</a> developed iterative, small-wisdom, later developed into <a href="http://codekata.pragprog.com/">Kata</a>'s, based learning of software design.<br />
<br />
<br />
<a name='more'></a><br />
<br /></div>
<div style="text-align: start;">
<b>Page 2</b>, Introduction. Modular Design: Also called Separation of concerns, coined by <a href="https://en.wikipedia.org/wiki/Separation_of_concerns#Origin">Edsger W. Dijkstra 1974</a>, or Information Hiding (see page 19)</div>
<div style="text-align: start;">
<br /></div>
<div style="text-align: start;">
<b>Page 2</b>, Introduction. Agile Development: <a href="http://agilemanifesto.org/">Agile Manifesto 2001</a>, here sounds more like <a href="https://martinfowler.com/articles/evo-arch-forward.html">Evolutionary Design</a> as increments were used earlier e.g. in the <a href="https://en.wikipedia.org/wiki/Rational_Unified_Process#History">Rational Unified Process</a> and <a href="https://en.wikipedia.org/wiki/Kanban#Origins">Kanban</a> as early as 1940ies</div>
<div style="text-align: start;">
<br /></div>
<div style="text-align: start;">
<b>Page 3</b>, Chapter 1, Introduction. Incremental works well: Some good proofs in <a href="https://books.google.com.sg/books/about/Accelerate.html?id=85XHAQAACAAJ&source=kp_cover&redir_esc=y">Accelerate, Forsgren et al. 2018</a> and yearly <a href="https://www.standishgroup.com/outline">Chaos Report</a>.</div>
<div style="text-align: start;">
<br /></div>
<div style="text-align: start;">
<b>Page 3</b>, Chapter 1, Introduction. Continuous Redesign: <a href="https://en.wikipedia.org/wiki/Design_thinking#History">Design Thinking</a> as popularized e.g. by IDEO 1991 or <a href="https://www.nngroup.com/articles/iterative-design/">Nielsen 1993</a>, but already before that in Computer Art (<a href="http://www.girlwonder.com/papers-articles">Steenson 2018</a>) or <a href="https://de.wikipedia.org/wiki/Kaizen">Kaizen</a> in manufacturing (later called Lean, see Kanban above) and iterative design during World War II and the Space Age</div>
<div style="text-align: start;">
<br /></div>
<div style="text-align: start;">
<b>Page 4</b>, Chapter 1, Introduction. Red Flags: Also called Code Smells, coined by <a href="https://en.wikipedia.org/wiki/Code_smell">Kent Beck in the 1990ies</a>, or Anti-Patterns, by <a href="https://en.wikipedia.org/wiki/Anti-pattern">Andrew Koenig 1995</a>.<br />
<br />
<b>Page 4</b>, Chapter 1, Introduction, Code Reviews. <a href="https://blog.codinghorror.com/code-reviews-just-do-it/">Atwood 2006</a> and many others have written about the need for reviews, also DevOps e.g. <a href="https://books.google.com.sg/books/about/Continuous_Delivery.html?id=6ADDuzere-YC&redir_esc=y">Humble 2010</a>.</div>
<div style="text-align: start;">
<br />
<b>Page 5,</b> omitted because there it lots of other definitions of Complexity and Complex Systems in Computing (<a href="https://en.wikipedia.org/wiki/Cyclomatic_complexity">Cyclomatic Complexity</a> 1976, <a href="http://www.adaptivesd.com/pubs.html">Highsmith 2000</a>, <a href="https://en.wikipedia.org/wiki/A_New_Kind_of_Science">Wolfram 2002</a>) all the way up to Emergence which, I agree, would be too much for the scope of this book.<br />
<br /></div>
<div style="text-align: start;">
<b>Page 6</b>, Chapter 2. Complexity: First, Design of Design came to mind but actually Fred Brooks mentioned it first in "<a href="https://en.wikipedia.org/wiki/No_Silver_Bullet">No Silver Bullet</a>" from 1986, coining the term "accidental complexity" based on <a href="https://ieeexplore.ieee.org/document/1702388/">McCabe 1976</a>. Beyond, more generally, Herbert A. Simon, "<a href="http://www2.econ.iastate.edu/tesfatsi/ArchitectureOfComplexity.HSimon1962.pdf">The Architecture of Complexity</a>" 1962 and Weinberg's "<a href="https://books.google.com.sg/books/about/An_Introduction_to_General_Systems_Think.html?id=eU9gDxt9X0wC&redir_esc=y">General Systems Thinking</a>", which includes this excellent chart (in a version by <a href="https://www.info.ucl.ac.be/~pvr/VanRoyChapter.pdf">Peter Van Roy</a> from 2012) explaining which complexity domain programming fits in:</div>
<div style="text-align: start;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3ClidBXhO9W7t12eHB9ERL5rUHmx_6PT1Q9_X_M6-WJ4vLR4UAOuIZNPG913_-Fn8tMDzxTn36sCBnbZ49pVXf41uGy36kFYmwerZIDTJZ3pIYADn3WWSR_s8fuAwLr3z9qi_FCz9OGjU/s1600/peter+van+roy+2012+weinberg+1975.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="362" data-original-width="440" height="263" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3ClidBXhO9W7t12eHB9ERL5rUHmx_6PT1Q9_X_M6-WJ4vLR4UAOuIZNPG913_-Fn8tMDzxTn36sCBnbZ49pVXf41uGy36kFYmwerZIDTJZ3pIYADn3WWSR_s8fuAwLr3z9qi_FCz9OGjU/s320/peter+van+roy+2012+weinberg+1975.png" width="320" /></a></div>
<div style="text-align: start;">
<b>Page 6</b> at the end nicely speaks about to write for the reader. To be fair, <a href="http://johannesbrodwall.com/2018/06/24/forget-about-clean-code-lets-embrace-compassionate-code/">Compassionate Code</a> was only published this year and says it better than Uncle Bob, but <a href="https://www.goodreads.com/book/show/3735293-clean-code">Clean Code, 2008</a> did speak earlier about <i>authoring</i> the code itself with empathy for the reader in mind (see also page 149). Writing for a reader goes back as much as <a href="https://en.wikipedia.org/wiki/The_Elements_of_Programming_Style">Kerningham's 1974 "Elements of Programming Style"</a>. Good book on reading code <a href="https://www.spinellis.gr/codereading/">Spinellis 2003</a>.<br />
<br />
<b>Page 7</b>, Chapter 2, Cognitive Load: Herbert A. Simon developed the concept of <i><a href="https://en.wikipedia.org/wiki/Bounded_rationality">bounded rationality</a>,</i> an early way to formulate limited rationality or bias outside of philosophy and a core idea on the way to <i>bounded contexts</i> in Domain-Driven Design (see page 22)<br />
<br />
<b>Page 7</b> further below, <a href="https://en.wikipedia.org/wiki/The_Mythical_Man-Month">Brooks' Mythical Man Month, 1975</a> is the canonical reference why lines of code are not a useful metric. Also see <a href="https://en.m.wikipedia.org/wiki/Hofstadter%27s_law">Hofstadter's law</a><br />
<br />
<b>Page 8, </b>Chapter 2, Unknown Unknowns: Refers to the <a href="https://en.wikipedia.org/wiki/Johari_window">Johari Window</a> model as popularized by <a href="https://en.wikipedia.org/wiki/There_are_known_knowns">Donald Rumsfeld in 2002</a>.<br />
<br />
<b>Page 9</b>, Chapter 2, Obvious behaviour: The <a href="https://books.google.com.sg/books/about/The_Field_Guide_to_Understanding_Human_E.html?id=NTP5iLn4XHkC&redir_esc=y">Field Guide to Understanding Human Error, 2006</a>, <a href="https://how.complexsystems.fail/">How Complex Systems Fail</a> 1998, and <a href="https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow">Thinking fast, thinking slow, 2012</a> talk about biases and how to overcome them with techniques such as <a href="https://en.wikipedia.org/wiki/Choice_architecture">Choice Architecture in Nudge, 2008</a>.<br />
<br />
<b>Page 9</b>, Chapter 2, Causes: Network Protocol Dependencies are part of the old <a href="https://programmingisterrible.com/post/42215715657/postels-principle-is-a-bad-idea">TCP/IP adage</a> "<i>Be conservative in what you do, be liberal in what you accept from others</i>" coined <a href="https://en.wikipedia.org/wiki/Robustness_principle">Postel's Law 1980</a>.<br />
<br />
<b>Page 11</b>, Chapter 2, Incremental Complexity: Some references to Tech Debt, Software Rot in Martin <a href="https://martinfowler.com/books/refactoring.html">Fowler and Beck's Refactoring 1999</a> could be given and how to tackle it e.g. the Scout Rule (<a href="https://www.goodreads.com/book/show/3735293-clean-code">Clean Code, 2008</a> or <a href="https://www.amazon.com/Things-Every-Programmer-Should-Know/dp/0596809484">97 Things, 2010</a>), Fowler's <a href="https://martinfowler.com/bliki/OpportunisticRefactoring.html">Opportunistic Refactoring</a>, 2011, or <a href="https://martinfowler.com/bliki/DesignStaminaHypothesis.html">Design Stamina, 2007</a>. Same on page 18.<br />
<br />
<b>Page 15</b>, Chapter 3, Investment for Design: All references to Investment - See above on Design Stamina and (Tech) <a href="https://steveblank.com/2015/05/19/organizational-debt-is-like-technical-debt-but-worse/">Debt</a> but more generally Architecture, in particular <a href="http://files.catwell.info/misc/mirror/2003-martin-fowler-who-needs-an-architect.pdf">Fowler 2003 "Who needs an architect"</a>, the <a href="http://wiki.c2.com/?BigDesignUpFront">BDUF</a> discussion (<a href="https://www.joelonsoftware.com/2005/08/17/the-project-aardvark-spec/">Spolsky</a> 2005 etc) how it relates to iterative development and "Just Enough Design" (<a href="https://www.georgefairbanks.com/book/">Fairbanks 2010</a>). Numbers on the success in <a href="https://books.google.com.sg/books/about/Accelerate.html?id=85XHAQAACAAJ&source=kp_cover&redir_esc=y">Accelerate, Forsgren et al. 2018</a>.<br />
<br />
<b>Page 16</b>, Chapter 3, Spaghetti Code: <a href="http://www.laputan.org/mud/">Foote and Yoder, 1999, the seminal "Big Ball of Mud"</a> begins with a picture of pasta.<br />
<br />
<b>Page 17</b>, Chapter 3, Fun at Work: Chapter on <a href="https://landing.google.com/sre/sre-book/chapters/eliminating-toil/">Toil in the SRE Book, 2017</a>, earlier in Chad Fowler's <a href="https://www.goodreads.com/book/show/6399113-the-passionate-programmer">Passionate Programmer 2010</a>, empathy is a topic of Clean Code as well as for Fred Brooks.<br />
<br />
<b>Page 19</b>, Chapter 4, <a href="https://en.wikipedia.org/wiki/SOLID">SOLID</a> paraphrased; Modules should be deep (one of the major design principles), Information Hiding, <a href="https://en.wikipedia.org/wiki/Information_hiding#History">David Parnas in 1972</a> (actually <a href="https://youtu.be/bmSAYlu0NcY?t=753">mentioned</a> in the video and in the Preface of the book)</div>
<div style="text-align: start;">
<br /></div>
<div style="text-align: start;">
<b>Page 21</b>, Chapter 4, Technical Abstractions: <a href="https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/">Leaky Abstractions, Spolsky 2002</a>. Could even go back to the <a href="https://en.wikipedia.org/wiki/SOLID">SOLID</a> principle of "Low Coupling, High Cohesion", coming from the 70ies or even 60ies structured programming, e.g. in Dijstra's references to <a href="https://en.wikipedia.org/wiki/The_Elements_of_Programming_Style">Programming Style</a>. Referring here to OOP, Single Responsibility Principle, <a href="https://www.amazon.com/Object-Design-Roles-Responsibilities-Collaborations/dp/0201379430">Wirfs-Brock 2002</a> and <a href="https://en.wikipedia.org/wiki/GRASP_(object-oriented_design)">GRASP</a>, Larman 2005.</div>
<div style="text-align: start;">
<br /></div>
<div style="text-align: start;">
<b>Page 22</b>, Chapter 4, Real-Life Abstractions: Most importantly in <a href="https://de.wikipedia.org/wiki/Domain-driven_Design">Domain-Driven Design, Evans 2003</a>, but already in <a href="https://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670">Code Complete</a>, 1993, speaking about "Consistent Abstractions" and "Real-World Objects". System <a href="https://www.press.uchicago.edu/ucp/books/book/chicago/M/bo3637992.html">Metaphors</a>, for instance when Steve McConnell talks about Metaphors in <a href="https://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670">Code Complete</a>, 1993, comparing the <i>farming</i> (or <a href="https://unarchitectedsystems.blogspot.com/2015/07/documenting-service-interactions.html">gardening</a>) and the <i>construction</i> (or architecture) approach. Obligatory <a href="https://en.wikipedia.org/wiki/On_Exactitude_in_Science">Borges reference</a>.<br />
<br />
<b>Page 23</b>, Chapter 4, Deep Modules: Could use a notion whether state or functionality (logic) is encapsulated. See <a href="http://wiki.c2.com/?AlanKaysDefinitionOfObjectOriented">Alan Kay 1972</a> on Object-Oriented Programming (OOP) with Messages or <a href="https://www.amazon.com/Things-Every-Programmer-Should-Know/dp/0596809484">Henney 2010 </a>"Encapsulate Behavior, Not Just State"<br />
<br />
<b>Page 24</b>, Chapter 4, Classes should be small: Word "Micro" in Microservices, canonical reference <a href="https://martinfowler.com/articles/microservices.html">Lewis 2014</a> although earlier e.g. <a href="http://highscalability.com/blog/2012/5/9/cell-architectures.html">Hoff 2012</a><br />
<br />
<b>Page 30</b>, Chapter 5, Information Leakage: <a href="https://en.wikipedia.org/wiki/Law_of_Demeter">Law of Demeter</a>, 1987. Also see Leaky Abstraction, State and OOP above as in no Shared State and <a href="https://en.wikipedia.org/wiki/Shared-nothing_architecture">Shared-nothing architecture</a> since the 70ies. Nice Temporal Decomposition anti-pattern is an interesting red flag leading to <a href="https://microservices.io/patterns/data/event-sourcing.html">Event Sourcing / CQRS</a>.<br />
<br />
<b>Page 35</b>, Chapter 5, Example: Those methods use primitive types in their names (usually frowned upon e.g. in <a href="https://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670">Code Complete</a>, 1993 and, as this is Java, <a href="https://books.google.com.sg/books?id=ka2VUBqHiWkC&vq=naming&hl=de&source=gbs_navlinks_s">Effective Java, Bloch 2008</a> refer to <a href="https://en.wikipedia.org/wiki/Duck_typing">Duck Typing</a>) and <a href="http://wiki.c2.com/?DontUseExceptionsForFlowControl">exceptions for control flow</a> (assertions), might benefit from an explanation why that is better. See page 106.<br />
<br />
<b>Page 36</b>, Chapter 5, Default: Principles behind <a href="https://en.wikipedia.org/wiki/Convention_over_configuration">Convention over Configuration </a>popularized by Ruby, 2007.<br />
<br />
<b>Page 38</b>, Chapter 5, Conclusion: <a href="https://de.wikipedia.org/wiki/Domain-driven_Design">Domain-Driven Design, Evans 2003</a>, especially for a ubiquitous language for naming.<br />
<br />
<b>Page 39</b>, Chapter 6, Increments: <a href="https://en.wikipedia.org/wiki/Minimum_viable_product">Minimum Viable Product</a>, 2001-2009, <a href="https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it">YAGNI</a> 2001, <a href="https://semver.org/">Semantic Versioning</a> 2010 and Chrome's' Evergreen concepts for incremental updates.<br />
<br />
<b>Page 40</b>, Chapter 6, General Purpose: <a href="https://en.wikipedia.org/wiki/Composition_over_inheritance">Composition over Inheritance</a>, <a href="https://martinfowler.com/books/refactoring.html">Refactoring 1999</a> again (see also page 152)<br />
<br />
<b>Page 46</b>, Chapter 7, Pass-Through Methods: Decomposition of pass through is one of the main concepts of the <a href="https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s06.html">Unix Philosophy</a>, McIlroy 1978, also <a href="http://www.catb.org/esr/writings/taoup/html/">Rule of Modularity, Raymond 2003</a>.<br />
<br />
<b>Page 49</b>. Chapter 7, Decorators: First time Design Patterns are mentioned, yet without explanation - only on page 156. Canonical is GoF (<a href="https://en.wikipedia.org/wiki/Design_Patterns">Gamma) 1994</a> which includes Decorator, <a href="https://en.wikipedia.org/wiki/A_Pattern_Language">Alexander 1977</a>. As this is Java, <a href="https://en.wikipedia.org/wiki/Aspect-oriented_programming">Aspect-oriented programming (AOP)</a>, 2001, <a href="https://en.wikipedia.org/wiki/Metaprogramming">Metaprogramming</a>, Erlang behaviours or function decorators in Python might be interesting.<br />
<br />
<b>Page 51-53</b>, Chapter 7, Pass-Through Variables: See Pass-Through Methods above. Global State in Context came from Automata / <a href="https://en.wikipedia.org/wiki/Unbounded_nondeterminism">Nondeterminism (Dijkstra 1976)</a>, badly done in J2EE <a href="https://www.amazon.com/dp/0131422464/?tag=stackoverflow17-20">introduced in 2003</a> typically now considered an anti-pattern (would be good to qualify "<i>unlikely that would need multiple instances in production</i>" and talk about <a href="https://haisum.github.io/2015/10/08/problems-and-classification-of-distributed-systems/">distributed systems</a>), better in Erlang as discussed in "<a href="http://highscalability.com/blog/2014/10/22/paper-actor-model-of-computation-scalable-robust-information.html">Actor Model of Computation</a>" 1974. Otherwise discuss State in UI Patterns e.g. MVC and contract with <a href="https://medium.com/swlh/the-case-for-flux-379b7d1982c6">Unidirectional Flow in Flux</a>, 2014 which is basically Data Flow / Reactive similar to Actors. Mentions "immutable" (<a href="https://12factor.net/processes">12-Factor Apps</a>, <a href="https://en.wikipedia.org/wiki/Immutable_object">Functional Programming</a>) only at the very end.<br />
<br />
<b>Page 57</b>, Chapter 7, Configuration: See Convention over Configuration above but also consider Feature Toggles or Flags as a means to achieve <a href="https://en.wikipedia.org/wiki/Observability">Observability</a> of application state run <a href="https://en.wikipedia.org/wiki/A/B_testing">A/B Tests</a> or Experiments, control Circuit Breakers etc. - see <a href="https://books.google.com.sg/books?id=Ug9QDwAAQBAJ&dq=release+it+nygard&hl=de&source=gbs_navlinks_s">Nygard 2007</a> (on page 55 "System Administrator" is mentioned which suggests Immutable Infrastructure / Continuous Delivery are not considered).<br />
<br />
<b>Page 62</b>, Chapter 9, Goto: Canonical reference <a href="https://en.wikipedia.org/wiki/Considered_harmful">Dijkstra 1968</a>. Repetition / Duplication see DRY at least since <a href="https://en.wikipedia.org/wiki/The_Pragmatic_Programmer">Hunt and Thomas 1999</a>, also Rule of three in <a href="https://martinfowler.com/books/refactoring.html">Refactoring 1999</a>. Not always bad e.g. a <a href="https://www.microservices.com/talks/dont-build-a-distributed-monolith/">Distributed Monolith</a> can be worse.<br />
<br />
<b>Page 64</b>, Chapter 9, Layering: Think in Behaviour or <a href="http://wiki.c2.com/?PortsAndAdaptersArchitecture">Ports & Adapters, 2005,</a> rather than Tiers. Obligatory <a href="https://en.wikipedia.org/wiki/Indirection">Too many layers of Indirection reference</a>.<br />
<br />
<b>Page 70</b>, Chapter 9, Splitting and Joining: See page 24, Microservices, canonical reference <a href="https://martinfowler.com/articles/microservices.html">Lewis 2014</a>.<br />
<br />
<b>Page 71</b>, Chapter 9, Do one thing: See page 30 <a href="https://en.wikipedia.org/wiki/Law_of_Demeter">Law of Demeter</a>, 1987, page 46 <a href="https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s06.html">Unix Philosophy</a>, McIlroy 1978.<br />
<br />
<b>Page 76</b>, Chapter 10.1, Transaction State and Rollback in Abort: See page 51ff on unbounded nondeterminism, should introduce concept of state. <a href="https://en.wikipedia.org/wiki/Transaction_processing">Transaction Monitoring</a> and ACID principles at least since <a href="https://dl.acm.org/citation.cfm?id=291">Haerder and Reuter 1982</a>. Should mention development in <a href="https://en.wikipedia.org/wiki/Software_transactional_memory">Software Transactional Memory</a> popularized by Akka 2009 and Persistent State in <a href="https://www.oreilly.com/library/view/streaming-systems/9781491983867/">Streaming Systems, Lax 2018</a>.<br />
<br />
<b>Page 78</b>, Chapter 10.2, Exceptions: See page 35, Exceptions as control flow. Worthwhile mentioning other ways, e.g. <a href="https://blog.golang.org/error-handling-and-go">Golang Error Types</a> and Scala <a href="https://www.scala-lang.org/api/current/scala/util/Try.html">Try's</a> (<a href="https://dl.acm.org/citation.cfm?id=186031">Läufer 1994</a>) - <a href="https://en.wikipedia.org/wiki/Type_system#Static_type_checking">Type safety</a> and <a href="https://en.wikipedia.org/wiki/Finite-state_machine">statefulness</a> are part of the interface.<br />
<br />
<b>Page 82</b> (also page 80), Chapter 10.7, Exception aggregation: Error Hiding and Exception Chaining / Wrapping was introduced e.g. in Java 1.4, <a href="https://www.ibm.com/developerworks/library/j-jtp05254/index.html">Bloch 2001</a>, Scala and Golang support Exception pattern matching.<br />
<br />
<b>Page 85</b>, Chapter 10.7 below: Ways of handling exceptions vs error for instance in <a href="https://en.wikipedia.org/wiki/Object-Oriented_Software_Construction">Meyer 1998</a>.<br />
<br />
<b>Page 86</b>, Chapter 10.7 below: Recreating servers is part of Immutable Infrastructure (Containers, e.g. <a href="https://www.kernel.org/doc/ols/2007/ols2007v2-pages-45-58.pdf">Menage 2007</a>, <a href="http://chadfowler.com/2013/06/23/immutable-deployments.html">Fowler 2013</a>) popularized by <a href="https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116">Netflix' Chaos Engineering 2011</a>.<br />
<br />
<b>Page 86</b>, Chapter 10.8 Crashing: See page 78 for Golang approach, <a href="http://erlang.org/doc/reference_manual/errors.html">Erlang</a> 1992 is famous for traditionally rather <i>monitoring</i> processes calling the "just crash" philosophy defensive programming .<br />
<br />
<b>Page 91</b>, Chapter 11, Design twice: See page 57 on <a href="https://en.wikipedia.org/wiki/A/B_testing">A/B Tests</a>. Mention <a href="https://en.wikipedia.org/wiki/V-Model_(software_development)">V-Model</a> as historic idea to facilitate 4-eye principle. <a href="https://books.google.com.sg/books?id=Ug9QDwAAQBAJ&dq=release+it+nygard&hl=de&source=gbs_navlinks_s">Nygard 2007</a> also covers Architectural Tradeoffs documented for instance in <a href="https://adr.github.io/">Architecture Decision Records</a>. (nice comment on page 93 on Self-serving bias / <a href="https://en.wikipedia.org/wiki/Overconfidence_effect">Overconfidence Effect</a> / <a href="https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect">Dunning-Kruger</a>)<br />
<br />
<b>Page 95-96</b>, Chapter 12, Comments: Discussion popularized in <a href="https://www.goodreads.com/book/show/3735293-clean-code">Clean Code, 2008</a> but earlier e.g. <a href="https://en.wikipedia.org/wiki/The_Elements_of_Programming_Style">Kerningham's 1974</a> and <a href="http://www.developerdotstar.com/mag/articles/reeves_design.html">Reeves 1992</a> mentioned code as design / documentation instead of comments. Visual Programming, UML and <a href="https://en.wikipedia.org/wiki/Model-driven_engineering">MDA 1980ies</a> as counter-examples, <a href="http://lighttable.com/2012/04/12/light-table-a-new-ide-concept/">Lighttable 2012</a> as future direction.<br />
<br />
<b>Page 97</b>, Chapter 12.1, Comments: May help to mention TDD, <a href="https://en.wikipedia.org/wiki/Behavior-driven_development">Behaviour-Driven-Development</a> (BDD) DSL e.g. North 2009 for tests as <i>specification</i> and <a href="https://en.wikipedia.org/wiki/Trait_(computer_programming)">Traits</a> / Aspects as a means to comment behaviour in code instead of "close to code" (page 137, may be omitted due to page 155).<br />
<br />
<b>Page 101</b>, Chapter 13: see page 95-95, <a href="https://en.wikipedia.org/wiki/The_Elements_of_Programming_Style">Kerningham's 1974</a> and <a href="http://www.developerdotstar.com/mag/articles/reeves_design.html">Reeves 1992</a>.<br />
<br />
<b>Page 101</b>, Chapter 13, Interface comments: Generating interface / API balance between type safety, code readability and uniformity / consistency in various <a href="https://en.wikipedia.org/wiki/Interface_description_language">IDL's</a> e.g. <a href="https://en.wikipedia.org/wiki/Include_directive">COBOL Copybooks 1968, C Header Files 1972</a>, <a href="https://en.wikipedia.org/wiki/Common_Object_Request_Broker_Architecture">CORBA 1991</a> , <a href="https://en.wikipedia.org/wiki/Resource_Description_Framework">RDF 1997</a>, <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm#fig_5_6">REST 2000</a>, <a href="https://en.wikipedia.org/wiki/OpenAPI_Specification">Swagger 2010</a><br />
<br />
<b>Page 103</b>, Chapter 13.2; See page 11, repetition of comments and page 95-96.<br />
<br />
<b>Page 105</b>, Chapter 13.3, Abstraction via comment: Nice wording precision and intuition. See page 22 but also <a href="https://www.ibm.com/developerworks/library/j-jtp05254/index.html">Bloch 2001</a> on abstraction levels for error messages.<br />
<br />
<b>Page 106 (and 110)</b>, Chapter 13.3, below: Unclear why variable names can't have units or invariants. <a href="https://en.wikipedia.org/wiki/Guard_(computer_science)">Guards</a> at least since Beck 1997 and earlier. Defect detection in <a href="https://jcp.org/en/jsr/detail?id=305">Java since 2006</a>. Example on heartbeat could be an enum to describe behaviour / attribute as advised in <a href="https://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670">Code Complete</a>, 1993. See page 35.<br />
<br />
<b>Page 116</b>, Chapter 13.6, what and why: Instead of loop iterations mention documentation with recursion, <a href="https://en.wikipedia.org/wiki/Higher-order_function">higher-order functions</a> and streams. Page 117 Comment to bug tracker may better be in blame layer (<a href="https://blog.codinghorror.com/who-wrote-this-crap/">Atwood 2007</a>).<br />
<br />
<b>Page 118</b>, Chapter 13.7, Design Notes: See page 91 on why-decision. Markdown or <a href="https://structurizr.com/help/documentation">Architecture as Code (e.g. Brown 2014)</a> and <a href="https://en.wikipedia.org/wiki/Infrastructure_as_code">Infrastructure as Code</a> even with <a href="https://itnext.io/vistio-visualize-your-istio-mesh-using-netflixs-vizceral-b075c402e18e">Service Meshes</a> better than text.<br />
<br />
<b>Page 122-123</b>, Chapter 14.2, vague naming. See <a href="https://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670">Code Complete</a>, 1993 how difficulty to name should raise doubts and metaphors (see page 22). Obligatory reference to <a href="https://twitter.com/codinghorror/status/506010907021828096?lang=de">Naming</a> things is hard.<br />
<br />
<b>Page 125</b>, Chapter 14.3, weak naming. Choosing names can also be <a href="https://en.wikipedia.org/wiki/Law_of_triviality">bikeshedding (Parkinson 1957)</a> and accelerate <a href="https://en.wikipedia.org/wiki/Conway%27s_law">Conway's Law 1967</a> by having too tight bounded contexts resembling existing bias and assuming an ideal state rather than evolutionary processes (e.g. <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm#sec_5_2">REST temporal quality, Fielding 2000</a>).<br />
<br />
<b>Page 126</b>, Chapter 14.4, consistency. Nice remark on consistency, already main principle in <a href="https://en.wikipedia.org/wiki/The_Elements_of_Programming_Style">Kerningham 1974</a>.<br />
<br />
<b>Page 129</b>, Chapter 15, comments as design. See <a href="https://en.wikipedia.org/wiki/Literate_programming">Literal Programming, Knuth 1984</a> popularized recently with the Wolfram Language and <a href="https://en.wikipedia.org/wiki/Project_Jupyter#Jupyter_Notebook">Jupyter Notebooks</a>. Also <a href="https://en.wikipedia.org/wiki/User_story">User Stories</a>, Cockburn 1998.<br />
<br />
<b>Page 131</b>, Chapter 15, simplicity: See <a href="https://books.google.com.sg/books?id=gJrmszNHQV4C&dq=code+beauty+simplicity&hl=de&source=gbs_navlinks_s">Wilson and Oram 2007</a>, earlier <a href="https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s06.html">Unix Philosophy</a>, McIlroy 1978 and <a href="https://martinfowler.com/bliki/BeckDesignRules.html">Beck 1999</a>.<br />
<br />
<b>Page 135</b>, Chapter 16, existing code: See Working with Legacy Code, <a href="https://www.oreilly.com/library/view/working-effectively-with/0131177052/">Feathers 2004</a>, <a href="https://martinfowler.com/books/refactoring.html">Refactoring 1999</a> and <a href="https://www.martinfowler.com/bliki/EventInterception.html">Event Interception since the 70ies</a>.<br />
<br />
<b>Page 136</b>, Chapter 16, below: See <a href="https://www.thoughtworks.com/books/building-evolutionary-architectures">Evolutionary Architecture</a> 2016 and lots of earlier examples on continuous improvement (page 3) and scout rule (page 11)<br />
<br />
<b>Page 138-19</b>, Chapter 16.3, no comments in repo, findability: See page 95, Lighttable, page 116, use annotations instead of plain text, Dependency Injection (<a href="https://martinfowler.com/bliki/InversionOfControl.html">Inversion of Control since 1990ies</a>) instead of direct dependency, Blame Layer and Google Code Search e.g. <a href="https://dl.acm.org/citation.cfm?id=2786855">Sadowski 2015</a>, <a href="https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext">Potvin 2016</a>.<br />
<br />
<b>Page 143</b>, Chapter 17.1, consistency: see page 126.<br />
<br />
<b>Page 148</b>, Chapter 18.2, events: First time events are mentioned, move to front, discuss FP, Declarative and <a href="https://en.wikipedia.org/wiki/Reactive_programming">Reactive</a> (Data Flow) e.g. <a href="http://conal.net/fran/tutorial.htm">Elliott 1998</a> (check out <a href="https://vvvv.org/blog/vl-reactive-programming">vvvv</a>)<br />
<br />
<b>Page 153</b>, Chapter 19.2, agile. Iterations are older than Agile e.g. Spiral, <a href="https://en.wikipedia.org/wiki/Rational_Unified_Process#History">Rational</a>, non-iterative Agile systems are e.g. <a href="https://en.wikipedia.org/wiki/Kanban_(development)#Software_development">Kanban</a> and XP e.g. <a href="https://www.goodreads.com/book/show/6278270-the-principles-of-product-development-flow">Reinertsen 2009</a> on product flow. See page 2.<br />
<br />
<b>Page 154</b>, Chapter 19.3, testing. Separate team is <a href="https://en.wikipedia.org/wiki/V-Model_(software_development)">V-Model</a> (see page 51), QA roles are orthogonal to Agile (<a href="https://ieeexplore.ieee.org/document/951502?arnumber=951502&sortType%3Dasc_p_Sequence%26filter%3DAND(p_IS_Number:20576)=">Cunningham 2001</a>), trend is more towards automation and <a href="https://en.wikipedia.org/wiki/Verification_and_validation">validation</a> (in the sense of <i>Reliability</i>, <a href="https://www.goodreads.com/book/show/2383920.Software_Reliability">Meyers 1979</a>) similar to Ops to SRE (<a href="https://landing.google.com/sre/books/">Beyer 2016</a>). Test-Driven-Development known from <a href="http://www.extremeprogramming.org/">XP, Cunningham 1999</a>, Behaviour-Driven-Development. Names for Test Stages canonical defined in <a href="https://books.google.com.sg/books/about/Continuous_Delivery.html?id=6ADDuzere-YC&redir_esc=y">Humble 2010</a> (may be omitted due to page 155)<br />
<br />
<b>Page 156</b>, Chapter 19.5, Patterns. See page 49. Higher level patterns are <a href="https://en.wikipedia.org/wiki/Software_architecture#Architectural_styles_and_patterns">Styles</a> e.g. <a href="https://dl.acm.org/citation.cfm?id=865128">Shaw 1994</a><br />
<br />
<b>Page 159</b>, Chapter 20, performance. Obligatory <a href="http://wiki.c2.com/?PrematureOptimization">Knuth 1974</a> reference. Developing an awareness, <a href="https://en.wikipedia.org/wiki/Fermi_problem">Fermi Estimate</a>, Numbers everyone should know <a href="http://norvig.com/21-days.html#answers">Norvig 2001</a> (<a href="https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html">Animation</a>)<br />
<br />
<b>Page 161</b>, Chapter 20.2, measure first. See Page 159 Knuth, Observability <a href="https://blog.twitter.com/engineering/en_us/a/2013/observability-at-twitter.html">Twitter 2013</a>, <a href="https://landing.google.com/sre/sre-book/chapters/service-level-objectives/">SRE SLA's 2016</a>, <a href="https://www.impactmapping.org/book.html">Impact Mapping 2012</a><br />
<br />
<br />
<div>
<br /></div>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-82399528889753385742018-04-28T12:02:00.001+02:002018-05-06T13:15:31.738+02:00Meshup<i>After a <a href="https://opensource.googleblog.com/2018/03/googlers-on-road-fossasia-summit-2018.html">recent conference</a> I had some good questions and discussions about the current state of services meshes (Istio in this case), thus decided to note down what I find interesting about them.</i><br />
<i><br /></i>I'm a fan of infrastructure-level (<a href="http://www.oracle.com/technetwork/oracle-labs/program-languages/overview/index.html">polyglot</a>) service meshes. The premise excites me as much as Android ten years ago. Working over those years with <a href="http://blog.christianposta.com/microservices/application-network-functions-with-esbs-api-management-and-now-service-mesh/">SOA, EDA, REST Hypermedia API's, API Management, Microservices</a> and language-bound Services Meshes (<a href="http://blog.christianposta.com/microservices/comparing-envoy-and-istio-circuit-breaking-with-netflix-hystrix/">Netflix</a> / Pivotal, Lightbend's <a href="https://www.lightbend.com/products/reactive-platform">Reactive</a>), I tend to be careful about the resolution of their promises though.<br />
<br />
This post is about 3 lesser known effects of service meshes. My <a href="https://unarchitectedsystems.blogspot.sg/2017/12/mandala-or-end-of-control.html">last post</a> already covered complexity, emergence and observability in a more general way so I'll limit those topics.<br />
<br />
<h3>
Reasoning</h3>
<br />
Horizontal scalability in the multi-core age is one of the main arguments behind basically all modern stateless software architectures. <a href="https://speakerdeck.com/cmeiklejohn/language-support-for-cloud-scale-distributed-systems">Consistency in distributed systems</a>, immutability and state-handling are often mentioned common properties, typically used to justify functional programming paradigms. To me it seems though, their most important common property is a complex, emergent network graph structure. Layers and tiers cannot represent contemporary systems anymore. Those systems have a complex adaptive graph structure in space (infrastructure, users, component interactions) and time (<a href="https://12factor.net/codebase">versioning / one codebase</a>, <a href="http://brandon.dimcheff.com/2018/02/rainbow-deploys-with-kubernetes/">rainbow</a> releases, experiments, <a href="https://blog.sonatype.com/promise-theory-and-devops">DevOps</a>, event order, routing).<br />
<br />
We could observe a lot of frameworks in the last years quietly move towards <a href="https://stackoverflow.com/questions/602444/functional-declarative-and-imperative-programming/8357604#8357604">declarative</a> graph alterations. The first I remember were <a href="https://developer.android.com/guide/components/intents-filters.html">Android Intents</a> and <a href="https://puppet.com/blog/puppet%E2%80%99s-declarative-language-modeling-instead-of-scripting">Puppet</a>. But more recently it was <a href="https://codeburst.io/declarative-vs-imperative-programming-a8a7c93d9ad2">React</a>, <a href="https://www.tensorflow.org/programmers_guide/graphs">Tensorflow</a>, <a href="https://beam.apache.org/documentation/programming-guide/">Beam</a>, <a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/architecture.md">Kubernetes</a>, <a href="http://futureofcoding.org/essays/eve/">Eve</a> (RIP) not to forget the re-emergence of <a href="https://www.cockroachlabs.com/docs/stable/architecture/transaction-layer.html#write-intents">SQL combined</a> with flexible <a href="https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed">consistency models</a> and <a href="https://www.confluent.io/blog/making-sense-of-stream-processing/">stream processing</a>.<br />
<br />
All of those come with a robust, well-defined domain vocabulary and set of patterns that allows to precisely define desired behaviour. A graph encourages modularization and reuse, it allows for <a href="https://books.google.com.sg/books?id=3qHs-XYXN0EC&printsec=frontcover&dq=brian+arthur+nature+of+technology&hl=de&sa=X&ved=0ahUKEwjUpbKgw83aAhVHMY8KHZ_cDrkQ6AEIJzAA#v=onepage&q&f=false">division of labor</a>: Better specialization while making the overall concept better understandable. This, in turn, allows a wider, more diverse group of people to <a href="https://jods.mitpress.mit.edu/pub/issue3mothersill">reason and converse</a> about the <a href="https://www.empear.com/blog/software-revolution-part1/">behaviour of the system</a>. The shared language and <a href="https://queue.acm.org/detail.cfm?id=3185224">culture may hopefully enable</a> them to learn alongside the system, what <a href="https://the-composition.com/the-origins-of-opera-and-the-future-of-programming-bcdaf8fbe960">Nora Bateson calls "symmathesy"</a>. It requires all actors in the system to define goals and dependencies, versioned together in one <a href="https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext">codebase across layers and components</a>, documentation, test (spec), customer support feedback and <a href="https://www.thoughtworks.com/radar/techniques/lightweight-architecture-decision-records">architectural decisions</a>. That's why all good <a href="http://isa-principles.org/">(micro) service-architecture principles</a> contain continuous delivery and lean.<br />
<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
There is no single continuous integration and delivery setup that will work for everyone. You are essentially trying to automate your company's culture using bash scripts.</div>
— Kelsey Hightower (@kelseyhightower) <a href="https://twitter.com/kelseyhightower/status/932654629648719872?ref_src=twsrc%5Etfw">November 20, 2017</a></blockquote>
<br />
The biggest difference between those graph-based declarative approaches and Model-Driven concepts (MDD/MDA) is that they are bottom-up, and designed to support evolution*. Instead of requiring a canonical model, tribal ("bounded") domain language or strict interface contracts externally, it is very easy to implement <a href="https://www.infoq.com/presentations/ddd-microservices-2016">domain</a>-event messaging on the infrastructure level, because the infrastructure itself has meaning, allowing for <a href="http://microservices.io/patterns/data/saga.html">independently composed</a> distributed systems - in other words <a href="https://queue.acm.org/detail.cfm?id=2898444">choreographed</a> rather than orchestrated.<br />
<br />
The declarative, domain-event-driven approach shares some advantage of MDA though: The vocabulary, patterns, and visible graph of dependencies. It makes it a lot easier to follow, though, and a lot harder to ignore. Once the implications of changes to the graphs are commonly understood, it's a lot easier to reason about the graph (see, for instance, the <a href="https://research.google.com/pubs/pub35650.html">original Flume paper</a>). On the low level that service meshes target, the infrastructure level, this quality makes it a lot easier to <i>reason and iteratively learn the entire system</i> (including the mesh itself and the DevOps process around), and to version, document, track and test the system.<br />
<br />
<br />
<br />
<a name='more'></a><br />
<br />
<h3>
Convergence</h3>
<br />
Obviously missing from the list of declarative approaches above was <a href="https://maven.apache.org/">Maven</a>. Dependency Management has been an under-appreciated enabler for evolutionary architectures. Imho RPM, Maven (including Android), NPM, GitHub (<a href="https://thenewstack.io/gitops-kubernetes-devops-iteration-focused-declarative-infrastructure/">GitOps</a>) and Docker made platform-agnostic isolated components possible. Maybe it's just nostalgia having wrestled each of those, learning the standard lifecycles, dependency graphs (and <a href="http://wiki.c2.com/?DependencyHell">hell</a>) and application composition along the way: But I do like to think those brought the internet, the complex, over time emerging network graph, into software engineering (as close as we can get to a "Software Internet" in the <a href="https://youtu.be/AnrlSqtpOkw?t=3m10s">words of Alan Kay</a> or, in other words, the "<a href="https://www.morganclaypool.com/doi/abs/10.2200/S00516ED2V01Y201306CAC024">Datacenter as a Computer</a>") - and still pushes for innovations. Imagine <a href="https://youtu.be/ZZmUwXEiPm4?t=8m43s">route-based code splitting</a> used for serverless scaling.<br />
<div>
<br /></div>
My interest in mobile application always came from an integration perspective, from the idea of ubiquitous / pervasive computing, hence my focus on cross-platform and mixed-model approaches. Some might remember my excitement about <a href="https://web.archive.org/web/20160728222128/https://developer.yahoo.com/cocktails/">Yahoo Cocktails</a> and the initial promise of Node (and <a href="http://www.oracle.com/technetwork/articles/java/jf14-nashorn-2126515.html">Nashorn</a>) that JavaScript could bridge client and server. Luckily I was wrong and rather both client and server moved towards polyglot and mixed semantics concepts with many frameworks available in various languages. This extends the dependency graph not only into a versioning time dimension but an additional platform, or runtime (experiments, rolling updates etc) dimension. As I mentioned in "Reasoning", the dependency graph across platforms sparked my interest into Kubernetes. It allows the dependencies of my application, and therefore the exact state of my application, to be described as virtually one graph. It allows convergence, bringing mobile app, (progressive) web apps, server development and backends much closer together. The individual platforms or runtimes become less isolated, they acknowledge their interdependency. The application converges because the whole system moves alongside the vertical slices, the features, rather than in horizontal layers.<br />
<br />
For instance, the Istio service mesh has a lot of architectural, but also release management and DevOps principles baked in - and it allows to <a href="https://istio.io/docs/tasks/telemetry/servicegraph.html">visualize this as graphs</a>. Istio follows the Kubernetes (aka <a href="http://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s06.html">Unix</a> aka <a href="https://en.wikipedia.org/wiki/Robustness_principle">TCP</a>) philosophy of flexibility and robustness by being able to run in concert with Consul, Eureka, Mesos DC/OS and others. The principles and idea behind Istio and Kubernetes translate across platforms, therefore the application styles converge. API's for mobile apps can be defined in universally acceptable ways, and concept like <a href="https://grpc.io/docs/guides/concepts.html#server-streaming-rpc">streaming</a> can be used on both client- and server-side yet having a robust, secure and efficient platform. That's what <i>cloud native</i> really means. It means mashups between service components as they share the same <i>ideas</i>.<br />
<br />
<div>
With API's also <a href="https://dev-blog.apollodata.com/graphql-vs-rest-5d425123e34b">turning into declarative graphs</a> (REST evangelists would add the word "finally"), it's only a matter of time until service meshes will unify call- with event- and function-semantics, and make consistency and synchronisation a <a href="https://grpc.io/docs/guides/concepts.html#synchronous-vs-asynchronous">property (as in gRPC)</a>. I am not sure whether <a href="https://thenewstack.io/the-future-of-kubernetes-is-serverless/">serverless Kubernetes is the future as Brendan</a> or <a href="https://thenewstack.io/the-future-of-kubernetes-is-serverless/">Simon</a> say, as functions themselves can be just as iterative and incomprehensible. But assuming this starting point, hopefully with a bit more transparency, it's possible serverless systems would pick up data from <a href="http://www.fluxicon.com/technology/">process mining</a> and reroute service requests or events to the best backend just like traffic directions do today - based on <a href="https://medium.com/tensorflow/introducing-tensorflow-probability-dca4c304e245">probabilistic distributions</a> and quality guarantees (and trade-offs, see below) the engineers define.</div>
<div>
</div>
<br />
<h3>
Approachability</h3>
<div>
<br />
Declaration is Documentation. As mentioned above, services meshes are a great tool to describe not only the requirements but also the expected and actual behaviour of the system. This makes the system <a href="https://metaparticle.io/posts/welcome-to-metaparticle/">approachable by non-engineers</a>, it allows every stakeholder, from usability designer to product manager to customer support specialist to understand system primitives, their trade-offs and express their requirements in the same terms.<br />
<br />
One problem with distributed, evolutionary mesh, as all graph-based frameworks are exactly those system primitives: The basic principles of the domain have to be <a href="https://www.scottaaronson.com/blog/?p=2410">universally accepted</a> and therefore simple enough to be both learnable and approachable from the outside as well as powerful to describe complexity - without requiring a canonical model or increasing <a href="http://projects.exeter.ac.uk/babbage/rosenb.html">coordination cost (i.e. managers)</a>. You can think of it as essentially the same problem as orchestration vs choreography. Orchestration, i.e. implicit models, work better in the start of a project when someone has a clear idea what to do, they are faster to iterate. However, at some point the communication effort of translating that idea to a team eliminates the iteration efficiency. Smaller components help up to a certain point, but they still rely on the one engineer with this concept in her head. Declarative approaches, like choreography, require more learning in the beginning, but can scale horizontally based on a <a href="https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life">few core rules</a> and <a href="https://hackernoon.com/a-hitchhikers-guide-to-consensus-algorithms-d81aae3eb0e3">consensus</a> mechanisms.</div>
<div>
<br /></div>
<div>
It took me a long time to understand why SRE / CRE are so focused on <a href="https://www.youtube.com/watch?v=tEylFyxbDLE&t=2s">measuring SLO</a>, toil and error. Finally it clicked: In order to improve effects on risks, they have to be <a href="https://www.whatmatters.com/">measurable</a> (finely balanced with <a href="https://www.lesswrong.com/posts/YtvZxRpZjcFNwJecS/the-importance-of-goodhart-s-law">Goodhart's Law</a>), as risk itself is just probabilities. In the end, architecture is risk management, and risk vs <a href="https://speckyboy.com/building-minimum-viable-products-spotify/">delight</a> is something most designers, support teams, product managers, product owners, business or clients can work with (or at least has some <a href="https://en.wikipedia.org/wiki/Planning_fallacy">defined</a> <a href="https://en.wikipedia.org/wiki/Roy_Amara#Amara's_law">biases</a> to test for). There is <a href="http://alvaro-videla.com/2017/01/metaphors-we-code-by.html">enough metaphors in software engineering</a> for everyone to have their favourite. Describing system behaviour as a hierarchy of interacting behaviours is something that can be translated into metaphors for almost every domain. Good metrics are not just figures, they are metaphors, a model, a map, that naturally favours simplicity over complexity. Approachable system boundaries also allow for easier planning (estimation, decision-making) and <a href="https://queue.acm.org/detail.cfm?id=3197520">focusing types of work or automation</a>.<br />
<br />
Stakeholders can all define the confidence level and tune the behaviour the meet the requirements. The evolutionary* aspects of <a href="https://itrevolution.com/book/accelerate/">Agile, Lean, Design Thinking and DevOps</a> are, in this sense, all variations of <a href="https://am207.github.io/2017/wiki/boxloop.html">Box's loop</a>. From this perspective, a <a href="https://landing.google.com/sre/book/chapters/service-level-objectives.html">SLO</a> can be seen as a <a href="https://books.google.com.sg/books?id=pYI2DwAAQBAJ&pg=PA15&hl=de&source=gbs_toc_r&cad=4#v=onepage&q&f=false">fitness function</a>*, an objective assessment of an architectural concern (sometimes called the "<a href="https://en.wikipedia.org/wiki/List_of_system_quality_attributes">ilities</a>") and how to optimize it within the system (that is, <a href="http://alistair.cockburn.us/Characterizing+people+as+non-linear,+first-order+components+in+software+development">including the team</a>), for instance <a href="https://michaelfeathers.silvrback.com/is-technical-debt-just-a-metaphor">how to fix tech tebt</a>. In other words, as I said in the talk: <a href="https://landing.google.com/sre/book/chapters/communication-and-collaboration.html">SRE</a> is probably closer to the Architect role than most Architects were ever before.<br />
<br />
With measures against your model (the architecture), you get <a href="https://www.infoq.com/articles/charity-majors-observability-failure">observability in time</a>. Observability is to a software system what <a href="https://en.wikipedia.org/wiki/Sallie_Gardner_at_a_Gallop">Muybridge's series</a> was to the real world. It makes formerly incomprehensible behaviour visible, quantifiable and understandable. It helps the team to <a href="https://medium.com/@rakyll/microservices-observability-26a8b7056bb4">reason about the behaviour (see JBD)</a>, to compare mental models and concepts, to express and test risk as well as delight. Observability is one of the main tenets of Serviceability.<br />
<br />
<a href="https://commons.wikimedia.org/wiki/File:The_Horse_in_Motion_high_res.jpg#/media/File:The_Horse_in_Motion_high_res.jpg"><img alt="The Horse in Motion high res.jpg" height="136" src="https://upload.wikimedia.org/wikipedia/commons/d/d2/The_Horse_in_Motion_high_res.jpg" width="220" /></a><br />
By Provided directly by Library of Congress Prints and Photographs Division, Public Domain, <a href="https://commons.wikimedia.org/w/index.php?curid=57260211">Link</a><br />
<br />
Or, to count for the added dimensions it's similar to the psychogeography work of situationists like <a href="https://www.cddc.vt.edu/sionline/si/theory.html">Guy Debord with his Dérive</a> who set out to provide an objective, historical, both temporal and spatial view of the city by mapping it's movements. Thanks to this group our cities are finally becoming more liveable and include community-based planning.<br />
<br />
<blockquote class="tr_bq">
Making the city observable implies allowing the city to become an environment for learning</blockquote>
This quote from Richard Saul Wurman's to make "the city observable" comes from M. Steenson's "Architectural Intelligence" p. 92 who also has a <a href="https://books.google.com.sg/books?id=x8lDDwAAQBAJ&lpg=PP1&dq=architectural%20intelligence&hl=de&pg=PA61#v=onepage&q&f=false">great quote on how patterns rearrange power on p. 59</a>. Without stretching the <a href="https://unarchitectedsystems.blogspot.sg/2013/04/atzende-architekturmetaphern.html">urbanism metaphor too much</a>, I do like the connection between observability and learning here - and in Steenson's book. It highlights why we need approachable systems that allow reasoning about them - because that's the major pre-requirement for any kind of real iterative development able to deal with complex emerging features, not just a type of external power defining a fitness for an evolutionary architecture.<br />
<blockquote class="tr_bq">
Debugging is an iterative process, involving iterative introspection of the various observations and facts reported by the system [...] Evidence needs to be reported by the systems in the form of highly precise and contextual facts and observations</blockquote>
Learning is what Cindy Sridharan in <a href="https://medium.com/@copyconstruct/monitoring-and-observability-8417d1952e1c">Monitoring and Observability</a> quoted above calls introspection. Introspection is more than just iterative or feedback in evolution, because it includes both reason and empathy. Introspection is the philosophy of serviceability. We can apply Actor-Network-Theory and <a href="http://markburgess.org/promises.html">promise theory</a> to describe the characteristics not only in individual SLA's, but in consistency terms, all the way up to user expectations and to the analysis of unexpected failures.<br />
<br />
Having those models, metaphors at hand, and being able to relate the metrics through introspection we can explain systems. We can use tools from machine learning, for instance <a href="https://christophm.github.io/interpretable-ml-book/">explainability</a>, do dig deeper it understanding <a href="https://arxiv.org/abs/1803.03453">surprising behaviour of evolutionary* systems</a>. ML is already applied to a wide range of software engineering problems, from <a href="https://deepmind.com/blog/deepmind-ai-reduces-google-data-centre-cooling-bill-40/">data center infrastructure</a> to <a href="https://arxiv.org/abs/1712.01208">database indices</a> and <a href="http://blog.mgechev.com/2018/03/18/machine-learning-data-driven-bundling-webpack-javascript-markov-chain-angular-react/">bundling JavaScript dependencies</a>. There is no reason why we can't apply it to code itself, and to service connections and dependencies not in an AI sense, but in a tool sense to help us understand.<br />
<br />
I wrote about the <a href="https://unarchitectedsystems.blogspot.sg/2017/12/mandala-or-end-of-control.html">end of control</a> earlier, but what excites me about these new, descriptive, observable mashups of systems is that they tackle the <a href="https://queue.acm.org/detail.cfm?id=3185224">socio-technical</a> problem. Maybe they will allow a psychogeography of our systems one day. And with that making software engineering not only more reasonable and scientific but more open, more communal and less tribal.</div>
<div>
</div>
<br />
<div style="font-family: times;">
<div style="margin: 0px;">
<br /></div>
<div style="margin: 0px;">
<br /></div>
<div style="margin: 0px;">
<span style="font-size: x-small;">*) Using Evolution here mostly in the currently prevailing sense of development as popularized <a href="https://www.thoughtworks.com/books/building-evolutionary-architectures">in the book</a>. However, I am personally more learning towards an Emergence model which allows more radical changes and smells less like <a href="https://en.wikipedia.org/wiki/Naturalism_(philosophy)#Karl_Popper">Naturalism</a>. <a href="http://www.simonandschuster.com/books/The-Nature-of-Technology/W-Brian-Arthur/9781416544067">Brian Arthur</a> would call the first "Evolution in the narrow sense" whereas I prefer "Evolution in the full sense" of the system as a whole being interconnected in its fate.</span></div>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-84001131158406341262017-12-31T17:24:00.000+01:002018-01-07T04:36:10.794+01:00Mandala or The end of control<span style="font-family: inherit;"><i>A good friend of mine asked me why I don’t blog anymore, so I took my new-years flight as an opportunity to write some random thoughts down. Happy new year!</i></span><br />
<span style="font-family: inherit;"><i><br /></i>We used to build Information Systems or Control Systems. Sometimes, they were clumsily merging - but finally become something entirely new: Intelligent Intent Systems. I don’t like catchy slang, though, let’s just say we finally have universal “Systems”.<br /><br />A sufficiently <a href="http://www.informit.com/articles/article.aspx?p=1380369">lean online shop</a> is essentially an easy interface for sending a signal into an extremely complex, often entirely automated, <a href="https://rctom.hbs.org/submission/online-to-offline-o2o-and-the-future-of-e-commerce/">logistics chain</a> that translates information into physical control commands - the Information System part manages the feedback loop to humans. The more feedback, an idea from Control Systems, became incorporated into Information Systems, the smarter, faster, and more intuitive we were able to interact. Like frames in a movie, we’re now at the point where it becomes seamless and continuous. The IoT, <a href="https://twitter.com/chenoehart/status/945201794241781760">spatial computing</a>, but actually just <a href="https://www.wired.com/story/the-other-tech-bubble/">technology becoming synonymous with information</a> close the loop back into our world (the <a href="https://dynamicland.org/">"real" world</a>). We used to interpret our systems as essentially closed and deterministic, in both imperative and functional programming styles. But the new types of systems have rapidly become <a href="https://www.coursera.org/specializations/probabilistic-graphical-models">Probabilistic Systems</a>.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">With the <a href="https://research.google.com/pubs/pub35290.html">planet-scale cloud</a> of distributed services risk-driven models, from <a href="http://unarchitectedsystems.blogspot.sg/2017/03/on-other-side-of-certainty.html">complexity and uncertainty</a> theory, have taken center stage. With Machine Learning rapidly <a href="https://twitter.com/JeffDean/status/940122424871227392">surpassing human experience</a> in programming <a href="https://www.theguardian.com/technology/2017/dec/07/alphazero-google-deepmind-ai-beats-champion-program-teaching-itself-to-play-four-hours">and research</a>, we will ourselves have to model the real, human, world in a <a href="http://blog.kubernetes.io/2017/05/managing-microservices-with-istio-service-mesh.html">descriptive</a>, <a href="https://www.cs.cornell.edu/courses/cs4110/2016fa/lectures/lecture33.html">probabilistic</a> way as part of those systems, <a href="https://medium.com/@copyconstruct/testing-microservices-the-sane-way-9bb31d158c16">observing</a> and inferring, rather than imperatively defining the <a href="https://twitter.com/evgenymorozov/status/943101303445704704">message flows between agents</a>, and its consistency properties, the <a href="https://cloud.google.com/dataflow/blog/dataflow-beam-and-spark-comparison#logistics">data flows between processors</a> or structural limitations (think column- vs row-oriented data).<br /><br />We don’t observe outside of the system, but as a part of it. Much like quantum physics, psychology or sociology (especially of power), told us. We humans are only agents receiving signals in this system. We inhabit a <a href="http://matteopasquinelli.com/eye-of-the-algorithm/">second-order Cybernetics</a> <a href="http://www.hkw.de/de/programm/projekte/2015/technosphere/technosphere_start.php">Technosphere</a> we call <a href="https://www.theatlantic.com/science/archive/2016/04/the-illusion-of-reality/479559/">reality</a>, built on <a href="https://stories.platformdesigntoolkit.com/the-best-reads-on-platforms-from-2017-d744284c368a">platforms that</a> define what they want and <a href="https://boingboing.net/2017/12/30/cooccurrence-not-causation.html">value economically</a> in entirely new, sometimes alien ways, like real-time exchanges including spatial and social dimensions - with the <a href="https://twitter.com/evgenymorozov/status/925814881617678338">gig- and experience economy only being the tip of the iceberg</a>. It’s not a matrix or a mastermind though, it’s just a real, <a href="https://mitpress.mit.edu/books/techno-human-condition">techno-human</a> ecosystem with its own, uncontrollable <a href="https://en.wikipedia.org/wiki/Instrumental_convergence">evolutionary goals</a>.<br /><br />Despite me not liking the gig economy, I like the idea of evolutionary systems. <a href="https://martinfowler.com/articles/evo-arch-forward.html">Most writing</a> focuses on the iterative process to avoid unpredictability, though, assuming some external, given, linearity, to incorporate feedback. That’s important for organizations, but it’s not mentioned where the feedback comes from: The goal, the adversary, the <a href="https://www.inc.com/magazine/19950915/2622.html">predator</a>, or the local maximum.</span><br />
<div>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">I mainly work on the integration between real world and software, the hard end of mobile / ubiquitous / spatial / pervasive computing. Building independent, distributed service meshes, in a DevOps and Design Thinking (or <a href="http://designopssummit.com/2017/11/08/tripnote-jeff-sussnas-designops-can-learn-devops/">DesignOps</a>, whatever the cool kids call it these days) way. In such systems, you’re always going after the local maximum, the goal is unclear, more often than not multiple conflicting ones. You’re naturally dealing with (domain) <a href="http://philcalcado.com/2017/08/03/pattern_service_mesh.html">verticals</a> <a href="http://wiki.c2.com/?SliceSystemsVertically">rather</a> than horizontals. Every day has a new trade off between best experience possible and the technical realities. Those systems are not linear evolutions, they are <a href="https://en.wikipedia.org/wiki/Mandala_(political_model)">mandalas</a> of expanding and contracting system boundaries. They have to be <a href="https://medium.com/@rakyll/googles-approach-to-observability-frameworks-c89fc1f0e058">observable</a>, though, as with observation comes empathy, and with empathy learning.<br /><br />In the future, we may refactor the parameters of our systems based on deeper insights about their <a href="https://youtu.be/Qi1Yry33TQE?t=12m13s">non-deterministic behaviour</a>. That's what I like about SRE-style work. It's the non-deterministic, the probabilistic part of software engineering. It focuses on <a href="https://landing.google.com/sre/book/chapters/effective-troubleshooting.html">observability and serviceability</a>, and ML <a href="https://landing.google.com/sre/book/chapters/postmortem-culture.html">RCA</a>’s involve <a href="https://www.darpa.mil/program/explainable-artificial-intelligence">explainability</a> - correctness becomes an optimization goal, not an axiom. When I spoke about <a href="http://unarchitectedsystems.blogspot.sg/2013/10/the-application-server-is-dead.html">Spanner</a> first time publicly in 2012, compared it to the twisted experience of time in movies like Spaceballs and The Hitchhiker's Guide to the Galaxy. The powerful takeaway is: Nothing is fixed, if we can reason about it, <a href="https://medium.com/civic-tech-thoughts-from-joshdata/so-you-want-to-reform-democracy-7f3b1ef10597">we can change it</a>.<br /><br />To understand the magnitude of change to our profession, we have to understand the societal context. Of all the possible futures, a <a href="https://twitter.com/doctorow/status/941381972277977088">dystopian</a> scenario is interesting here - I shorten my version after watching Charly Stross’ <a href="https://media.ccc.de/v/34c3-9270-dude_you_broke_the_future">talk from 34c3</a> which tells it better: In this scenario, the we is not a harmonic, transhuman, unity. The <a href="https://thefrailestthing.com/2017/12/03/the-rhetorical-we-and-the-ethics-of-technology/">new we</a> is us and algorithms from us, for us. A <a href="https://www.theguardian.com/technology/2017/oct/05/smartphone-addiction-silicon-valley-dystopia">dark</a> (in the sense of dark matter) singularity not of eternal life but <a href="http://www.academia.edu/14269575/Hannah_Arendt_and_Algorithmic_Thoughtlessness">thoughtless</a>, <a href="https://en.wikipedia.org/wiki/Homo_Deus:_A_Brief_History_of_Tomorrow">mutual uncertainty</a>, where <a href="https://www.technologyreview.com/s/608986/forget-killer-robotsbias-is-the-real-ai-danger/">biased</a> <a href="https://www.amazon.de/Weapons-Math-Destruction-Increases-Inequality-ebook/dp/B019B6VCLO/ref=sr_1_1_twi_kin_2?ie=UTF8&qid=1483849919&sr=8-1&keywords=Weapons+of+Math+Destruction">algorithms</a> and biased, <a href="http://cramer.pleintekst.nl/essays/crapularity_hermeneutics/">dumbed down</a> or even corrupt, <a href="https://hackernoon.com/cognitive-biases-in-programming-5e937707c27b">people</a> <a href="https://thefrailestthing.com/2016/12/30/our-digital-idiocy/">push each other</a> further into the edges, not becoming market segments but mobs which <a href="https://www.nytimes.com/2017/12/26/opinion/trump-liberals-armageddon.html">reinforce</a> themselves. The algorithm is as helpless as their users, because the society and <a href="https://www.nytimes.com/2017/06/16/upshot/the-amazon-walmart-showdown-that-explains-the-modern-economy.html">economy</a> around it <a href="https://en.wikibooks.org/wiki/Systems_Theory/Entropy#Combating_Entropy">require the entropy as fuel</a> making regulation impossible.<br /><br />Quick, <a href="https://twitter.com/chethaase/status/925715289244819458">personalized</a>, adjustment, <a href="https://www.amazon.de/Verlernen-Denkwege-bei-Hannah-Arendt/dp/3882216050/ref=asap_bc?ie=UTF8">unlearning</a>, or one-shot learning, can maybe avoid this scenario - it seems AI is <a href="https://www.technologyreview.com/s/609710/neural-networks-are-learning-what-to-remember-and-what-to-forget/">already forgetting</a> easier than us, <a href="https://www.technologyreview.com/s/603381/ai-software-learns-to-make-ai-software/">controlling and optimizing itself</a> faster than us, collaborating and sharing <a href="https://distill.pub/2017/aia/">surprising insights</a> <a href="http://www.wired.co.uk/article/collaborative-robots-work-with-humans">nicer than us</a>. In a "<a href="https://davidbarber.github.io/blog/2017/11/07/Learning-From-Scratch-by-Thinking-Fast-and-Slow-with-Deep-Learning-and-Tree-Search/">thinking fast, thinking slow</a>" model, maybe the human is ought to become <a href="https://twitter.com/jessitron/status/938961410129788928">the "slow" part</a> - as Nate Silver <a href="http://fivethirtyeight.com/features/rage-against-the-machines/">once put it</a> "complementary roles that computer processing speed and human ingenuity can play in prediction". Instead of turning away, the human needs to be enabled to act.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Right now, we train machine learning systems by saying “yes” - soon we will reach the point where <a href="http://ruder.io/transfer-learning/index.html#continuouslearning">transfer</a> is so good that we’ll start saying “no”. That “no” has to be <a href="https://qz.com/662009/the-sec-tried-to-fix-a-finance-problem-and-created-a-computer-science-problem-instead/">slow enough</a>, it has to be <a href="https://plato.stanford.edu/entries/arendt/#JudWinTho">thoughtful</a>, and it has to weigh more than assumed silent consent. We have to introduce reason, <a href="https://www.smashingmagazine.com/2017/11/designing-ethics/">empathy and ethics</a>, not only into individual machine learning models, but into the whole system that is driven by technology, into all of the information and the human organizational complex around it.<br /><br />What excites me about working with systems from an observability, serviceability, and <a href="https://en.wikipedia.org/wiki/Explainable_Artificial_Intelligence">explainability</a> perspective is that we can bring all of that rich knowledge from physics, psychology or sociology, hermeneutics, but also art in, and start reasoning about the overall behaviour, rather than deterministic, imperative requirements. We only have to keep talking to each other - and try to understand.</span></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-54756399522167726952017-03-19T16:26:00.003+01:002017-03-19T16:26:44.832+01:00On the other side of Certainty<div style="text-align: right;">
"<i>We have to create the preconditions under which</i></div>
<div style="text-align: right;">
<i>Exaptation can happen naturally...</i></div>
<div style="text-align: right;">
<i>which actually means introducing inefficiency into systems</i>"</div>
<div style="text-align: right;">
<a href="https://youtu.be/pHjeFFGug1Y?t=7m">Dave Snowden</a></div>
<div style="text-align: right;">
<br /></div>
<div style="text-align: right;">
"<i>The question is no longer how systems behave ...</i></div>
<div style="text-align: right;">
<i>but how to ask for the probability distribution</i></div>
<div style="text-align: right;">
<i>of the properties that change the system</i>"</div>
<div style="text-align: right;">
Helga Nowotny, <a href="http://as.wiley.com/WileyCDA/WileyTitle/productCd-074568761X.html">The Cunning of Uncertainty</a></div>
<br /><br />Moving from project- into product world last year, I wanted to experience the real long-term view, because only strategy can tackle complexity. And our software grows more complex, though arguably less complicated, every year. But I also wanted to understand uncertainty, the dark matter of complexity, better. After some time in, I understand it's probably more important for an architect* to have strong operations knowledge than very strong <a href="https://medium.com/make-better-software/against-the-whiteboard-f1df0013954f#.t98hxw6kr">algorithmic skills</a>.<br /><br />Resilience is sometimes <a href="http://www.improvement-services.nl/blog/?p=525">mistaken</a> as being adaptive, or agile. It just means expecting uncertainty and disruption, but also working properly under normal conditions. Just calling <a href="https://www.theguardian.com/technology/2017/mar/07/uber-work-culture-travis-kalanick-susan-fowler-controversy">everything disruptive</a>, and reacting in an agile way to every random demand, is not resilient. For instance, "<a href="http://highscalability.com/blog/2013/8/22/the-datacenter-as-a-computer-an-introduction-to-the-design-o.html">The datacenter as a computer</a>"** came to us, rather unexpected, from a relatively "boring" infrastructure level, rather than from new frameworks and startups, but also not from consortium standards and language ecosystems. Similarly, in true grassroots manner, polyglot programming and <a href="https://blog.codinghorror.com/the-principle-of-least-power/">JavaScript</a> on top of Unix principles catapulted us into the 21st century and will <a href="https://www.youtube.com/watch?v=GNR9El3XWYo">eventually</a> enable domain logic to exist, as Adrian Cockcroft calls it, in <a href="https://medium.com/m/global-identity?redirectUrl=https://read.acloud.guru/evolution-of-business-logic-from-monoliths-through-microservices-to-functions-ff464b95a44d">functions</a>, unaware of most technological constraints. In order to understand resilience, you need to <a href="https://www.youtube.com/watch?v=B_W48ZsIqSo">care</a> about your product, and want to improve it, have a real goal, a story you want to tell. Ironically, you have to be a little bit un-adaptive, inefficient and un-agile in order not to <a href="https://en.wikipedia.org/wiki/Overfitting"><i>overfit</i></a>, but to really improve.<br /><br />We are still waiting for the <a href="http://wiki.c2.com/?ThereAreExactlyThreeParadigms">4th paradigm of programming</a>. My guess is it's going to be more than just <a href="https://plus.google.com/+JuhaniLehtim%C3%A4ki/posts/Lb4A1EefaUP">goal-oriented</a>, it will be <a href="https://www.oreilly.com/ideas/probabilistic-programming">probabilistic</a>, in the sense of an <a href="http://blog.fastforwardlabs.com/2017/01/30/the-algorithms-behind-probabilistic-programming.html">abstract goal corridor</a>. While engineers will live inside the goal corridor, making sure its workings are predictable, specializing in certainty, architects live on the outside, <a href="https://blogs.technet.microsoft.com/johnla/2015/09/26/the-inside-story-behind-ms08-067/">in the long tail</a> of the probability distribution, the Multiverses where the <a href="https://en.m.wikipedia.org/wiki/Dragon_King_Theory#Black_swans_and_dragon_kings">Dragon Kings</a> live, outside predictive models, specializing in uncertainty. That’s what makes a system resilient. It implies engineers have a fairly static/discrete/fitted view of a system, the perfect snapshot, the position, whereas architects have a time-smeared continuum perspective, the story, the momentum. It's the whole <a href="https://medium.com/@mrettig/notes-on-emergence-5290adb48319#.cgniv15c0">story</a>, not only the goal, that differentiates between emergence and evolution. But it's also important to understand that both of those roles are equally <a href="https://salfreudenberg.wordpress.com/2013/07/09/who-says-programming-isnt-creative/">creative</a> and forward-looking, none is a "higher level", they are two sides of the same coin.<br /><br />Helga Nowotny's distinction between risk and uncertainty fits nicely here - risk can be computed, uncertainty not. Risk a relation between snapshots, uncertainty is the continuum in between. A weather forecast has risk, the climate is uncertainty. In that sense, a <a href="https://www.crcpress.com/Just-Enough-Software-Architecture-A-Risk-Driven-Approach/Fairbanks/p/book/9781439812341">risk-driven architecture</a> involves everyone, but uncertainty needs a different perspective. The future leaves to the architect (the systems-of-systems-carer, archeological gardener, forensic librarian, ontological cartographer or whatever you like to call her) the role of the curator, and maybe narrator, of the <a href="https://placesjournal.org/article/a-city-is-not-a-computer/#.WKFe3IjsATg.facebook">uncomputable</a>. Paraphrasing what <a href="http://akkaarchitects.com/vision-and-process/">Akka</a> Architects say: <i>Architects don't design system interactions, they curate the context for a discourse about system interactions</i>.<br /><br /> <div style="text-align: right;">
"<i>It is not a question of establishing limits with walls,</i></div>
<div style="text-align: right;">
<i>but by other means</i>"</div>
<div style="text-align: right;">
<a href="https://www.theguardian.com/artanddesign/2017/mar/01/pritzker-architecture-prize-rcr-arquitectes">RCR Architects</a></div>
<br />Why is all of this important, why should we get used to speak in terms of probabilities, but, most importantly, tackle certainty and uncertainty differently, but with the same importance?<div>
<br /></div>
<div>
In my last job I learned the importance of maintainability and traceability. In every single one of my projects the first thing I introduced was proper monitoring, alerting and analytics - Robustness was core, as was accountability, to be able to become lean and agile. With new infrastructure, whether in the cloud or not, this has become the default. The battle for certainty, traceability, and robustness is won.<div>
<br /> While we were busy fighting this battle, uncertainty has come back, as Ms. Nowotny would put it, "systemic risk", as risk deeply embedded in the complex relationships of our services: "Uncertainty switches <i>gestalt</i>". A cloud service going down is a risk that can be predicted - the dashboard not showing it, because it is hosted <a href="https://www.theregister.co.uk/2017/03/01/aws_s3_outage/">on the <i>same</i> cloud service is systemic risk</a>, the type of Dragon King uncertainty which we don't expect. Soon it will be mainstream that coders will <a href="https://openreview.net/pdf?id=ByldLrqlx">pair with AI</a>, and operations and product teams will regularly train models rather than manually define metrics. The relationships between components will become so complex that, more often than not, it will take a long time to even <i>recognize</i> errors, or feature usage, let alone find the root cause or customer need.<br /><br />Despite all the <a href="https://twitter.com/hjameswilson/status/843250971195162624">data</a> we accumulate, <a href="https://qconsf.com/sf2016/sf2016/presentation/creating-culture-observability-stripe.html">Observability</a> does still not mean <a href="http://www.darpa.mil/program/explainable-artificial-intelligence">Explainability</a> (<i>sometimes as beautifully <a href="https://research.googleblog.com/2016/11/zero-shot-translation-with-googles.html">visualized</a> as our old architecture diagrams</i>) or <a href="https://hips.seas.harvard.edu/blog/2013/04/29/introspection-in-ai/">Introspection</a>, the ability to rationalize the <a href="https://motherboard.vice.com/en_us/article/we-need-to-tell-better-stories-about-our-ai-future?utm_source=mbtwitter">inscrutability</a> of AI.<br /><br />Most of our architecture diagrams are nothing more than <a href="http://99percentinvisible.org/episode/thomassons/">Thomassons</a>, useless depictions of a fictional state which we only keep because it took us so long to create them. But similar to code, it's actually more productive to delete them. What we need is a visualization of the statistical complexity of actual state. In my book, a good friend of mine wrote a story how we made network traffic audible in order to hear inconsistencies - that's what we need to understand the architecture of a system. A real-time <i>explanation</i> of what's going on, mapped to the architecturally relevant components, such as interfaces, deployment units (like functions) and, last but not least, rules inside machine learning systems.<br /><br /> I am looking forward to new, real-time programming and data toolkits, think <a href="http://www.chris-granger.com/2015/08/17/version-0/">Eve</a>, <a href="http://jupyter.org/">Jupyter</a> and <a href="https://glitch.com/">Glitch</a>, especially because they enable a different kind of coder to build software. And with serverless and deep learning we will be able to scale those apps and the data required quicker than ever before. But it will require architects to understand them, if something is wrong, if a use case needs to be developed or a feature is behaving <i>unexpectedly</i> - i.e. in operations and product development. These architects won't be able to look at diagrams anymore, they won't be certain about documentation (not that they ever were), and they won't even be able to observe the system in its entirety. Architects will indeed become archeological gardeners or forensic librarians. Most importantly, they will become like anthropologists or biologists in the early days, trying to understand evolution, with the help of experimentation and collection. We'll finally see architects developing models instead of diagrams. And with that, we will see very different people in this role too. Which is very exciting.<br /><span style="font-size: x-small;"><br />*) separating this from a <a href="https://leanpub.com/talking-with-tech-leads">Tech Lead</a> here, sometimes it can make sense to combine the two roles, though<br />**) e.g. serverless computing platforms such as <a href="https://aws.amazon.com/de/lambda/">Lambda</a> and globally distributed databases such as <a href="https://quizlet.com/blog/quizlet-cloud-spanner">Spanner</a>, with them a different approach to time, where evergreen becomes an axiom, but also a comeback of <a href="http://www.felienne.com/archives/2974">Spreadsheets</a> through functional programming and <a href="https://cloud.google.com/bigtable/">k/v stores</a></span> <style>
<!--
/* Font Definitions */
@font-face
{font-family:Times;
panose-1:2 0 5 0 0 0 0 0 0 0;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:3 0 0 0 1 0;}
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;
mso-font-charset:1;
mso-generic-font-family:roman;
mso-font-format:other;
mso-font-pitch:variable;
mso-font-signature:0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:-536870145 1073786111 1 0 415 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0cm;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:Calibri;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
a:link, span.MsoHyperlink
{mso-style-noshow:yes;
mso-style-priority:99;
color:blue;
text-decoration:underline;
text-underline:single;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-noshow:yes;
mso-style-priority:99;
color:#954F72;
mso-themecolor:followedhyperlink;
text-decoration:underline;
text-underline:single;}
p
{mso-style-noshow:yes;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;}
span.apple-converted-space
{mso-style-name:apple-converted-space;
mso-style-unhide:no;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-family:Calibri;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;
mso-header-margin:35.4pt;
mso-footer-margin:35.4pt;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
-->
</style></div>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-28976153543466156832016-11-26T08:32:00.003+01:002016-11-26T09:05:50.104+01:00Letter to my future self<i>This is a variation on a <a href="https://www.futureme.org/">letter to my future self</a>, just a public one. A reminder what I thought, written on the last day in my old job - exactly 6 months ago. I would write it differently now, so I’ve added many current links. Don’t read it as a manifesto, read it as a snapshot.</i><br />
<i><br /></i>
When I quit my first full-time employment in a start-up, after my own one, it was because I felt start-ups disguised the struggles for power under a hood of coolness and artificial problems.<br />
<br />
<div>
I joined consulting to be outside of power structures, and tackle real problems (as real as it gets in corporations). Accenture Digital was inspiring, and <a href="http://creativegeneralist.com/2005/07/thinking-linking-doing/">t-shaped/widened</a> my horizons - more than once. It allowed me to thrive in technology, strategy, product prototyping, <a href="https://www.infoq.com/news/2014/10/pragmatic-dave-agility">Agile</a>, <a href="https://contently.com/strategist/2016/07/29/can-accenture-take-over-advertising/">design thinking</a> and to <a href="https://uxmag.com/articles/busting-the-myth-of-the-giant-green-button">trust liberal arts</a>. I will always remember the learnings in those years: All the opportunities I had to meet great, diverse, truly interesting people across the globe, to learn about dozens of domains, to build digital-first, mobile-first, <a href="http://hood.ie/blog/say-hello-to-offline-first.html">offline-first</a> products, omni-chanel <a href="http://www.quirksmode.org/blog/archives/2016/09/web_development.html">SPA</a>, Microservices with vertical slices, <a href="http://unarchitectedsystems.blogspot.sg/2016/05/you-dont-manage-api-api-manages-you.html">API management</a> and <a href="http://camel.apache.org/">EAI</a>, or <a href="https://blog.codecentric.de/en/2016/08/cqrs-es-akka/?utm_content=38895284">CQRS</a> systems. In short I am grateful to have been inspired and had the chance to inspire others.<br />
<br />
Really building stuff around existing problems is what makes technology consulting so amazing. Seeing what you’ve built in the wild, changing the way an industry runs. Building a team around a problem, changing their perspective, and mine. Unlearning is just as important a skill as learning. We are constantly developing, iterating, behaviours, patterns and principles of <a href="http://systemswe.love/">good systems</a>, including its human actors. These principles always come with a drawback, outweigh by the advantages. <a href="http://queue.acm.org/detail.cfm?id=2956643">Microservices</a> might manifest <a href="https://twitter.com/neil_conway/status/743086761493008384">team</a> <a href="https://twitter.com/colettecello/status/749052871719591937">separation</a> and can lead to <a href="https://medium.com/elepath-exports/spatial-interfaces-886bccc5d1e9#.kpnncvvmm">incoherent</a> <a href="https://www.smashingmagazine.com/2016/06/designing-modular-ui-systems-via-style-guide-driven-development/">style</a> in <a href="http://blog.gardeviance.org/2016/05/wardleys-doctrine.html">end-user</a> product experience. Event systems enforce eventual consistency, and not every organization is ready for <a href="http://pbs.cs.berkeley.edu/">probabilistic</a> thinking. <a href="https://twitter.com/kventil/status/725645207799209984">Containers</a> can degrade <a href="https://blog.docker.com/2016/05/docker-security-scanning/">security</a> (and with it privacy) to a part of <a href="http://thenewstack.io/assessing-the-state-current-container-security/">commerical opt-in</a> <a href="https://www.infoq.com/news/2016/06/docker-containers-lifecycle">dependency-hell</a> rather than reliable infrastructure;<a href="https://hbr.org/2015/09/design-thinking-comes-of-age"> Design Thinking</a> either glosses over <a href="https://hbr.org/2011/08/henry-ford-never-said-the-fast">faster horses</a> or <a href="http://bikeshed.com/">fakes ownership</a> because agencies just <a href="https://contently.com/strategist/2016/07/29/can-accenture-take-over-advertising/">rebrand</a>; <a href="http://www.monotype.com/blog/articles/qa-hayley-hughes-talks-good-brand-health">Patterns</a> sometimes promote solutions that only look good on paper; CI, Lean and Feedback prefer simplistic, <a href="http://dl.acm.org/citation.cfm?id=203256">biased</a>, <a href="http://www.theatlantic.com/science/archive/2016/09/the-inevitable-evolution-of-bad-science/500609/">measurable KPI’s</a> over <a href="http://cdixon.org/2014/03/16/theres-just-a-tremendous-amount-of-craftsmanship-in-between-a-great-idea-and-a-great-product/">long term goals</a>. <a href="http://jvns.ca/blog/2016/10/26/a-few-questions-about-open-source/">OSS</a>, <a href="http://www.theregister.co.uk/2015/01/08/erik_meijer_agile_is_a_cancer_we_have_to_eliminate_from_the_industry/">Agile</a> and <a href="http://www.computerweekly.com/news/450300377/Chef-CTO-urges-action-to-prevent-DevOps-staff-burnout">DevOps</a> can establish a status-quo burn-out <a href="https://www.infoq.com/news/2016/05/agile-dead-again">treadmill</a> of minor, over-hyped, <a href="http://nymag.com/selectall/2016/09/andrew-sullivan-technology-almost-killed-me.html">pseudo-improvements</a>, and <a href="https://hbr.org/2016/07/beyond-the-holacracy-hype">holocracy</a> could build a cast system; Opinionated often just means <a href="http://brandonhays.com/blog/2014/10/19/open-source-tech-and-tribalism/">tribal</a> or <a href="http://conversations.e-flux.com/t/jurgen-habermas-on-how-to-derail-right-wing-populism/5372">elitist</a> (for lack of a better word); competing <a href="http://www.theregister.co.uk/2016/07/07/oracle_java_ee_8/">API</a>, <a href="https://openai.com/blog/openai-gym-beta/">AI</a> and <a href="https://github.com/Netflix/eureka/issues/600#issuecomment-207027933">Cloud</a> standards with unexplainable, magic automation and <a href="http://www.christenseninstitute.org/blog/the-demise-of-googles-project-ara-and-modularity-in-computing/">integration</a> can cement <a href="http://www.slideshare.net/myfear/nine-neins-where-java-ee-will-never-take-you?utm_content=bufferc6862">vendor</a> lock-in; Cross-vendor services are still not solved, maybe due to academic REST and DDD being inherently anti-<a href="http://restalk-patterns.org/">evolution</a> with their focus on a <a href="http://www.25hoursaday.com/weblog/2009/09/27/DuctTapeProgrammersAndTheCultureOfComplexityInSoftwareProjects.aspx">Xanadu</a>-ideal, <a href="http://www.michaelnygard.com/blog/2016/07/wittgenstein-and-design/">fixed state</a>. And <a href="http://quoteinvestigator.com/2012/04/28/shorter-letter/">simplicity</a> does never illustrate success, while <a href="https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD898.html">complexity</a> does. Not everyone wants to hear those drawbacks, and few openly make the tradeoffs. Most want to hear the innovative, the disruptive, which sometimes simply excludes the best option for the largest group.</div>
<div>
<br />
In addition, the problem with the typical project organization is that it takes a lot of effort to not disguise debt with another <a href="http://billgyang.blogspot.sg/2009/10/dark-side-of-abstraction.html">layer</a> but actually change the underlying <a href="https://en.wikipedia.org/wiki/Clay_Shirky#Shirky_principle">problem</a> (a post on this is coming up). I always enjoyed this challenge, and I had the pleasure of being on many projects which achieved this, and turned into real, strong products. But kickstarting this "digital transformation" every so often started to nag on my belief in <a href="https://www.innoq.com/en/talks/2016/10/microservices-size-goto-london-2016/">actual progress</a>.<br />
<br />
An unfortunate side effect of the project and <a href="http://www.linuxjournal.com/content/whats-our-next-fight">OSS</a> “<a href="http://www.economist.com/news/science-and-technology/21710792-scientific-publications-are-getting-more-and-more-names-attached-them-why">publishing</a>” treadmill seems communal wisdom is often discouraged in favour of new <a href="https://twitter.com/ruthmalan/status/751775043156185089">marketing</a> slang. Or, like <a href="http://idlewords.com/talks/sase_panel.htm">Maciej</a> puts it: <br />
<blockquote class="tr_bq">
<i>"This is not the first time an enthusiastic group of <a href="http://conversations.e-flux.com/t/what-was-the-nerd-the-myth-of-the-bullied-white-outcast-loner/5376">nerds</a> has decided to treat the rest of the world as a science experiment. Earlier attempts to create a rationalist Utopia failed for interesting reasons, and since we bought those lessons at a great price, it would be a shame not to learn them"</i></blockquote>
<br />
Sometimes it feels through <a href="http://www.tandfonline.com/doi/abs/10.1080/00014788.2016.1222262?scroll=top&needAccess=true&journalCode=rabr20&">lean and measurability</a> the<a href="https://en.wikipedia.org/wiki/Peter_principle"> Peter Principle</a> and pop-culture <a href="http://atom-morgan.github.io/in-defense-of-douglas-crockford">power</a> struggles in the form of Deutungshoheit have eventually arrived in what used to be an idealistic <a href="https://www.wired.com/2010/12/ff_angrynerd_geekculture/">geek world </a>of IT, pushing practitioners from actual consulting into <a href="http://www.vulture.com/2016/09/tyranny-of-art-history-in-contemporary-art.html">meta</a>- and<a href="https://www.thoughtworks.com/insights/blog/it-s-time-discard-digital"> consult-speak</a>. If software indeed eats the world, it seems to prefer junk food. To avoid this trap I took a step back on conference talks and wrote an obscure book. But I am interested in <a href="https://twitter.com/ruthmalan/status/776522408266268672">historical patterns</a>, across disciplines, which is hard to communicate. The nagging feeling stayed. <br />
<br />
So I decided it’s time to move away from a project into a product organization. I wanted to feel out of my comfort zone again, but still contribute to technology. A few of my considerations which company to join, in no specific order, were: <br />
<ul>
<li>It should be first class in building <a href="http://uxmastery.com/formula-delight/">delightful</a> products with end-user impact, where<a href="http://www.wired.com/2014/12/disappearing-business-of-design/"> good design</a> is a strategic asset. Not just <a href="https://en.wikipedia.org/wiki/Doraemon">gadgets</a>. I see <a href="http://www.cjr.org/tow_center_reports/constructive_technology_criticism.php?CJR=#criticism">technology</a> simply as a tool to materialize curiosity. </li>
<li>Many traditional enterprises consider themselves digital companies now and like to use “<a href="http://www.foundingfuel.com/article/technologies-do-not-disrupt-business-models-disrupt/">disruptive</a>” technologies, but I haven’t seen many who actually <a href="https://www.youtube.com/watch?v=yLyALWAp8IM&app=desktop">care</a>. Mostly it's a hedging strategy. I want to work in a real digital enterprise, maybe on <a href="http://ben-evans.com/benedictevans/2016/3/30/chat-bots-conversation-and-ai-as-an-interface">intelligent</a> and intuitive products (e.g. what I often miss in AI and technology as a whole is the capacity to trace the <a href="https://en.wikipedia.org/wiki/Belief%E2%80%93desire%E2%80%93intention_software_model">intent</a> “<a href="http://www.nature.com/news/can-we-open-the-black-box-of-ai-1.20731">why did you do that</a>”?) </li>
<li>It should be first class in technology. I am more of an operational excellence person, so I don't care if it's <a href="http://mcfunley.com/choose-boring-technology">boring</a> technology, as long as there is a clear rationale and it makes a real impact. In short: An <a href="https://techcrunch.com/2015/02/28/beware-the-pretty-people/">engineering</a> mindset. </li>
<li>It should see information and technology as a narrative. Over the years I’ve got more and more interested in <a href="http://systemswe.love/">systems of systems</a>, the <a href="https://en.wikipedia.org/wiki/Lewis_Mumford#Polytechnics_versus_monotechnics?wprov=sfsi1">polytechnic</a>, involving <a href="https://www.weforum.org/agenda/2016/09/jobs-of-future-and-skills-you-need">people</a>, and believe the role of an <a href="https://leanpub.com/software-architecture-for-developers">Architect</a> is to enable strategic, <a href="http://markburgess.org/certainty.html">uncertain</a>, but fair, <a href="http://www.cjr.org/tow_center_reports/constructive_technology_criticism.php">critique</a> and decisions within the team. </li>
<li>It should have a strong and clear vision, not only building products but aligning those products to a long term goal. However, I would prefer that goal to be <a href="http://www.wfeo.org/ethics/">ethical</a>, not to <a href="http://www.salon.com/2013/05/12/jaron_lanier_the_internet_destroyed_the_middle_class/">wipe out the middle class</a>, centralize <a href="http://motherboard.vice.com/read/uber-begins-its-endgame-replacing-humans?utm_source=mbfb">means</a> of production and <a href="http://www.businessinsider.sg/amazons-logistics-re-imagined-by-deutsche-bank-2016-6/?r=US&IR=T#qF4uxiJdhFUDBrLK.97">distribution</a>, or make everyone <a href="http://www.motherjones.com/politics/2016/10/what-donald-trumps-cabinet-might-look-like">dumber</a> (Note: This was written before the US election, there would be much more to say on this in hindsight). </li>
<li>While I believe my strength is to be a good <a href="https://www.thekua.com/atwork/2014/11/the-definition-of-a-tech-lead/">tech lead</a> and <a href="https://medium.com/@ednapiranha/want-to-be-an-engineering-manager-74fb6c69d932#.q5jizpg6w">manager</a>, and I enjoy building teams, I’d prefer less hierarchical organization (ever wondered that <a href="https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/">squad</a> is a military term?), rather a swarm - what I liked about consulting. I’d prefer a strongly influencing, not necessarily managing position and a slightly <a href="https://medium.com/@aarron/7-problems-growing-design-teams-face-5fd94292d405#.dzy50rydn">slower</a>, maybe even less visionary, but <a href="https://posts.philipkd.com/wobbly-tables-and-the-problem-with-futurism-934468d2308#.tcnzlvmzz">more humane</a> company. </li>
<li>It should be global, distributed and <a href="https://hbr.org/2016/07/why-diversity-programs-fail">diverse</a>. Living in many places, working with people from all kinds of backgrounds has enriched my life more than anything else. I have enough startup experience to not believe fast-moving (hierarchical) unicorns necessarily make better corporate citizens. Hence I prefer truly multinational, larger organizations. I like <a href="http://www.wsj.com/articles/the-reality-behind-isaiah-berlins-fox-and-hedgehog-essay-1408144444">foxes, not hedgehogs</a>. </li>
<li>And finally, for now I want to stay in <a href="https://medium.com/studio-d/6-1-glimpses-of-the-future-e3fdb510dcc1#.oaz9jmi88">Asia</a>, with its <a href="https://en.wikipedia.org/wiki/Multiplicity_(philosophy)">multiplicity</a> an ideal model for the ever more complex and <a href="http://www.nature.com/ncomms/2016/160610/ncomms11609/full/ncomms11609.html">uncertain</a> systems in his world, the opposite of the <a href="http://rightbrainbusiness.blogspot.sg/2005/10/deleuze-guattari-and-rhizome.html">power</a> structures I encountered in the past. Someday I might want to leave corporate world, and work on the <a href="http://janchipchase.com/2014/10/sdr-traveller/">grass roots</a>. Asia is the best place to keep this door open. </li>
</ul>
<br />
Google’s <a href="https://www.google.com/about/company/philosophy/">10 things</a> always resonated with me. They maintain a <a href="http://www.bonkersworld.net/organizational-charts/">hive-mind culture</a> based on <a href="https://research.google.com/teams/brain/">research</a> while at least sometimes trying to <a href="http://www.nytimes.com/roomfordebate/2015/07/22/is-silicon-valley-saving-the-world-or-just-making-money">improve</a> the world (yes, I know <a href="https://www.quora.com/Silicon-Valley-Season-2-Does-the-line-I-dont-want-to-live-in-a-world-where-someone-else-is-making-the-world-a-better-place-better-than-we-are-accurately-summarize-the-twisted-sense-of-altruism-in-startup-culture">Silicon Valley</a>). I am continuously impressed which brilliant people work there, and the degree of freedom they have. Let’s see if I can meet a few and learn.</div>
Unknownnoreply@blogger.comtag:blogger.com,1999:blog-6849806902838215926.post-89075591393586117652016-06-05T16:43:00.001+02:002016-06-25T17:04:33.659+02:00Towards a progressive REST API model<blockquote class="tr_bq" style="text-align: right;">
<span style="background-color: white;"><i>"Although REST interaction is two-way, the large-grain data flows of hypermedia interaction can each be processed like a data-flow network, with filter components selectively applied to the data stream in order to transform the content as it passes"</i></span></blockquote>
<div style="text-align: right;">
<span style="background-color: white;">Roy Fielding's <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm#sec_5_1_6">dissertation</a></span></div>
<span style="background-color: white;"><br /></span><span style="background-color: white;"><i>This is the last post of my 5-piece API story "<a href="http://unarchitectedsystems.blogspot.sg/2016/05/you-dont-manage-api-api-manages-you.html">You don't manage the API, the API manages you</a>".</i></span>
In my first real full-time job in a startup, founded around 2004, I had the role of both CIO and CMO. Back then, outsiders were not only laughing at this combination, but investors directly took is an example why the organization would be immature and not able to survive. It did, though, and prospered for a long time. Fast forward 10 years later and <a href="http://www.mckinsey.com/insights/business_technology/getting_the_cmo_and_cio_to_work_as_partners" target="_blank">McKinsey</a>, <a href="http://www.deloittedigital.com/us/blog/the-key-to-cio-and-cmo-collaboration-be-customer-centric" target="_blank">Deloitte</a>, <a href="https://www.accenture.com/us-en/insight-cmo-cio-alignment-digital-summary.aspx" target="_blank">Accenture</a> and <a href="http://blogs.adobe.com/digitaldialogue/events/cmo-cio-relationship/" target="_blank">Adobe</a> are promoting the CIO/CMO model and more, <a href="https://hbr.org/2015/09/design-thinking-comes-of-age">design thinking as culture</a> and <a href="http://www.horsesforsources.com/design-thinking-blueprint_012416">blueprint</a>. And CMO/CFO get closer with <a href="https://hbr.org/2015/06/is-programmatic-advertising-the-future-of-marketing">programmatic content and advertising</a>. From my own experience in startups, but also consulting and service design, I know this model is not only about making the company customer-centric, and <a href="http://www.svpg.com/product-vs-it-mindset">product-centric,</a> but focusing the culture on <a href="http://www.forbes.com/sites/stevedenning/2014/06/17/why-the-worlds-dumbest-idea-is-finally-dying/#394f9f764ef8" target="_blank">most value</a> [<a href="http://martinfowler.com/bliki/OutcomeOriented.html">outcomes</a>] instead of cost savings. It's about having a common goal, and allowing fast feedback for the product or service of the organization. This <a href="https://www.smashingmagazine.com/2013/02/designing-great-feedback-loops/">fast feedback</a> requires fast, agile changes, a lean cycle, a risk culture and a real collaborative <a href="http://www.wired.com/2013/05/accenture-fjord/">service, or product, design</a> - I don't like singularity but it seems that <a href="http://critique-of-pure-reason.com/the-use-and-abuse-of-vegetational-concepts-adam-curtis-on-cybernetics-ecology-and-power/">cybernetics has won</a>.<br />
<br />
What does that mean for our API's? Can we really still revolve around a relatively static* hyperlink model that does not embody this feedback?<br />
<br />
<br />
<span style="font-size: x-small;">*) I am aware of solutions to that, like Gateway API's, Content Negotiation and Redirects but the core issue of no common ontology still exists and DDD alone cannot solve this</span><br />
<br />
<a name='more'></a><br class="Apple-interchange-newline" />
Funnily enough, one of the most hyped silver bullets these days, Microservices, are being advertised as the solution to own <a href="https://www.thinkwithgoogle.com/articles/win-every-micromoment-with-better-mobile-strategy.html" target="_blank">Micro-Moments</a> along a customer journey. With the customer moving not only faster, but at different speeds, and jumping between channels, the organization <a href="http://www.cio.com/article/3020488/cio-role/top-5-actions-cios-should-take-in-2016.html" target="_blank">has to do so too</a>.<br />
<br />
What I like in above's quote from the original REST source is that it describes the REST idea so well, like a DDD flow. Which is a superb concept. That works nicely as long as all dependencies and variants are universally known, and managed, and state can be contained. Nowadays however, such <a href="https://genehughson.wordpress.com/2013/04/08/dependency-management-is-risk-management/">dependencies</a> are seen as too much (in the <a href="https://books.google.com.sg/books?id=ITsWdAAzVYMC&dq=Just+Enough+Software+Architecture&hl=de&source=gbs_navlinks_s">Fairbank's sense</a>) risk, and not lean enough. A nice example is Java 9, <a href="https://www.youtube.com/watch?v=l1s7R85GF1A">which is right now being decomposed</a>, jigsawed, but you can take Microservices and reactive architecture as an example as well. I feel <b>we need a progressive REST</b>, a REST 2.0 that rethinks its tradeoffs and comes up with new primitives for a cybernetic, yet <a href="http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html">resilient</a>, web in constant <a href="http://fluxxor.com/what-is-flux.html">flux</a>.<br />
<br />
<h3>
A progressive micro-state model for REST</h3>
<br />
With <a href="https://blog.prototypr.io/the-conversational-user-interface-cui-of-the-future-the-google-now-example-ef4bcf041f9#.b1hdkutk8">conversational</a> interfaces that <a href="http://www.theatlantic.com/past/docs/unbound/digicult/dc9710.htm">break from</a> <a href="http://time.com/21039/tim-berners-lee-web-proposal-at-25/">the document metaphor</a>, where <a href="https://www.ted.com/talks/steven_johnson_on_the_web_as_a_city#t-665861">the relationships</a> are more emergent, more <a href="https://developers.google.com/google-apps/realtime/overview#how_does_it_work">realtime</a> and <a href="https://api.slack.com/rtm">event-based</a>, more <a href="http://www.tristanharris.com/2016/05/how-technology-hijacks-peoples-minds%E2%80%8A-%E2%80%8Afrom-a-magician-and-googles-design-ethicist/">content curation</a> happens, <a href="https://customers.reuters.com/developer/choosingTheRightAPI.aspx">quality</a> and <a href="http://www.ca.com/us/products/ca-risk-authentication.html">risk</a> are invariants of API's, resilience is key, and <a href="http://techblog.netflix.com/2015/11/global-continuous-delivery-with.html">systems change faster than data</a>, we need to decouple resource representations from their state in the overall system, and rethink the <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm#sec_5_1_3">stateless</a> tradeoff - we need to arrive at <i>Micro-state</i>.<br />
<br />
One solution to this problem could be Event Sourcing. Not seeing Hyperlinks as static references to effectively Actions, but descriptions of actions, like intents in more contemporary data flow models like <a href="http://redux.js.org/docs/basics/DataFlow.html">Redux</a> and <a href="http://www.reactive-streams.org/">Reactive Streams</a> with a server state. This would remove the need for meta formats like HAL etc and most of versioning, because the Link header would only contain be very basic references to directories, accessible via OPTIONS request. Those references would assume the API is semantically declared somewhere outside, and a developer knows how to handle these semantics. New headers could be used to declare context, an idea could be to reuse the "scope" and "state" that <a href="http://openid.net/connect/">OAuth2/OpenID</a> already has. They could reference a state on the server that would not be a resource yet could be queried by GET. URI's would only address entities by UID where there actually is one, while it could be perfectly fine to use GET/HEAD to obtain short-lived representations of state such as paginated lists rather than using fake-UIDs for GET requests. Even better would be that GET would allow request bodies to get around cases where URL's are too long, to brittle, uncacheable or not secure. The only problem with such a state, not a session, would be that it is <i>non-transferrable</i> on the server, but usable across channels, hence PATCH would have to be used to modify it and send <a href="https://twitter.com/danielbryantuk/status/719941886174224385">intents as messages</a> in an Event Sourcing system. It could even be synchronized in a CouchDB style, between multiple applications using a type of request that actually <i>patches</i> and versions cross-domain (in both the DDD and HTTP sense) state.<br />
<br />
While this would break some of the REST principles, it would make integration much simpler and de-risk dependencies. More importantly, it would force API consumers and producers to think about state earlier, and what context information to submit. It would emphasise the need for sensible content-types and clear relationships, rather than meta-formats and machine-to-machine descriptions which only fulfill silver-bullet and AI dreams. It would also remove pointless philosophical discussions about idempotency and replace it with real user requirements of a constantly changing, emerging, system that needs to remain accessible for humans, machines and REST dogmatics.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-7563405974973698412016-06-02T11:29:00.000+02:002016-06-05T17:14:52.446+02:00Caching means Retention and other Web API quirks<blockquote class="tr_bq" style="text-align: right;">
<i>"Fun here becomes related to formal logic and repetition, to the question of where software starts and ends, to mental states, to what operations it can carry out on the world, to the cultures and usages of software, to its building upon itself, to its aesthetics"</i><br />
<a href="http://www.spc.org/fuller/interviews/fun-with-software-a-discussion-with-annet-dekker-and-olga-goriunova/">Olga Goriunova</a></blockquote>
For everyone who the other 3 posts in this series were to philosophical, relax, this is a technical one.<br />
<br />
Thanks <a href="https://www.dgsiegel.net/news">Daniel</a> for suggesting Vinay Sahni's "<a href="http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api">Best Practices for Designing a Pragmatic RESTful API</a>"; I like that he also emphasizes on documentation, what works in the real world and still believes in some principles like links. Also, Pedro's "<a href="http://pmhsfelix.github.io/http-api-specs/">List of HTTP API Specs</a>", thanks to Mike for this tweet. And of course the ubiquitous <a href="http://restcookbook.com/">REST Cookbook</a> and the always good <a href="http://apigee.com/about/resources/ebooks/web-api-design">Apigee guides</a> (I would reference CA Layer 7 here as well but they <a href="http://transform.ca.com/498048-API-Management-Playbook-PS.html?source=apiacademy">hide their guides</a> behind a spamwall), the <a href="http://www.pragmaticapi.com/">pragmatic API blog</a> and <a href="http://blog.steveklabnik.com/posts/2011-07-03-nobody-understands-rest-or-http">Steve's rant</a>.<br />
<br />
As the 2nd last piece in this series, let's have a look into some technical quirks of Web API's.<br />
<br />
<div>
<br />
<a name='more'></a><br />
<br />
<h3>
What lies behind REST's boundary</h3>
</div>
<div>
<br />
Take for instance <a href="https://experiencinginformation.wordpress.com/2014/10/18/resources-on-faceted-navigation-design/">faceted</a> (by the guy who wrote "Mapping Experiences") search, a way more complex way of interacting with a website which I've used in the past. Instead of a navigation, we used a <a href="https://govinsider.asia/smart-gov/why-britain-banned-mobile-apps/">combination of search</a>, filters and context to allow a more fluid experience. I had mentioned pagination in my first post, how complex this is to get it right, especially when there is lots of concurrent refreshes, sorting, filtering, masking and state changes involved. The same goes for <a href="https://blog.intercom.io/principles-bot-design/">bot interfaces</a>, gestures, or more generally <a href="https://spring.io/blog/2016/02/09/reactive-spring">reactive</a> and ubiquitous interfaces. You cannot send the full context, or state of the system, around with every request, you might not even know it or don't want to depend on it. If you look at <a href="http://www.adweek.com/news/technology/programmatic-content-will-bring-human-happiness-if-great-storytelling-behind-it-161778">programmatic content</a>, already dozens of context parameters define an experience. In my opinion, <a href="http://restful-api-design.readthedocs.io/en/latest/methods.html">PATCH is already RPC</a>, and while it is needed, it shows this limitation of the original HTTP spec. Users don't think in CRUD (<i>I am aware this is the wrong metaphor for REST</i>), they don't care if the interface is universal (<i>which is both good and bad and I am aware too easy to critique</i>), and certainly not in <a href="http://restcookbook.com/HTTP%20Methods/idempotency/">idempotency</a>, but our API's need to fulfill <a href="http://tractionsoftware.com/traction/permalink/Blog2443">human needs</a> for the developer too. Yes, there is <a href="http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api#method-override">Method Overriding</a> to push more complex data that are impossible via GET, or data that <a href="https://blog.httpwatch.com/2009/02/20/how-secure-are-query-strings-over-https/">should not be seen in logs and proxies</a>, but it feels dirty and breaks the idea.<br />
<br />
However, REST has it's advantages and a large number of properties that are just generally a good idea, I had mentioned clear domain-driven <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/evaluation.htm">state transitions</a> on otherwise immutable data, like messages, for instance, or content negotiation and the idea of a universal interface. Or caching. I could write a (short) book about that, but <a href="https://developers.google.com/speed/docs/insights/LeverageBrowserCaching?hl=en#recommendations">Google</a> and <a href="https://devcenter.heroku.com/articles/increasing-application-performance-with-http-cache-headers">Heroku</a> did a better job at it. Maybe the only hint I can give you is: <b>Free yourself from seeing caching as buffer for performance</b>. Caching forces you to think about state as 1st order architectural citizen, which is great. First and foremost, Caching is a functional information about data quality and retention. There is even the Warning header to give more functional information about data retention! Sometimes you get requirements like "this data must not be cached" - but you cannot prevent someone from displaying a response for as long as they want. If you have such a requirement, you need <a href="http://foshttpcache.readthedocs.io/en/stable/invalidation-introduction.html">invalidation</a> or <a href="http://www.confluent.io/blog/making-sense-of-stream-processing/">reconciliation</a>, not anti-caching. For whatever data you have, you must define its validity and discuss what that means for the API consumer, and the user experience. If your user experience has different sub-states and data validity rules, you might also need to modularize your code differently and serve it differently.<br />
<br />
Also, ETags can have a nice use as state indicator, irrespective of caching for performance reasons - you can have a distributed cache (<a href="http://redis.io/">Redis</a>, <a href="http://cassandra.apache.org/">Cassandra</a>, <a href="https://hazelcast.com/">Hazelcast</a>) that links UID's for representations to their current resource state. You can even push that further and use ElasticSearch as your primary <i>representation</i> store in a <a href="https://aphyr.com/posts/323-jepsen-elasticsearch-1-5-0">potentially inconsistent</a> CQRS model to be able to query it more flexible, and sync it back to a transactional document store, such as <a href="https://www.cockroachlabs.com/">CockroachDB</a> or <a href="http://couchdb.apache.org/">CouchDB</a> using a solid queue. Or, just use <a href="https://aphyr.com/posts/329-jepsen-rethinkdb-2-1-5">RethinkDB</a> or <a href="https://firebase.google.com/">Firebase</a> if you fancy that kind of magic.<br />
<br />
<h3>
Some good practices from my experience</h3>
<a href="http://www.restapitutorial.com/lessons/httpmethods.html"><br /></a>
<a href="http://www.restapitutorial.com/lessons/httpmethods.html">Much can be said about status codes</a>, there is never enough, but here is a few tricks that I found handy over time:<br />
<ul>
<li>Understand 200 just means OK - it just means the server has understood what you said (It's like a "Yes" in Asia ;) )</li>
<li>Use a <a href="http://restcookbook.com/Resources/asynchroneous-operations/">202 status code and Location header</a> for asynchronous processing - you may even think about chunked responses or ranges (206, see below). The good thing is, the client can send an Expect to chose it's preferred behaviour (very good for occasionally connected applications).</li>
<li>All HTTP responses can have a payload - make use of proper error format e.g. a common JSON error response format (interestingly, <a href="http://json-schema.org/latest/json-schema-validation.html">JSON Schema Validation</a> does not come with one) in order to distinguish between application and infrastructure errors such as 404, and make sure they are included in your content negotiation. Some examples are a <a href="https://tools.ietf.org/html/draft-nottingham-http-problem-03">2013 IETF Draft</a> and <a href="https://github.com/blongden/vnd.error">vnd.error</a> which is HAL-based. But as a consumer, make sure you use some quick checks whether the response is in that format, otherwise you end up parsing 500's with ultra-long stacktraces...</li>
<li>Some error codes seem too special in the beginning but become really important when you think about it, my favourites are 409 (Conflict) and whatever is used by <a href="https://support.cloudflare.com/hc/en-us/sections/200820298-Error-Pages">CloudFlare</a> in their nice machine-readable error format for connection problems</li>
</ul>
<div>
<blockquote class="twitter-tweet" data-lang="de">
<div dir="ltr" lang="en">
HTTP status ranges in a nutshell:<br />
<br />
1xx: hold on<br />
2xx: here you go<br />
3xx: go away<br />
4xx: you fucked up<br />
5xx: I fucked up</div>
— Steve Losh (@stevelosh) <a href="https://twitter.com/stevelosh/status/372740571749572610">28. August 2013</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
</div>
<div>
<br /></div>
I mentioned Method Overriding above as an example of HTTP headers; a few other really helpful headers are:<br />
<ul>
<li>Understand the power of Content Negotation and Encrpytion headers for performance (zipping) and internationalization, not only for REST resource descriptions and data types</li>
<li>Learn the differences between CORS and <a href="https://www.owasp.org/index.php/Content_Security_Policy">CSP</a> - the latter might solve your problem better - and how the <i>Forwarded</i> header can help you moving such information closer to your service (yes, <a href="https://tools.ietf.org/html/rfc7239">RFC 2736</a> standardized it, welcome to the future!)</li>
<li>Understand why the <a href="https://stormpath.com/blog/where-to-store-your-jwts-cookies-vs-html5-web-storage"><i>Authorization</i> header</a> should be used and not others, and never GET, to store tokens such as a JWT Token and why you might not only want to use Cookies (<i>mainly because too many legacy servers link them to session state which requires sticky sessions and is <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/evaluation.htm">not RESTful</a></i>)</li>
<li>The Pinning Headers from <a href="https://tools.ietf.org/html/rfc7469">RFC 7469</a> which enhance your client's privacy (in addition to your proper configuration of <a href="https://www.owasp.org/index.php/Transport_Layer_Protection_Cheat_Sheet">SSL parameters like ciphers</a> and Cookies)</li>
<li>The <a href="https://devcenter.heroku.com/articles/http-request-id"><i>X-Request-ID</i></a> semi-standard which is really helpful as a correlation ID to pass across layers, helpful for debugging and performance monitoring</li>
<li>As mentioned above, the Location header is very helpful for asynchronous processing or streaming responses (chunked data) but should also always be returned from PUT, POST or PATCH (<a href="http://blog.steveklabnik.com/posts/2011-07-03-nobody-understands-rest-or-http">Hypermedia, the good parts</a>). Check <i>Alt-Svc </i>as a cheap way of geographic optimization or legacy integration (e.g. URL versioning, I always thought it's a bit of pity it does not come with a qualifier e.g. for different SLA's or versions)</li>
<li>And yes, even though I despised it before, the Link header and especially the <a href="https://tools.ietf.org/html/rfc7234#section-5.5">Warning</a> header might be helpful (see below), but check first if a <a href="http://otac0n.com/blog/2012/11/21/range-header-i-choose-you.html">Range</a> or asynchronous request might not make more sense (for pagination)</li>
</ul>
Same goes for verbs, and the general flow. I have mentioned hyperlinks are good to optimise for <a href="http://blog.gardeviance.org/2016/05/wardleys-doctrine.html">Flow</a> inside of one context, but not for streams and events across contexts. There is a few ways to change this:<br />
<ul>
<li>If you are not sure what I meant above with REST is not CRUD, <a href="http://restcookbook.com/HTTP%20Methods/put-vs-post/">read again on POST vs. PUT</a></li>
<li>Also, I have mentioned the PATCH method. There has been some discussion around it recently, when it finally became standard with HTTP/2, but the core is to understand that for updating you should still use PUT, but you can use PATCH with <i><a href="http://williamdurand.fr/2014/02/14/please-do-not-patch-like-an-idiot/">instructions</a></i> how to change which can be considered a message/action/intent in the redux/dataflow sense</li>
<li>For me, PATCH is the natural counterpart of <a href="https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events">SSE</a> and Websockets, where they would be a UI change event stream (the problem here is state ordering/reconsiliation, as mentioned above)</li>
<li>I think HEAD requests are wildly underused, especially if you use headers and caching a lot. Take a look and consider using them for instance for eager fetching (better user experience) of potentially cached resources or to get range, link, location and alternative service options (in a 300 return on a high-level resource)</li>
<li>Speaking of, the OPTIONS request is <a href="http://zacstewart.com/2012/04/14/http-options-method.html">even more obscure</a>, but in my opinion even more useful. If you use nice Link headers it might very well describe all hyperlinks that make sense (as you know, I don't think that's many) in order to get rid of formats such <a href="http://jsonapi.org/">JSON API</a> (a.k.a. SOAP over JSON), allowing to properly separate REST constraints from content types</li>
</ul>
And that's it really. What a refreshingly technical post.</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-78862976312245483372016-06-01T17:39:00.003+02:002016-06-05T17:15:03.052+02:00The API and the StreamAs <a href="http://www.patternlanguage.com/leveltwo/ca.htm">Alexander</a> mentions in pattern No. 110:<br />
<blockquote class="tr_bq">
<div style="text-align: right;">
<i>“The position of main entrances controls the layout of the building. It controls movement to and from the building, and all the other decisions about layout flow from this decision. […] the first step in placing the entrances is to consider the mainlines of approach to the site”</i></div>
</blockquote>
<div>
In my last post I have written about API's as portals into your systems, a kind of grey box that gives enough semantics away to understand your system, but abstracts it for the target user group. While there is a lot material about patterns <a href="https://apigee.com/about/cp/api-design-patterns">within</a> API's and when to use API's (in <a href="http://www.enterpriseintegrationpatterns.com/patterns/messaging/MessagingGateway.html">messages</a>/<a href="http://microservices.io/patterns/apigateway.html">services</a>), about the <a href="http://broadcast.oreilly.com/2011/06/the-good-the-bad-the-ugly-of-rest-apis.html">good and the bad</a>, I still miss a holistic picture of systems-of-systems architecture that focuses on the API as <i>the service itself</i>. A behaviour that goes beyond a contract. I hope this will soon evolve in an interesting discussion, given that a few really clever people have <i>literally</i> picked it up:</div>
<div>
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
A reminder that things we have long treasured, can be new -- and exciting -- to someone else!<a href="https://t.co/oRJFFNVjxK">https://t.co/oRJFFNVjxK</a></div>
— VisArch (@ruthmalan) <a href="https://twitter.com/ruthmalan/status/736602096800124928">May 28, 2016</a></blockquote>
<br />
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
<br />
<a name='more'></a><br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
We shall not think of thinking, without imagining ourselves thinking *somewhere*.<a href="https://t.co/UyjVpfwSB3">https://t.co/UyjVpfwSB3</a> <a href="https://t.co/HoppSu65ST">pic.twitter.com/HoppSu65ST</a></div>
— Bret Victor (@worrydream) <a href="https://twitter.com/worrydream/status/720842763244154880">April 15, 2016</a></blockquote>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
<a href="http://emergenturbanism.com/">Emergence</a> becomes a big topic in urbanism as well, as with emergence, patterns of forces become more interesting again. I had written before about <a href="http://unarchitectedsystems.blogspot.sg/2013/03/architecture-after-art.html">Tatami mats</a> and the Japanese structure of the city, how cities grow much more like Microservices nowadays - an observation that <a href="http://markburgess.org/blog_cities.html">Mark Burgess</a> is investigating in now. But for services, doesn’t mean you can’t <a href="http://c2.com/cgi/wiki?DesignByContract">design by contract</a>. Just because you can’t create a <a href="http://blackcoqui.tumblr.com/post/29469203982/propaedeuticist-circular-urbanism">grid</a> on city doesn’t mean you can’t <a href="http://espace.library.curtin.edu.au/R/?func=dbin-jump-full&object_id=20916&local_base=GEN01-ERA02">control</a> it. <a href="http://blog.fusedgrid.ca/2013/03/13/navigation-and-legibility/">Legibility</a> and Indexicality just become more important than order. How a <a href="https://www.infoq.com/articles/consumer-driven-contracts">consumer reads</a> an API becomes more important than how you specify it.<br />
<br />
I've used <a href="https://www.scrumalliance.org/community/articles/2013/august/creating-an-agile-roadmap-using-story-mapping">Story Mapping</a> in the past and am, thanks Henrik for the suggestion, looking forward to read "<a href="http://shop.oreilly.com/product/0636920038870.do">Mapping Experiences</a>", a book that will also include user journeys and ecosystem mapping. In Story Mapping, also the grid, the system of stories is key. I am looking forward to service documentation that <a href="http://apievangelist.com/2014/07/15/an-api-definition-as-the-truth-in-the-api-contract/">includes SLA, Quality, Roadmap etc</a>, maybe in <a href="https://github.com/OAI/OpenAPI-Specification">OpenAPI</a>, and that covers topics such as <a href="http://facebook.github.io/immutable-js/">immutability instead of state transfer</a> and <a href="https://twitter.com/danielbryantuk/status/719941886174224385">message passing</a>. I've made good experiences with <a href="http://camel.apache.org/swagger-java.html">Swagger in Camel </a>(again OSGi), and I believe approaches such as <a href="http://docs.spring.io/spring-restdocs/docs/current/reference/html5/">Spring REST Docs</a> are interesting - I just don't like that Pivotal seems to build it's platform <a href="https://github.com/spring-cloud/spring-cloud-netflix/issues/846">mainly for Netflix</a> and continuously ignores standards such as JAX-RS 2.0.<br />
<br />
How do keep documentation <a href="http://zeroturnaround.com/rebellabs/resilience-is-by-design-by-jonas-boner/">resilient</a> I still struggle with, with the right <a href="http://unarchitectedsystems.blogspot.sg/2015/07/documenting-service-interactions.html">proportions for architecture</a> visualisation - <a href="http://www.codingthearchitecture.com/2016/05/31/agile_software_architecture_documentation.html">Simon Brown does that better anyways</a>, and he has the nicer city travel analogies. But maybe a little more collaborative - like t<a href="https://github.com/aosabook/500lines">he 500 lines project</a>? Or by using more interesting patterns like the way <a href="http://redux.js.org/docs/introduction/ThreePrinciples.html">Redux thinks about data flows</a>, as streams with an intent towards an information change, like a <a href="http://unarchitectedsystems.blogspot.sg/2016/01/the-ephemeral-context.html">reverse CQRS</a>, and see how we could show flows more like they really happen instead of weird concepts such as "<a href="http://camel.apache.org/enterprise-integration-patterns.html">routes</a>" in EIP that seem to be more aimed at restricting flows.</div>
<div>
<br /></div>
<div>
Just like <a href="https://youtu.be/Ex1Cl3k3do8?t=35m42s">Rem Koolhaas once said</a> in a talk about the Metabolists who were the first to see the </div>
<blockquote class="tr_bq">
<i>"city as stream of events instead of agglomeration of form"</i></blockquote>
<div>
<br /></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-84957241740107056452016-05-31T18:05:00.001+02:002016-06-25T17:07:14.979+02:00A calm API uses situational awareness<div class="MsoNormal">
<span lang="EN-US">In “<a href="http://www.nowurbanism.org/">Now Urbanism</a>” Jan 13, 2011 on Informal Urbanism Celine D’Cruz explains:</span></div>
<blockquote class="tr_bq">
<div style="text-align: right;">
<i>“It’s important to look at all these aspects … In 20 years what will the structure look like … what do we have to do to make it simple … it’s not so complex to repair”</i></div>
</blockquote>
This post is part 2 of the series "<a href="http://unarchitectedsystems.blogspot.sg/2016/05/you-dont-manage-api-api-manages-you.html"><b>You don't manage the API, the API manages you</b></a>", which is a story of my experiences with API's over the last years and by no means a guideline for others.<br />
<div class="MsoNormal">
<br />
In my <a href="http://unarchitectedsystems.blogspot.sg/2016/05/you-dont-manage-api-api-manages-you.html">first post</a>, I had argued that the the distributed model of stand-alone API gateways is interesting, because it has some semantics from within the system, but no tools on the market fully support it. While there are some standards, like <a href="https://tools.ietf.org/html/rfc5988">RFC 5988</a> and <a href="http://json-ld.org/">JSON-LD</a>, and some implementations like <a href="http://www.odata.org/">OData</a>, if you like XML/Atom, JSON API (<a href="http://katharsis.io/">Katharsis</a>) or HAL/ALPS (<a href="http://projects.spring.io/spring-data-rest/">Spring Data Cloud</a>), they are very limited and with it limit the use of REST itself.<br />
<span lang="EN-US"><br /></span>
<span lang="EN-US"></span><br />
<a name='more'></a><br />
<h3>
<span lang="EN-US">REST Semantics vs. the distributed Model</span></h3>
<span lang="EN-US"><br /></span>
<span lang="EN-US">You might have noted I tried to avoid the term REST in this series on API's. That's a pity, but I don't want to get into a <a href="http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven">passive-aggressive argument </a>over REST terminologies. I am personally just not interested into the platonic ideal of the semantic web, because I believe there is <a href="http://plato.stanford.edu/entries/wittgenstein/#Lan">no one ideal language</a> and hence such semantics are always just an additional, hard to explain, abstraction. That does not mean I don't like standards like <a href="http://roca-style.org/">ROCA</a> or <a href="http://www.narwhl.com/">NARWHL</a>. I still believe they are valid in many cases, and if you have a <a href="http://time.com/21039/tim-berners-lee-web-proposal-at-25/">document/distributed</a> state kind of application model, e.g. a web shop user interface or the </span><a href="http://www.programmableweb.com/news/how-guardian-approaching-hypermedia-based-api-infrastructure/2015/04/27">Guardian</a>, you should use them, and probably together with some form of <a href="http://alistapart.com/article/interaction-is-an-enhancement">progressive enhancement</a> e.g. isomorphic apps with <a href="https://www.smashingmagazine.com/2015/04/react-to-the-future-with-isomorphic-apps/">React</a> (<a href="https://facebook.github.io/relay/">React/Relay/GraphQl</a> is a <i>document patching framework</i> hence <a href="https://news.ycombinator.com/item?id=7672545">perfect</a> for this use-case). I have even made good experience with a workflow and insurance calculation engine based on <a href="https://apihandyman.io/hypermedia-api-maturity-model-part-i-hypermedia-ness/">hypermedia</a> but all other cases, the good ol' blog and order examples, seem rather random to me in terms of semantics.<br />
<span lang="EN-US"><br /></span>
<span lang="EN-US">Many systems are naturally either more event based or service based than document* based, or simply or too fuzzy, <a href="http://techblog.netflix.com/2013/01/optimizing-netflix-api.html">reactive</a>, <a href="http://devops.com/2016/04/01/multi-speed-versus-bi-modal/">multispeed</a>-evolving or eventually consistent to fix all relationships at the moment of access like a timefreeze. Thus Messaging protocols such as <a href="https://stomp.github.io/">Stomp</a> or RPC protocols such as <a href="https://thrift.apache.org/">Thrift</a> or <a href="http://www.grpc.io/posts/principles">gRPC</a> might just be the better fit, and SPA's or <a href="https://adactio.com/journal/10708">Regressive Web Apps</a> might represent the <a href="http://unarchitectedsystems.blogspot.sg/2016/01/the-ephemeral-context.html">ephemeral</a> state of a system better. I've chosen HTTP-aligned examples here, because the ubiquity of HTTP makes it in my opinion almost unavoidable if you want your services and API to be independent of your infrastructure. I don't see a need to adhere to REST if you want to use HTTP; using a technology in unintended ways is at the core of innovation and hacking. Otherwise we would only have <a href="https://youtu.be/R4SDkBKiqr8?t=14m55s">one telephone per town</a> and no internet.</span><br />
<span lang="EN-US"><br /></span>
<br />
<h3>
<span lang="EN-US">API's only have human users</span></h3>
<span lang="EN-US"><br /></span>
One of the issues with REST is that it places itself somewhere between a human to machine and a machine to machine interface, with hyperlinks suggesting application logic can be universally understood by both. That puts it into an uncanny valley, and so many <a href="https://github.com/OAI/OpenAPI-Specification">competing standards</a> to fill this void out there. The problem is, while an API might be <a href="https://news.ycombinator.com/item?id=11800162">deep-learned</a>, even a machine to machine interface eventually has a human user: The developer. And it is better for the system and its users if the developer understands what the UI does. My apologies, but a nice understanding for an architect is only no. 2 on the priority list of system properties, after the service experience (try <a href="https://apiary.io/">Apiary</a> for that). That's why, personally, I prefer a well-documented RESTish API (say RMM Level 2) that maps to a nice domain model, with "<a href="https://speakerdeck.com/olivergierke/ddd-and-rest-domain-driven-apis-for-the-web">domain events as state transitions</a>" and prefer to rather break REST principles than <a href="http://dontpanic.42.nl/2012/04/rest-and-ddd-incompatible.html">DDD principles</a>.<br />
<span lang="EN-US"><br /></span>
<span lang="EN-US">Web API's are at the top of the onion layers of abstraction of semantics, and as such the hardest to reconcile with REST principles. A good user experience does not modify whole documents, it's as simple as that. You can build a lot of magic, for instance <a href="http://schemaform.io/">JSON Forms</a>, but that only makes your architecture harder to understand for the real users. In the phone example above, if you click the link, James May speaks about </span><a href="https://en.wikipedia.org/wiki/Poka-yoke">Poka-Yoke</a>, the Japanese version of <a href="https://en.wikipedia.org/wiki/KISS_principle">KISS</a>. The ambiguity and dogma of rest makes it harder to use that it should actually be. REST is, thanks to HTTP, especially well suited for the constantly failing systems we build today, it should <a href="http://lawsofsimplicity.com/2007/02/12/the-aesthetics-of-failure/">embrace these failures</a> more.<br />
<br />
<h3>
Calm is the new cool</h3>
<span lang="EN-US">The standard failure of API design is to mix up API simplicity with domain simplicity. Most domains are not simple. </span>The easiest example is versioning. An API and its dependencies should be <a href="https://genehughson.wordpress.com/2013/04/08/dependency-management-is-risk-management/">as stable as possible</a> (I used contract tests, behaviour tests, integration tests, monkey tests, circuit breakers, mocks, patching, polymorphic API's and so on to push this stability as far as possible) for consumers, but at the same time in every modern company the domain model is constantly evolving. <a href="http://semver.org/">Semantic Versioning</a> can be used in API's, but there is simply still no technical standard. Content-Types are too rigid, other Headers too unsafe and unstandardised, but URL's surprisingly easy. I usually go with full backwards compatibility - and rather name an API after a very specific domain context, in the hope this would go away with the business process (e.g. EndOfDayPersonalAccountTransaction2016).<br />
<span lang="EN-US"><br /></span>
<span lang="EN-US">Pagination is frequently used as example for hypermedia, especially because the link relations are indeed <a href="http://www.iana.org/assignments/link-relations/link-relations.xhtml">standardised</a>. However, it's not standardised <a href="http://pilhuhn.blogspot.sg/2013/02/best-practice-for-paging-in-restful-apis.html">how they should be used</a>. What I miss in all the examples is a complete model of pagination. What is a page even? Are those examples clear whether they mean a sliding window or a snippet? Which point in time is the reference? How do you check for updated records? How do you handle filter, or language or channel changes? And how to you enable a UI refresh that feels natural? Nothing of that fits standard semantics.</span><br />
<span lang="EN-US"><br /></span>
<br />
<h3>
<span lang="EN-US">Qualitative Domain Models</span></h3>
<span lang="EN-US"><br /></span>
<span lang="EN-US">In Java world, Objects are still used as the standard data structure, because they have many powerful attributes (pun intended). At the same time, concepts of stateless and immutable systems have flattened their disadvantages. Libraries like <a href="https://github.com/FasterXML/jackson">Jackson</a> have become a de-facto standard of object mapping. Regardless whether you call it REST or not, I always believe in the principle of URI and UID where possible as lowest level of "maturity" of any API. But what is unique in our onion layers of abstraction? Every representation of a domain object can differ slightly depending user context, <a href="http://searchsecurity.techtarget.com/tip/Adaptive-authentication-An-introduction-to-risk-based-authentication">adaptive risk systems</a> for instance can change the quality or amount of data, a license might allow one user slightly more real-time data than another (e.g. market prices), a channel might require certain masking or filtering for regulatory reasons, and the order of fields might change for streaming. We can have UID's for all of those, but that's hard. Really hard.</span><br />
<span lang="EN-US"><br /></span>
<span lang="EN-US">This means there has to be something like a session as omni-channel UI state, mapped for every user, with all contextual settings applied to resource representations. All of that linked to real UID's and versions, and the database, linked to their MVCC. Doesn't that mean we just go back to Lotus Notes style of DB-backed applications, like Firebase, <a href="http://couchdb.apache.org/">CouchDB</a>, <a href="https://www.cockroachlabs.com/">Cockroach</a> or <a href="https://www.ethereum.org/">Ethereum</a> promoting it? Alternatively, some API Management systems come with something like that, a distributed <a href="http://projects.spring.io/spring-session/">Session store</a>. Both are an awkward hack to achieve a semi-time-freeze. Our systems deal with the real world now, there is no time freeze, and no single channel. We cannot just define a fixed set of context, between which we map when we like. We cannot apply a central planning approach anymore, where we dictate what's right and what's wrong for the user.</span><br />
<br />
<span lang="EN-US">In architecture, Constant’s “<a href="http://www.megastructure-reloaded.org/de/constant/">New Babylon</a>” is
based on the concepts of sublime moments, Situations, or what we would nowadays
call Emergence. Constant describes it as:<o:p></o:p></span></div>
<blockquote class="tr_bq">
<i><span lang="EN-US">“I see new Babylon as
a web […] a network covering the whole world”</span></i></blockquote>
<div class="MsoNormal">
<span lang="EN-US">Which sounds a lot like <a href="http://unarchitectedsystems.blogspot.de/2013/03/architecture-after-art.html">Ersilia</a>
or the Internet. If we really want the internet to span the world, to make it a <i><a href="http://www.ubiq.com/hypertext/weiser/acmfuture2endnote.htm">calm technology</a></i>, we need to build APIs that are as simple and flexible and beautiful as the real world. I like the word <i>situations</i> better than context, we have to build <a href="http://blog.gardeviance.org/2016/05/wardleys-doctrine.html">situational awareness</a> into our API's. We need to build situative API's - and they might not be RESTful.</span></div>
<div class="MsoNormal">
<br />
<span style="font-size: x-small;">*) I know REST purists would take the word "document" almost as an insult, and I apologise for this. But the fact that representations are always a fixed artifact and state is passed like a token makes this the best metaphor for me</span></div>
Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6849806902838215926.post-32429754504670830982016-05-30T13:08:00.005+02:002016-06-05T17:15:20.179+02:00You don't manage the API, the API manages you<blockquote class="tr_bq" style="text-align: right;">
<div style="text-align: right;">
<span style="background-color: white; color: #181818; font-family: "georgia" , "times new roman" , serif; font-size: 14px; line-height: 18px; text-align: left;"><i>We live lives that are waveforms constantly changing with time, now positive, now negative. Only at moments of great serenity is it possible to find the pure, the informationless state of signal zero.</i></span></div>
</blockquote>
<div style="text-align: right;">
<a href="https://www.goodreads.com/quotes/tag/electro-mysticism" target="_blank">Gravity's Rainbow</a></div>
<br />
<h3>
Kick-Off for a 5-part series on API's</h3>
Inspired by a good, simple, <a href="https://yahooeng.tumblr.com/post/141211499516/elements-of-api-design-delivering-a-flawless-nfl">API best practices post from Yahoo</a>, I figured it might help to collect all the stuff I've learned in different companies and projects over the last years on API's - anything non-trivial beyond <a href="http://12factor.net/">12 Factor App</a>, <a href="http://apigee.com/about/resources/ebooks/web-api-design">Web API Design</a> and <a href="http://restlet.com/blog/2016/03/23/fresh-from-the-press-market-scan-api-serverless-architecture/">Serverless Architecture</a>. For me, API's are the uniform access point to one or multiple systems. For the sake of this article I exclude API's within the same binary here, so you could argue I speak about stand-alone API's if you want (I don't like the term Web API or Front-End API too much).<br />
<div>
<br />
Stand-Alone API's, and with it <a href="http://martinfowler.com/articles/microservices.html">Microservices</a>, are being promoted as "<a href="https://jaxenter.de/microservices-soa-the-good-parts-768">SOA, the good parts</a>". Here, an API as front end to multiple systems is lean approach to <a href="http://www.infoq.com/articles/apis-soa">SOA with right-sized governance</a>, less canonical data models, and a distributed, scalable, containerised, architecture. Quite simply though, they are just a natural evolution of open ecosystems with ubiquitous, <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm#fig_5_6">uniform HTTP interfaces</a>, propelled by contemporary <a href="https://hbr.org/2014/11/what-airbnb-uber-and-alibaba-have-in-common">service-based business models</a> leveraging <a href="http://googletesting.blogspot.sg/2015/04/just-say-no-to-more-end-to-end-tests.html">duck testing</a> for continuous delivery. A good example of such an API is <a href="https://www.firebase.com/docs/rest/api/">Firebase</a> for instance, a portal to a PaaS.<br />
<div>
<br />
<b>Over the course of this week</b> I will share a few ideas I stumbled across in the last years. Some of them might have worked, others not, on some I might have worked directly, on others not. This is not a tutorial, this is a <a href="http://www.nytimes.com/2016/05/22/opinion/sunday/to-write-software-read-novels.html">story</a>.<br />
<div>
<a name='more'></a><br />
<h3>
My favourite dish is brussels sprouts haloumi or broccoli and tuna pasta or kale and lentil salad - Onion layers of service quality</h3>
<br />
The <a href="http://www.foodbeast.com/news/kale-broccoli-brussels-sprouts-and-cabbage-are-all-from-the-same-family/">brassica</a> plant has multiple layers but adjust them to the environment, forming brussels sprouts, broccoli, kale or <a href="http://www.theguardian.com/lifeandstyle/2015/feb/20/cauliflower-recipes-yotam-ottolenghi">cauliflower</a>. It's the perfect example of evolution, because none of its incarnations is "better", they all fit an environmental condition perfectly. The plant, as a genus, ensures its survival, in the sense of the <a href="https://www.theguardian.com/science/2016/may/29/selfish-gene-40-years-richard-dawkins-do-ideas-stand-up-adam-rutherford">selfish gene</a>. There is no "best fit", no single winner and no ideal state, but there is a l<a href="http://mathworld.wolfram.com/GeneticAlgorithm.html">ocal maximum</a> (see James May's version of <a href="https://youtu.be/ZOu6k7En6jc?t=23m">Chaos vs. Evolution</a>) - some brassica used the onion layer principle and shielded it's core against bacteria by layering boundaries, while others protected the core with heavy armour or simply chose to distribute faster and outpace pest.</div>
<div>
<br />
In evolution, your local context, your environment drives mutation - no wonder Google considers code archeology one of it's key success factors for having a singular, large, <a href="https://www.youtube.com/watch?v=W71BTkUbdqE&feature=youtu.be&t=13m10s">evolving code base</a>. As such, knowledge about API's is really applied knowledge of the world, the web, HTTP, <a href="http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven">Hypermedia/REST</a>, ontologies, simply relationships. Applied to a system of systems with various models of state and truth, a hostile environment and concurrent technological and non-technological processes fighting for resources, with information being the main resource. API's are the core of information exchange, and as such are the genes of our systems.<br />
<br />
<h3>
Part 1: You don't manage the API, the API manages you</h3>
<br />
API's are interfaces and as such naturally often leaky. Because the systems behind them are already leaky representations of the real world. But as API's also need to conform to certain technology requirements, they often become not only leaky, but also muddy. You cannot manage this mud, that's why I don't like the term "API Management" - it oversimplifies the interface, which is always dangerous. Because it's often the API, the contract, that stays, not the systems around it.</div>
<div>
<br /></div>
<div>
Just like <a href="https://adactio.com/journal/8245">Software</a>, API's are always political. Who controls the interface controls the systems. That's why the term "API Management" is so tempting. Because it gives the impression the Management could get the control back - more about this is a <b>follow-up post.</b></div>
<div>
<b><br /></b></div>
<div>
You have to understand the API not as a governance, or management, tool but as a boundary, a business card, a portal into the world of your systems with shared ownership. Currently there is 3 major models of building such an API boundary:</div>
<div>
<div>
<ol>
<li>The gateway model, like Apigee or <a href="https://www.axway.com/de/enterprise-solutions/api-gateway">Axway</a>. Sometimes called "<a href="http://apigee.com/about/blog/cto-musings/api-best-practices-learnings-beyond-silicon-valley">Experience API</a>" or "Web API"</li>
<li>The orchestration model, like <a href="https://www.infoq.com/news/2015/06/redhat-microservices-london">FUSE</a>, Layer 7 or WSO2, basically like an ESB for REST Services, often called a "<a href="http://samnewman.io/patterns/architectural/bff/">Backend for Frontend</a>" (BFF)</li>
<li>The distributed model, like <a href="https://blog.heroku.com/archives/2016/3/2/using_netflix_zuul_to_proxy_your_microservices">Zuul</a> or <a href="https://traefik.io/">Træfɪk</a>, basically dynamic reverse proxies with very little transformation logic but real semantic knowledge of the service topology (as made famous by <a href="http://techblog.netflix.com/2013/01/optimizing-netflix-api.html">Netflix</a>)</li>
</ol>
<h3>
</h3>
<h3>
API's are the Map to your landscape</h3>
<div>
<br />
An API is a flattened version of your service territory. You can stack API's to reduce complexity further, but this simplification always comes at the cost of <a href="https://en.wikipedia.org/wiki/On_Exactitude_in_Science">different semantics</a> that need to be explained. API's are a rabbit hole, they always stay a little bit mysterious just like genes. What I like about Nr. 3, the <i>distributed model</i> from above is that is is a real part of the architectural domain context, hence of its semantics. Strangely, though, I have not seen any of those distributed API gateways fully exposing those semantics - more about this is a <b>follow-up post.</b></div>
<div>
<br /></div>
<div>
Some of the Nr. 1 do have such a UID mapping. They are rather obviously what we often call API Gateway or Edge Server, typically employed for non-functional reasons, e.g. security, throttling, backwards compatibility or caching. All valid use cases but often <a href="https://developer.ibm.com/apiconnect/2016/02/18/announcing-ibm-api-connect-a-complete-api-lifecycle-offering/">misused to stay with a big muddy ball of legacy systems</a> behind it. And Nr. 2 is just as obvious for everyone who comes from an Enterprise Integration Patterns background, it is much heavier than models 1 or 3, might sometimes cause landgrab, but has the advantage of a clear separation of API logic which can be more complex then other business logic. Personally speaking, I tend to favour a mix of the model 2 and 3. I believe <a href="https://blogs.paremus.com/2015/03/microservices-platforms-osgi/">OSGi is a good model for a Microservice</a> API,'s because it achieves both separation of concerns and dynamic relationships even with non-REST components (which is great e.g. for CQRS).</div>
<div>
<br /></div>
<div>
But I also like that, on the Pivotal Cloud, both model 1 and 3 can be used, with <a href="http://finance.yahoo.com/news/apigee-pivotal-team-deliver-comprehensive-130000438.html">Apigee</a> and <a href="http://cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.html">Zuul with Eureka</a> and <a href="https://speakerdeck.com/olivergierke/ddd-and-rest-domain-driven-apis-for-the-web">Spring Hateoas</a> working together as a Domain-Driven Architecture. The only problem with such an approach is that too much non-functional logic becomes part of the service, which becomes slow and non-resilient in itself, especially if <a href="https://www.infoq.com/news/2014/03/rest-at-odds-with-web-apis">not all relationships are RESTful</a> which <a href="http://roy.gbiv.com/untangled/2009/it-is-okay-to-use-post">causes religious discussions only solvable by God himself</a>.</div>
<div>
<h3>
</h3>
<h3>
API's, not services, are your teams</h3>
</div>
<br />
For instance, the (otherwise great) <a href="http://scs-architecture.org/vs-ms.html">Self-Contained-Systems</a> model (that is similar to 1+3) makes the mistake that it prefers a team per service, with the assumption that the domain of a service is a feature - in my experience that leads, thanks to <a href="https://en.wikipedia.org/wiki/Parkinson%27s_law">Parkinson's Law</a> and the <a href="https://en.wikipedia.org/wiki/IKEA_effect">IKEA Effect</a> either to bloated systems or to frustration in the team. Often the domain of a service is not 1:1 a stand-alone API feature, especially across channels or in the absence of channels. I don't have an issue that, in some Microservices models, Web Components are a Microservice <a href="http://microservices.io/patterns/microservices.html">in itself as UI</a> and better than a BFF. Maybe not particularly RESTful, because the semantics internalise representation instead of externalising them, but elegant.</div>
<div>
<br /></div>
<div>
Without too much references to Conway's law it must be clear that the choice of API defines your organization. The mix of model 2 and 3 allows smaller services and logic that is clearer per feature, with a shared ownership of the whole API - but comes at the cost t<a href="http://philcalcado.com/2015/09/08/how_we_ended_up_with_microservices.html">hat teams must understand, and synchronize dependencies</a>, between multiple services. <a href="http://martinfowler.com/bliki/OutcomeOriented.html">Cross-cutting teams</a> aligned to real features, or to API's, tend to understand the platform better, which is great for distributed teams and high numbers of parallel features being developed. But be aware a platform can itself lead to a a <a href="http://www.microservices.com/ben-christensen-do-not-build-a-distributed-monolith">distributed monolith</a> that hinders refactoring.</div>
</div>
</div>
</div>
<div>
<br /></div>
<div>
If you build an API, understand you don't manage the API. At best, if you are lucky, you are managing <i>some</i> of the systems. And you get <i>some</i> consensus on how they can be accessed. This consensus is your API, your information DNA, and once it's distilled it's <a href="http://martinfowler.com/ieeeSoftware/whoNeedsArchitect.pdf">hard to change</a>. The way your teams handle those interfaces is much more important, and over time the API will become a standard that will shape the systems rather than the other way round. Hopefully you have someone who cares for the API by then, call that person architect if you like.</div>
<div>
<br /></div>
<div>
You don't manage the API, the API manages you.</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-1837929256157854812016-01-02T14:25:00.006+01:002016-06-01T17:48:33.804+02:00The ephemeral context<i>Over the holidays, I've read a nice <a href="http://trends.fjordnet.com/?disappearing-apps" target="_blank">outlook by FJORD</a> and an awesome <a href="http://idlewords.com/talks/website_obesity.htm" target="_blank">rant by Maciej</a>. Though the two publications could not be any more different in any sense, they touch on a similar topic:</i><br />
<h3>
The re-emergence of context</h3>
<br />
Back when I was studying mobile application development, it was all about context - changing texts and pictures, <a href="http://www.usability.gov/what-and-why/information-architecture.html" target="_blank">structure information</a>, <a href="http://www.contextbook.com/preface/" target="_blank">optimize information to location and usage</a>, cram information on a small screen, those things. With <a href="http://alistapart.com/article/responsive-web-design" target="_blank">responsive</a>, <a href="http://bradfrost.com/blog/post/atomic-web-design/" target="_blank">atomic</a> and <a href="https://www.google.com/design/spec/material-design/introduction.html" target="_blank">material</a> design, the focus shifted more and more into grid systems and ordering components efficiently, under the assumption that <a href="http://www.lukew.com/ff/entry.asp?1816" target="_blank">screens converge so much </a>that all information is equal.<br />
<br />
While responsive design works beautifully in many situations, the technical difficulties and trade-offs** superseded the initial idea of contextual design. However, with the "<a href="http://trends.fjordnet.com/?disappearing-apps" target="_blank">disappearance of apps</a>", context becomes more important than efficient design. With <a href="https://blog.intercom.io/the-end-of-apps-as-we-know-them/" target="_blank">smarter notifications</a>, <a href="http://www.asymco.com/2015/11/16/glance-a-deep-look-at-apple-watch/" target="_blank">smart watches</a>, the aggregation of information into other streams and more participatory and collaborative software service usage patterns, giving the right information at the right time becomes the major value proposition. Uber and Airbnb (with new Google Maps ads), Amazon (with Echo), Tinder but really anything that <i><a href="http://www.architectural-review.com/8687378.article" target="_blank">shapes an environment into a service</a></i>, like <a href="http://janchipchase.com/2015/04/mobile-home-theory/" target="_blank">IKEA</a>, Google Now (as in <a href="http://www.imdb.com/title/tt1798709/" target="_blank">Her</a>) or Nest, are on the forefront of this movement. They don't respond to context, they create their own, ephemeral, context.<br />
<br />
<h3>
Reverse CQRS</h3>
This means mobility and analytics, in the form of deep learning, <a href="http://www.gartner.com/technology/research/nexus-of-forces/" target="_blank">merge</a> more and more. Mobility brings its knowledge of context into analytics, turning the flow of applications around. This makes event sourcing the new user interaction, separating the query (as in <a href="https://msdn.microsoft.com/en-us/library/jj554200.aspx" target="_blank">CQRS</a>) from the command - the event becomes the query, really, and the command, as in the interaction, is executed on an interesting result.<br />
<br />
From a service design perspective, this means we need to build stronger <a href="https://medium.com/@nicolaerusan/conceptual-debt-is-worse-than-technical-debt-5b65a910fd46#.knddqiyls" target="_blank">mental models</a>, but as concept are <a href="http://www.kurzweilai.net/how-brain-architecture-relates-to-consciousness-and-abstract-thought" target="_blank">harder to reason than to visualize</a>, we need to find a way to explain them better first. We can't just respond to a demand, like opening an app, but must provide different entry points (not just API's) to services, that provide their own feedback loops. Not only, as mentioned above, as <a href="https://developers.google.com/project-tango/" target="_blank">entry points via operating system hooks</a>, like notifications. Rather with completely new, independent behaviour, starting from where we see it today for secure logins on a risk basis (e.g. the new <a href="https://googleonlinesecurity.blogspot.sg/2014/12/are-you-robot-introducing-no-captcha.html" target="_blank">Captcha</a>) or proximity-based suggestions. A service design flow looks very different if there is more inputs than outputs. There is a reason <a href="https://github.com/cdglabs/apparatus" target="_blank">SAP looks into interactive programming</a>, and I hope Tesla's ideas for c<a href="http://news.discovery.com/autos/future-of-transportation/tesla-may-start-up-autonomous-electric-car-sharing-150811.htm" target="_blank">arsharing and using cars for autonomous tasks</a> will soon be reality.<br />
<br />
As usual, I believe communication can solve this - call it patterns, guidelines, or I've written about <a href="https://www.reddit.com/r/AccidentalRenaissance/" target="_blank">wrong</a> <a href="http://unarchitectedsystems.blogspot.com.au/2015/07/documenting-service-interactions.html" target="_blank">proportion</a> recently, but you need to establish loosely coupled "atomic" interactions, or <a href="http://elixir-lang.org/getting-started/mix-otp/agent.html" target="_blank">agents</a>, that can react to "reverse CQRS", like <a href="http://redux.js.org/docs/introduction/Motivation.html">Redux</a> advertises it. The analytics is actually rather straight forward, it could even start with a simple subscription model, but splitting your application flow down will be hard work - especially for all the <i>responsive</i> web apps that used plain old MVC and hard-wired their controllers with the overall flow.<br />
<br />
<span style="font-size: x-small;">*) Disclaimer: I work for Accenture, which is the parent company of FJORD.</span><br />
<span style="font-size: x-small;">**) I have myself just been on a very service/product-driven software project where we had many of those</span>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6849806902838215926.post-88684525800915630692015-08-31T07:03:00.000+02:002015-12-30T07:04:32.337+01:009 Books about Architecture<div class="p1">
<div class="p1">
<span class="s1"><i style="color: #990000; font-family: 'times new roman', serif; font-size: 16px;"><span style="font-family: times;">This post is published in December 2015, when I finally found the time to proof read and check. This post is a snapshot of my thinking back at this time.</span></i></span><br />
<span class="s1"><i style="color: #990000; font-family: 'times new roman', serif; font-size: 16px;"><span style="font-family: times;"><br /></span></i></span>
<span class="s1">For a long time, I've had the reading list on the left panel. However, over time it felt a little bit un-curated. Inspired by a list of recent retweets from <a href="https://twitter.com/jeffsussna" target="_blank">Jeff Sussna</a>, especially a post by <a href="https://twitter.com/sall" target="_blank">Mike Sall</a>, I thought compiling it differently could make sense.</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Let's dive right in - here is my list of 9 (well, actually 12) Architecture Books every interested Software Practitioner should read:</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<h3>
<span class="s1">Basic Architecture</span></h3>
<div>
<span class="s1">These are books that give you an initial understanding of what Architecture is about (hint: something "<a href="https://www.youtube.com/watch?v=DngAZyWMGR0" target="_blank">entirely different</a>") without being to engineering-heavy (if you want that, read </span><a href="http://www.goodreads.com/author/show/39504.Francis_D_K_Ching" target="_blank">Francis D.K. Ching</a>), patronising or historic:<br />
<br />
Matthew <b>Frederick</b>, "<a href="https://mitpress.mit.edu/books/101-things-i-learned-architecture-school" target="_blank">101 Things I Learned In Architecture School</a>". A small, lovely, extremely simple book that gives you a first glimpse of how to think like an architect.</div>
<div>
<div class="p1">
<span class="s1"><br /></span></div>
<span class="s1"></span><br />
<div class="p1">
<span class="s1"><span class="s1">Doug <b>Patt</b>, "<a href="https://mitpress.mit.edu/books/how-architect" target="_blank">How to Architect</a>". An indexical volume of architectural elements, making it an easy reader and introduction to architecture. Mainly because of his great <a href="https://www.youtube.com/user/howtoarchitect" target="_blank">videos</a>, actually.</span></span><br />
<span class="s1"><span class="s1"><br /></span></span></div>
<span class="s1">
</span>
<div class="p1">
<span class="s1"><span class="s1"><a href="http://www.goodreads.com/author/show/5813212.Anthony_di_Mari" target="_blank">Anthony di Mari</a>'s "Conditional Design" and "Operative Design". Because it shows the commonalities of structural elements and patterns in building and software architecture in a simple, visual way (as opposed the Alexander's "<a href="http://www.goodreads.com/book/show/79766.A_Pattern_Language" target="_blank">Pattern Language</a>").</span></span><br />
<span class="s1"><span class="s1"><br /></span>
<span class="s1">As a bonus: Joel Kotkin, "<a href="http://www.goodreads.com/book/show/80854.The_City" target="_blank">The City</a>". Not strictly architecture, as it is about urbanism, but a great, short, intense history of human habitat, before you read <a href="https://en.wikipedia.org/wiki/Jane_Jacobs" target="_blank">Jane Jacobs</a>.</span></span><br />
<span class="s1"><span class="s1"><br /></span></span></div>
<span class="s1">
</span></div>
<h3>
<span class="s1">Elementary contemporary texts</span></h3>
<div>
These are texts that defined contemporary architecture, giving you a feeling of the philosophic foundations of "living and dwelling" - something that is often ignored by engineers talking about architecture.</div>
<div>
<br /></div>
<div>
<div class="p1">
<span class="s1">Robert <b>Venturi</b>, "<a href="http://www.goodreads.com/book/show/207129.Complexity_and_Contradiction_in_Architecture?from_search=true&search_version=service" target="_blank">Complexity And Contradiction In Architecture</a>". Not an easy read, but one of the most defining ones of contemporary architecture. A manifesto against dogma and black and white thinking, introducing the hybrid, notably one of the most often used concepts in IT (<i>often read as counter-example of </i></span><i>Le Corbusier's "<a href="http://www.goodreads.com/book/show/70134.Towards_a_New_Architecture" target="_blank">Towards a New Architecture</a>" and only followed by Bjarke Ingels "<a href="http://www.taschen.com/pages/de/catalogue/architecture/all/18509/facts.yes_is_more_ein_archicomic_zur_evolution_der_architektur.htm" target="_blank">Yes is More</a>"</i>).<br />
<span class="s1"><br /></span></div>
<div class="p1">
<span class="s1">Ram <b>Koolhaas</b>, "<a href="http://www.monacellipress.com/book/?isbn=9781885254009" target="_blank">Delirious New York</a>". Because it took complexity to the next level, added emergence, and cross-pollination between society and building. Read it as Conway's Law and be astounded. No idea why, after that, it took Architecture so long until <a href="https://www.ted.com/talks/marc_kushner_why_the_buildings_of_the_future_will_be_shaped_by_you" target="_blank">Marc Kushner</a> introduced feedback to it.</span><br />
<span class="s1"><br /></span>
<span class="s1">Kate <b>Nesbitt</b>, "<a href="https://www.papress.com/html/book.details.page.tpl?isbn=9781568980539" target="_blank">Theorizing a New Agenda for Architecture</a>". An extremely well curated collection of elementary texts that stands out not only as a reader, but for bringing all of those thoughts into a structure that, at least to me personally, was never before so clear.</span></div>
</div>
<div>
<br />
As a bonus: Fred Brooks' "<a href="http://www.wired.com/2010/07/ff_fred_brooks/" target="_blank">Design of Design</a>", because it is a software book dealing with architectural design.</div>
<h3>
</h3>
<h3>
Tertiary Literature</h3>
<div>
Those texts are not directly about architecture, but about the motivations of architecture, it's impact on society and the public.</div>
<div>
<br /></div>
<div>
Adam <b>Sharr</b>, "<a href="https://www.routledge.com/products/9780415415170" target="_blank">Heidegger for Architects</a>". Actually the whole "Thinkers for Architects" series is pretty good, and this is just one example. Architects like, and learn, to think from the perspective of other disciplines. Might be a good example for IT.</div>
<div>
<br /></div>
<div class="p1">
Stewart <b>Brand</b>, "<a href="http://www.goodreads.com/book/show/38310.How_Buildings_Learn" target="_blank">How buildings learn</a>". Brand is a classic, working on the edge of design thinking/service design since the "Whole Earth Catalog". This is a great introduction into the evolution of buildings and hence an answer to the question: <a href="http://c2.com/cgi/wiki?CanAnArchitectureEmerge" target="_blank">Can architecture emerge?</a><br />
<span class="s1"><br /></span></div>
<div class="p1">
Geoff <b>Manaugh</b>, "<a href="http://www.goodreads.com/book/show/5962048-the-bldgblog-book" target="_blank">The BLDGBLOG Book</a>". Imho the single best <a href="http://bldgblog.blogspot.sg/" target="_blank">blog</a> about architecture in the interweb, probably the only real alternative to an <a href="http://www.aaschool.ac.uk/PUBLIC/AAPUBLICATIONS/AAFiles.php" target="_blank">AA files</a> subscription. The book captures the full spectrum of cutting edge contemporary ideas into one volume.</div>
<div class="p1">
<br /></div>
<div class="p1">
<span class="s1">As a bonus: Jo Steffens<b>'</b>, "<a href="http://yalepress.yale.edu/book.asp?isbn=9780300158939" target="_blank">Unpacking My Library</a>: Architects and Their Books" is a nice tabletop book listing a wide range of architects and their favourite book inspirations - </span>Thomas Pynchon's "<a href="http://www.goodreads.com/book/show/415.Gravity_s_Rainbow" target="_blank">Gravity’s Rainbow</a>" is named surprisingly often.</div>
</div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-44193024248924025832015-07-26T13:06:00.001+02:002015-12-30T07:02:33.693+01:00Documenting Service Interactions<i>This post is published in December 2015 after I re-read much of <a href="http://www.codingthearchitecture.com/2015/11/25/software_architecture_diagrams_should_be_maps_of_your_source_code.html" target="_blank">Simon Brow's </a><a href="http://www.codingthearchitecture.com/2015/11/25/software_architecture_diagrams_should_be_maps_of_your_source_code.html" target="_blank">stuff</a> again and figured that, with <a href="https://github.com/structurizr/java/blob/master/docs/extracting-components.md" target="_blank">Structurizr extraction</a>, he is so much on the right way I can stop pondering about that topic - this post is a snapshot of my thinking back in July, which I never really finished due to a unconvincing conclusion.<i></i></i><br />
<br />
I’ve actually read <i>"<a href="http://www.amazon.com/Documenting-Software-Architectures-Beyond-Edition/dp/0321552687/">Documenting Software Architecture</a></i>". ATAM, V&B, DITA and ALM at least ring a bell in my memory. I’ve attended Jazz/RTC, Sparx EA and UML trainings. I can safely say that, in the last 5 years there has barely been a work day I did not use the <a href="https://www.atlassian.com/gartner">Atlassian</a> toolchain. Yet, to me it seems there is no truly elegant architecture documentation, let alone knowledge retention, techniques out there in custom software development land.<br />
<h3>
So why is architecture documentation still so bad?</h3>
<div>
<div>
<div>
You have the integrated toolchains on one hand, the legacy of <a href="http://pointersgonewild.com/2015/08/20/what-killed-smalltalk/">Smalltalk</a>, <a href="https://www.reddit.com/r/programming/comments/60uau/paul_graham_hates_on_lisp_machines_common_lisp/">Lisp Machines</a>, Jazz/RTC/Eclipse/Mylyn and Visual Studio. On the other hand the self-documenting-code and markup world, the legacy of Cobol, like Python, TDD/BDD, Javadoc, <a href="http://swagger.io/">Swagger</a> and so on. There are a few attempts to bring the two together, <a href="http://www.eclipse.org/Xtext/">XText</a>, <a href="http://lighttable.com/">Lighttable</a>, <a href="http://structure101.com/" target="_blank">Structure101</a>, and GitHub for that matter, and a few rather academic visual/structural coding systems like <a href="http://c2.com/cgi/wiki?DataflowProgramming" target="_blank">Dataflow</a> (e.g. <a href="http://vvvv.org/">vvvv</a>). Human-understandable documentation, though, still feels like a 2nd class citizen in any larger scale software development ecosystem I am aware of.
</div>
</div>
<div>
<div>
<br /></div>
</div>
<div>
<div>
The reasons are pretty clear: most documentation systems are either tied to the hip of their very language ecosystem, often as afterthought, and hence not being able to fulfil <a href="http://deanwampler.github.io/polyglotprogramming/">Polyglot Programming</a>. Or, they have been invented to enforce a given top-down structure, either for waterfall development or as operational support, which is by definition not agile and breaks <a href="http://blogs.atlassian.com/2014/02/code-issue-traceability-subversion-perforce/">traceability</a>. Both of these reasons make it tedious to work with documentation. It breaks the flow of feature development, and forces everyone to <a href="https://en.wikipedia.org/wiki/SECI_model_of_knowledge_dimensions">externalise</a> knowledge into the least common denominator. Hence people named “Architects” or “Business Analysts” still create static documents in more or less nice <a href="http://www.arc42.de/template/index.html">templates</a> and <a href="http://structure101.com/" target="_blank">models</a>, which are outdated the moment they are published, because "<a href="http://blog.codinghorror.com/learn-to-read-the-source-luke/" target="_blank">the code is the ultimate truth</a>" (unless you do MDD).
</div>
</div>
<div>
<div>
That’s a big problem in a digital world. Because software doesn’t just copy real-world business processes, and captures them in a box, anymore. It creates new processes, flows unseen before, that are <a href="http://blogs.wandisco.com/2013/09/05/why-we-dont-build-software-like-we-build-houses/">constantly modified</a> and <a href="http://tomdale.net/2013/05/evergreen-browsers/" target="_blank">updated</a> through feedback, until no human understands them anymore. We turn our software into deep learning, minus the human understanding.
</div>
</div>
</div>
<br />
<h3>
Documenting Service Interactions </h3>
At Dockercon, <a href="http://blog.docker.com/2014/12/dockercon-europe-keynote-state-of-the-art-in-microservices-by-adrian-cockcroft-battery-ventures/">Adrian Cockcroft </a>gave a nice talk about Microservices and how to document the architectural implications. It sounded a little bit like <a href="http://ieeexplore.ieee.org/xpl/articleDetails.jsp?reload=true&arnumber=6928966">Service/Process Mining</a>, a backwards-looking approach to architecture, more feature/<a href="http://www.gamasutra.com/blogs/ClintonKeith/20111201/90720/Why_We_Should_Stop_Saying_Vertical_Slices.php" target="_blank">vertical slice</a>/<a href="http://highscalability.com/blog/2012/5/9/cell-architectures.html" target="_blank">cell</a> rather than layer-based. As such, for me it would fall in the markup bracket from above, a fancier version of <a href="http://tech.domain.com.au/2014/09/monitoring-documenting-microservices/">Developer Portals</a> in API Management. What I still miss, though, is <a href="https://structure101.com/">structure</a>, the levels of abstraction. Not in the UML sense of <a href="http://www.oose.de/metamodellUML/" target="_blank">meta levels</a>, but in the sense of a lean take on <a href="http://pubs.opengroup.org/architecture/togaf8-doc/arch/chap31.html" target="_blank">views</a>. Without structure, I hate to say it, static documents will never die, and code and architectural documentation will stay detached.<br />
<br />
<div>
<div>
In order to reduce the overhead and eventual detach between code and documentation, we can look at non-architectural documentation, and see how the problem is solved here:
</div>
</div>
<ul>
<li>Interface and developer documentation, how to use a service or library, can, and should, easily be generated out of code itself (<i>Clean Code</i> meets <i>Convention over Configuration</i>), and versioned directly with it.
</li>
</ul>
<ul>
<li>Traceability to stories can be linked easily, e.g. per check-in. In a post-GitHub world, we must assume this is standard practice.
</li>
</ul>
<ul>
<li>Tests, as behaviour documentation, what is called a <a href="http://www.joelonsoftware.com/articles/fog0000000035.html" target="_blank">specification</a> in BDD, should be versioned and documented with the code, that means aligned to stories and features. Personally speaking I don't think a test can ever cover a real <i>requirement</i> (see below), but it can to certain degree specify a behaviour.
</li>
</ul>
<ul>
<li>Requirements traceability, i.e. feature documentation, is a tricky one, because it typically is orthogonal to stories. This is a vast, separate topic, think <a href="https://en.wikipedia.org/wiki/Model-driven_architecture">MDA</a>, <a href="http://www-03.ibm.com/software/products/en/ratidoorng">DOORS</a> and <a href="http://www.sophist.de/en/our-topics/requirements/requirements-engineering-en/">Sophist</a>.
</li>
</ul>
<ul>
<li>End-User documentation should be driven by feature documentation, and hence falls in the same area.
</li>
</ul>
<ul>
<li>Integration architecture and dependencies are a bit harder, typically <a href="http://www.enterpriseintegrationpatterns.com/" target="_blank">EIP</a> e.g. in <a href="http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/ab00030_.htm?lang=de" target="_blank">IBM Integration Toolkit</a> are used. A bit more lightweight, we could use Adrian’s approach from above. But right now I am not aware of an integrated, structured way to feed back monitoring information into developer documentation (like <a href="https://github.com/Netflix/Hystrix/wiki/Dashboard" target="_blank">Hystrix</a> for instance).
</li>
</ul>
<ul>
<li>Operations and maintenance documentation is another tougher step. Even if monitoring on service level is achieved, the impact of infrastructure or database issues is rarely documented and formalised into <a href="http://www.developer.com/tech/article.php/3637441/The-Project-Postmortem-An-Essential-Tool-for-the-Savvy-Developer.htm" target="_blank">post mortems</a> that can be simulated. DevOps is a cultural solution, but requires very experienced engineers with tacit knowledge.
</li>
</ul>
<ul>
<li><a href="https://www.ibm.com/developerworks/mydeveloperworks/blogs/requirementsmanagement/entry/the_importance_of_defining_a_requirements_management_information_architecture2?lang=en">Information</a> architecture would be a next step, a third dimension to story and requirement, the data view. This is, today, the weakest link.
</li>
</ul>
<h3>
Aesthetics of Architecture</h3>
<br />
<div>
When documenting architecture, one dives into the <i>aesthetics</i> of code - why something in the world was perceived in a way it deemed necessary to break into into parts of a machine. Which <i>immaterial</i> (as in non-functional) values shaped the design of this machine over time. And why those values were agreed in the first place by the group of authors. Hence, architecture documentation has to become a part of the <i>beauty</i> of code.
</div>
<div>
<br />
In building architecture, plans have evolved from models and experiments, towards guidelines and procedures. I have mentioned <a href="http://unarchitectedsystems.blogspot.com.au/2013/02/an-architecture-of-predators-and.html" target="_blank">Alberti earlier</a>, also in the <a href="http://www.goodreads.com/book/show/22118537-baukunst-f-r-softwarearchitekten" target="_blank">first chapter of my book</a>, the inventor of the modern plan as original artwork. In recent years, building architecture has gone back to a much more integrated practise, with plans showing <a href="https://en.wikipedia.org/wiki/Architectural_design_values" target="_blank">values and ideas</a> rather than <a href="https://en.wikipedia.org/wiki/Precedence_diagram_method" target="_blank">processes</a> and <a href="https://en.wikipedia.org/wiki/Conceptual_architecture" target="_blank">concepts</a>. This style has been made popular by <a href="http://www.archdaily.com/5918/house-atelier-atelier-bow-wow/">Atelier Bow Wow</a>. It enriches architectural section drawings, pushing them almost in the direction of perspective competition drawings*.
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container">
<tbody>
<tr>
<td><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhy0QHZ_9f6F-hcbY6qrawX6gnDtWNPKC6HkNeWcUA3ghi3HyM0VFPMNocKElZ9v8b8_0BCI7aOlQe5WKp0o5Zke-ttpeqUGIRklF1QefO3QPhwvhiskGdjufD40noK8AvdvyWqRehWVA7Y/s1600/atelier-bow-wow-graphic-anatomy.jpg" imageanchor="1"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhy0QHZ_9f6F-hcbY6qrawX6gnDtWNPKC6HkNeWcUA3ghi3HyM0VFPMNocKElZ9v8b8_0BCI7aOlQe5WKp0o5Zke-ttpeqUGIRklF1QefO3QPhwvhiskGdjufD40noK8AvdvyWqRehWVA7Y/s1600/atelier-bow-wow-graphic-anatomy.jpg" width="320" /></a>
</td>
</tr>
<tr>
<td class="tr-caption">Detail from "<a href="http://www.toto.co.jp/publishing/detail/A0340.htm" target="_blank">Graphic Anatomy</a>", Atelier Bow-Wow 2007, Toto Publishers
</td>
</tr>
</tbody>
</table>
</div>
<h3>
The myth of the right proportion</h3>
Recently, I've been to the Venice Biennale. In one of the exhibition catalogues, <a href="http://www.axel-vervoordt.com/en/inside/foundation/exhibitions/proportio" target="_blank">Pro Portio</a>, this remarkable passage about the usage of proportions caught my eye:<br />
<blockquote class="tr_bq">
<i>proportion is [...] </i><i>investigation of how elements and patterns are connected and interconnected across disciplines [...]</i><i> universal proportions guide our understanding of creation and the dynamic dance between order and chaos</i></blockquote>
The art of connecting the different views and perspectives of an architecture, understanding the values that make up our perception of a "good architecture", that's proportion. It's putting the elements of the machine into the right relation to each other. It's the grid in responsive design, the harmony of Pattern Languages, the Fibonacci sequence in planning, Amdahl's law in scalability and, after all, finding the right proportion between documentation and code.<br />
<br />
After Venice, we took the chance and drove up to a monument of proportion: Palladio's Villa "La Rotunda", which Colin Rowe called "the ideal villa". It is an example of perfect harmony, as Rowe writes in <i><a href="http://www.architectural-review.com/essays/1947-march-the-mathematics-of-the-ideal-villa-palladio-and-le-corbusier-compared-by-colin-rowe/8604100.article" target="_blank">Architectural Review</a></i>:<br />
<blockquote class="tr_bq">
<i>It was not in fact suggested that architectural proportions derived from musical harmonies, but rather that the laws of proportion were established mathematically and universally diffused. The Platonic and Pythagorean universe was compounded of the simpler relationships of numbers, and such a world was formed within the triangle made by the square and cube of the numbers 1, 2, 3</i></blockquote>
<div class="separator">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzTGZV-CKZeFlUDsrzeB7xb1kFfMQ10IqfZLAq71ZxuuybxFeTfR-_7fB3f5-4zV6V2LaPOhWMwi28cV4F5hHWWwjZvuApq7kcRtosx-LLpk-tYjcVmB9Jzs8fJQyVbVVKLhck5hgAla4u/s1600/IMG_3294.jpg" imageanchor="1"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzTGZV-CKZeFlUDsrzeB7xb1kFfMQ10IqfZLAq71ZxuuybxFeTfR-_7fB3f5-4zV6V2LaPOhWMwi28cV4F5hHWWwjZvuApq7kcRtosx-LLpk-tYjcVmB9Jzs8fJQyVbVVKLhck5hgAla4u/s320/IMG_3294.jpg" width="320" /></a></div>
<div>
<br /></div>
<div>
While it stands as an ideal of renaissance proportion, there is an issue: Inside it's absolutely hideous. It's a monument that proves proportions, or patterns, or a grid, can never be the sole source of beauty.</div>
<div>
<br /></div>
<div>
Which might be the reason taking pictures inside is not allowed. For us in software architecture it means while we need to follow a plan, and maybe even make a plan, we need to use proportions as a tool to understand why we made decisions to have relationships established. A 3-tier architecture is not better because it adheres to Fibonacci's sequence or the triangle made by the square. Gregor Hohpe has <a href="http://www.enterpriseintegrationpatterns.com/ramblings/85_sameoldarchitecture.html" target="_blank">called</a> this "paying more attention to the lines than to the boxes", or Stefan Tilkov the "Cross-System <a href="http://blog.matthewskelton.net/2012/03/20/breaking-the-monolith-by-stefan-tilkov-at-qconlondon-2012/" target="_blank">properties</a>" to care for.</div>
<div>
<br /></div>
<div>
In a wonderful talk, <a href="http://videlalvaro.github.io/2015/02/programming-myths.html">Alvaro Videla</a> hunts down some common myths (a little bit in <a href="http://worrydream.com/TenBrighterIdeas/">Bret style</a>). He argues the reason why we still believe them is because they "<i>are easy to repeat and to follow, than to actually learning the whole methodology</i>". Grids, patterns, number laws and thumb rules are myths, as one easily takes ratios for silver bullets instead of guidelines. An example of a myth from the talk is "<a href="http://uxmyths.com/post/931925744/myth-23-choices-should-always-be-limited-to-seven">Miller's Law</a>", the thumb rule of number of elements a human can put in context. Over time, humans have adapted, and the rules need to be changed. It's these changes, and the rules, which is the most important part of documentation. The tradeoffs and their interaction, the timeline, and the change in context and understanding.<br />
<br />
I have written about Choice Architecture <a href="http://unarchitectedsystems.blogspot.com/2012/08/structures-can-become-shackles.html" target="_blank">before</a>. I firmly believe an Architect is a gardener, and as such she needs to work with the seasons, with time, with taste and the weather, to allow emergence (or evolution if you want) of beauty of the plot. Yet it's true, <a href="http://en.wikipedia.org/wiki/The_Paradox_of_Choice">choice</a> needs to be limited, and we require <a href="https://seeingcomplexity.wordpress.com/2011/02/09/the-brain-as-a-pattern-recognition-machine/">categories</a> and guidelines to make decisions. Taking those decisions, and being able to explain them to others, so they can make them on their plot, is the art of our profession.<br />
<br />
<span style="font-size: x-small;">*) Architectural drawing style is a wide field, see e.g. <a href="http://books.google.com.sg/books/about/Architectural_drawing_and_draughtsmen.html?id=nt9LAAAAMAAJ&redir_esc=y" target="_blank">Blomfield</a></span></div>
Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-6849806902838215926.post-88409039809032581902014-10-19T11:29:00.001+02:002014-10-19T12:59:16.763+02:00Denying State<div>
Recently I attended a great talk about <a href="http://lambda-architecture.net/" target="_blank">Lamba Architecture</a> by <a href="http://www.infoq.com/presentations/lambda-arch-spring-xd" target="_blank">Carlos</a>. I particularly liked how information is underlying a tight control of accuracy, a concept that systems like <a href="http://antirez.com/news/75" target="_blank">HyperLogLog</a>, <a href="http://research.google.com/pubs/pub38125.html" target="_blank">F1</a> or Storm with <a href="https://github.com/twitter/algebird" target="_blank">Algebird</a> have since some time. In my <a href="http://jaxenter.de/artikel/Anwendungsserver-Eine-Zukunft-in-Filmszenen" target="_blank">JAX 2012 Keynote</a> I stated that Eventual Consistency will move into Hyper-Consistency that focuses more on perception than technical isolation. I could very well imagine Lamba Architecture being combined with <a href="http://javascript.hew.io/kaba-cceac/angular-cqrs" target="_blank">CQRS</a>, where some parts of the UI are "noisier"(a bit like VoIP scatter, coming from a <a href="https://www.rabbitmq.com/tutorials/tutorial-two-python.html" target="_blank">queues and workers</a> implementation) than others (maybe implemented using a functional paradigm). In functional programming as well as in event-driven systems, state is mainly neglected. But sometimes state is necessary, because it occurs in the real world, and addressing it in an immutable manner, just storing all information, can become <a href="http://www.xaprb.com/blog/2013/12/28/immutability-mvcc-and-garbage-collection/" target="_blank">costly, slow and inaccurate</a> and insecure. By encapsulating functional logic and immutability where it makes sense, pairing it with a more state-driven, fuzzy, approach an architecture can scale more heterogeneously. The functional parts could be RESTful, and the fuzzy ones streams.</div>
<div>
<br /></div>
But the more complex these ideas become, the more they seem to be used as an excuse for random technology choices - especially in <a href="http://www.infoq.com/news/2014/10/ser-lamport-interview" target="_blank">multi-paradigm solutions</a>, e.g. ones based on JavaScript. They are getting used by people I refer to as Hipster Coders. Although the term hipster has become non-descriptive and <a href="http://nymag.com/news/features/69129/" target="_blank">contradictory</a>, there is a core to it: You can call it quality or design but in the end it's simple: <a href="http://www.scientificamerican.com/article/what-me-care/" target="_blank">Not caring</a>. Caring is the opposite of cool, it's <a href="http://adamsilver.github.io/articles/the-boring-front-end-developer/" target="_blank">boring</a>. Hipster coders don't actually care about appropriate technology, the business context or the user. They care about their personal fun. The hipsters were just the last step in a long history of defaming care. They took away the rebellion. From Hip Hop, Rock, last but not least the Geek and Nerd world - hollowing out every possible idea until it could be mass-customized into an ironic <a href="http://www.newyorker.com/magazine/2014/09/15/naysayers" target="_blank">pop-culture</a> t-shirt. Hipsters celebrate obsolescence. Their hedonic need to be <a href="http://monumenttotransformation.org/atlas-of-transformation/html/o/off-modern/off-modern-svetlana-boym.html" target="_blank">"<i>on</i> the cutting edge" (Boym)</a> rather than <i>beyond</i> the edge is meagrely covering the capitalistic rationale behind it: The <a href="http://desiremachinecollective.in/WORKS/TEXT/sublime.htm" target="_blank">perpetual, infinite, reformulation of techno-scientific merchandise (Lyotard)</a>. Hipster coders use technology like <a href="http://www.wmagazine.com/culture/art-and-design/2008/10/helmut_lang/" target="_blank">fashion uses materials</a>, as a status symbol with an arbitrary defined lifespan.<br />
<br />
A good example is <a href="http://microservices.io/patterns/microservices.html" target="_blank">Microservices</a>, an architecture pattern that gained traction as fast as a cat meme. A mix between SOA, DevOps, REST and <a href="http://highscalability.com/blog/2012/5/9/cell-architectures.html" target="_blank">Cell Architecture</a>, Microservices build mainly on the availability of automated virtualization and out-of-the-box distributed infrastructure. They are a worthwhile answer to heavy infrastructure blocks that come with corporate processes designed to kill any form of agile solutioning. But they come with the cost of "<a href="http://capgemini.github.io/architecture/microservices-reality-check/" target="_blank">Integration at every level</a>" - a technical debt in the long term a Hipster Coder could not care less about. As mobile and social require distributed architectures, this debt is easily accepted in favour of time to market. But once the systems stabilise, operations cost will rise. I agree, versioned services running in their own container make a lot of sense. As does decoupling code from nonsense anachronistic operations onboarding processes. For some components, though, a shared infrastructure just makes total sense. Forcing everyone into a <a href="http://www.infoq.com/news/2014/10/thompson-reactive-manifesto-2" target="_blank">directed messaging model</a> will eventually lead to an uncontrollable network of dependencies. Once the append-only databases start stalling debugging will be hard. Microservices should be used to get rid of noise in the development cycle, not so isolate your fancy pants from the context of the world.<br />
<br />
Dear Hipsters, maybe you would be better off with a bit of <a href="http://www.nytimes.com/2014/04/03/fashion/normcore-fashion-movement-or-massive-in-joke.html?_r=0" target="_blank">Normcore</a>.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-20121553569375313752014-08-03T16:57:00.003+02:002014-08-03T17:12:42.524+02:00Fog ComputingRecently Cisco is promoting a new term, <a href="http://www.cisco.com/web/about/ac50/ac207/crc_new/university/RFP/rfp13078.html" target="_blank"><i>Fog Computing</i></a>. It's the application of cloud (IaaS) terms "to the edge of the network", incorporating ideas such as CDN, SDN, IoT and BaaS.<br />
<br />
In building architecture, this convergence between what they call public and private space is much discussed and a main feature of contemporary buildings, for instance the Blur Building which has fog instead of a boundary, or edge for the sake of the argument:<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><span style="margin-left: auto; margin-right: auto;"><a href="http://www.designboom.com/eng/funclub/dillerscofidio.html" target="_blank"><img alt="Diller & Scofidio Blur Building" border="0" src="http://www.designboom.com/eng/funclub/dillerscofido/1.jpg" title="Diller & Scofidio Blur Building" /></a></span></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="http://www.designboom.com/eng/funclub/dillerscofidio.html" target="_blank">(c) Diller & Scofidio at Designboom</a></td></tr>
</tbody></table>
What began as the discussion about buildings that have a flexible interior but a very clear, black box, exterior in <i><a href="http://mitpress.mit.edu/books/learning-las-vegas" target="_blank">Learning from Las Vegas</a></i>, has moved towards a view of information flows alongside people flows in cities (see <a href="http://www.archplus.net/home/archiv/artikel/46,3763,1,0.html" target="_blank">Trüby</a>, <a href="http://remixedcity.wordpress.com/2013/09/28/after-lynch-the-city-as-information-space/" target="_blank">Lynch</a> or the <a href="http://www.caad.arch.ethz.ch/blog/a-quantum-city/" target="_blank">Quantum City</a>). The bottom line is: From the smallest spatial piece of information, where we are located, how we experience space, to the urban landscape, evolving over time, the same patterns emerge. There is no clear distinction between inside and outside, static and dynamic, it's a <a href="http://en.wikipedia.org/wiki/Matryoshka_doll" target="_blank">Matryoshka</a> that unfolds in different speeds.<br />
<br />
Building up on the currently ongoing <a href="http://unarchitectedsystems.blogspot.sg/2013/07/something-between-micro-services-and-roa.html" target="_blank">Microservice</a> discussion this means Microservices, or Fog Computing, won't solve the complexity. They are just in the same state as OOP, CASE Tools or Dynamic Languages were in the beginning - before they hit the <a href="http://www.teamten.com/lawrence/writings/norris-numbers.html" target="_blank">Norris wall</a>. Our systems will grow in complexity, and while it's great not to have to worry about heterogeneous platforms, we need to understand them nevertheless. Scalability concepts still need to be understood from the bottom up, in combination with the real world problems they are trying to solve. They can be masked away by high-speed data synchronization concepts between <a href="http://facebook.github.io/react/docs/why-react.html" target="_blank">devices</a> and <a href="http://highscalability.com/blog/2010/12/23/paper-crdts-consistency-without-concurrency-control.html" target="_blank">algorithms</a>, but eventually they will pop up as <a href="http://alarmingdevelopment.org/?p=865" target="_blank">technical debt</a>, when change happens. It's the story of the magic <a href="http://juokaz.com/blog/start-with-a-spreadsheet.html" target="_blank">Excel Sheet</a> - which turns into Software and <a href="https://news.ycombinator.com/item?id=5198187" target="_blank">then back</a>. Just like a Matryoshka.<br />
<br />
The Blur building was not really comfortable, it's pretty wet. Many edgy contemporary buildings with "Open Space" concepts are also not comfortable. They seem to give maximum flexibility in a simple concept but actually are just a mask around technical debt. Which is fine as long as it's clear to the user. It won't make IT easier or more commoditized, in fact it promotes the opposite: Having to pay for highly specialized experts when you can't go back and are stuck with a tool that once seemed easy enough. In the past this situation occurred as well, but it was covered with new needs for client-server systems, GUI's, the Internet, Mobile and <a href="https://medium.com/the-future-interface/the-future-interface-design-to-make-technology-human-f4287574ee6b" target="_blank">Omnichannel</a>. If there is no new paradigm around the corner many users will sooner or later realize the technical debt they finally have to solve.<br />
<br />
Let's see which technologies will be chosen then, when they finally need to be harmonized with real change - when the users realize <a href="http://www.thebaffler.com/salvos/world-processor" target="_blank">they</a> work for the computers. I hope it's simple components, that need a fair bit of understanding - and not oversimplified magic bullets. The more important change in IT should be a culture that embraces problem solving, diversity and organization rather than fancy facades.<br />
<br />
<blockquote class="tr_bq" style="text-align: right;">
<i>The Next Challenge of the Web is Us</i></blockquote>
<blockquote class="tr_bq" style="text-align: right;">
<a href="https://www.youtube.com/watch?v=QPRqQH_30hU" target="_blank">Christian Heilmann </a></blockquote>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-28262133890091136412014-05-11T18:26:00.001+02:002014-05-11T18:28:47.165+02:00How doctors and pilots verifyThe current discussion around Test-Driven-Development (see <a href="http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html" target="_blank">here</a>, <a href="https://www.destroyallsoftware.com/blog/2014/tdd-straw-men-and-rhetoric" target="_blank">here</a> and the <a href="https://plus.google.com/events/ci2g23mk0lh9too9bgbp3rbut0k" target="_blank">Hangout</a>) seems a bit far-fetched, as I have rarely seen such dogmatic applications, but some aspects of it are indeed interesting.<br />
<br />
Mocking clearly has its limitations (whoever used Spring or O/R mappers knows the uncanny feeling of mocking tons of Spring internals). What I find fascinating more interesting, though, is the cultural discussion of pre-planned versus experimental approaches. In the <a href="https://www.youtube.com/watch?v=z9quxZsLcfo" target="_blank">Hangout</a>, Kent Beck makes a point about how he defines the problem and incrementally finds a solution whereas Hansson does not even define a problem but rather tests something directly with any "owner", prototyping. From their example it looks like Beck usually solves algorithmic problems whereas Hansson seems to face interface and integration issues. Fowler seems to see both advantages, and tries to reconcile without a strong opinion.<br />
<br />
The more interesting underlying question is, where verification is required and how it is achieved. This reminded me of a chat I recently had with a friend, a doctor, and another friend, a pilot. The pilot explained, how nowadays conflict resolution between the various (social and technological) subsystems of a plane, also between the various autopilot systems, is a major part of the training. The human mind becomes the reason between algorithms (an argument <a href="http://en.wikipedia.org/wiki/How_We_Decide" target="_blank">Lehrer</a> already wrote about, also see <a href="http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael-lewis.html?_r=1" target="_blank">Flash Boys</a>). Understanding feedback cycles and how systems interact is of course an important tool to solve conflicts. The doctor added that, interestingly, this systemic understanding becomes <a href="http://life.nationalpost.com/2011/11/22/why-doctors-should-be-treated-more-like-airline-pilots/" target="_blank">more and more relevant</a> in her work. She went on explaining that, just like in software engineering, medicine has a discussion about whether it's more of an art or more of a science. In her opinion, which I liked very much, medicine is <i>an art based on science*</i>.<br />
<br />
Software Engineering could be the same art based on science. Hypothesis (i.e. scientific)-driven Validation in the form of TDD would therefore be required for the scientific part, the algorithms and processes. Empiric (i.e. psychological, social)-driven Validation would be required the arts part, the composition, experience and elegance. The latter would be hard to test, it would probably need to be <a href="http://softwareflow.wordpress.com/2010/09/02/the-merging-of-lean-and-agile/" target="_blank">monitored</a> and replayed in simulations or regression-tests, at least documented, in order to be manifested for future generations of programmers. Clearly defining which part of a software system belongs into which domain might end many of the discussions, and bring back reason.<br />
<br />
<span style="font-size: x-small;">*) Apparently, this quotes originates from <a href="http://en.wikiquote.org/wiki/William_Osler" target="_blank">William Osler</a></span>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-6849806902838215926.post-7974753441385931252014-04-20T22:33:00.000+02:002014-04-20T22:33:21.131+02:00Simplicity CoachRecently I had a chat with colleagues from a very complex global <a href="http://en.wikipedia.org/wiki/Omni-channel_Retailing" target="_blank">omnichannel</a> project, rebuilding a huge solution from scratch. The end-user was only supposed to realize performance improvements and nicer user experience, no service interruption. The colleagues involved a service design firm in the process. Although the project was complex, it was so from a technical point of view - business processes would only be changed afterwards. Hence I asked the guys: Why involve a <a href="http://thisisservicedesignthinking.com/" target="_blank">service design firm</a>?<br />
<br />
Their answer: We needed a <i>Simplicity Coach</i>.<br />
<br />
You will find this term in many ways around the web, in fact there has been a true simplicity hype in recent year (e.g. <a href="http://lawsofsimplicity.com/" target="_blank">Maeda</a>, <a href="http://www.simplify.de/die-idee/tiki-kuestenmacher/" target="_blank">Küstenmacher</a>). What struck me, though, was how they used it informally as a standard role. The service design team was part of the architecture team, in fact it was their daily business to challenge each others ideas. The architects from a quality, scalability and maintainability perspective. The design team from a simplicity perspective. The architects asked: "Can it be done safer?" whereas the service designers asked: "Can it be done easier?". Together they achieved true resilience in building an architecture based on small, versatile services, yet with a strong vision, common understanding and buy-in from everyone in the project.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6849806902838215926.post-29755888402263367682014-03-18T09:23:00.001+01:002014-03-18T11:18:34.463+01:00Permanent CrisisMaybe it's just my selective perception, but it seems in the last weeks there has been a burst in interesting IT industry topics which did not only cover technology.<br />
<div>
<ol>
<li>Uncle Bobs move towards more <a href="http://blog.8thlight.com/uncle-bob/2014/02/27/TheTrustSpectrum.html" target="_blank">governance</a> and how <a href="http://readwrite.com/2014/01/24/github-meritocracy-rug" target="_blank">meriotocracy</a> might not be the right solution if <a href="http://modelviewculture.com/pieces/the-making-of-myths" target="_blank">myths and stereotypes</a> persist</li>
<li>The question whether <a href="http://cacm.acm.org/magazines/2014/1/170859-estimation-is-not-evil/abstract" target="_blank">estimation</a> is evil, if we should <a href="http://pragdave.me/blog/2014/03/04/time-to-kill-agile/" target="_blank">stop using the word Agile</a> and why software development <a href="http://scottberkun.com/essays/46-why-software-sucks/" target="_blank">still sucks</a></li>
<li>Fowlers <a href="http://martinfowler.com/articles/microservices.html" target="_blank">Micro-services</a> and the "history repeating" argument about <a href="http://literateprogrammer.blogspot.co.uk/2014/03/the-microservice-declaration-of.html" target="_blank">SOA</a></li>
</ol>
<div>
Despite the obvious lack of reason in most of the discussions*, for me they also have something else in common: They're all about trends a new generation of coders have embraced particularly dogmatic in the last 5-10 years which seems to reveal they are <a href="http://en.wikipedia.org/wiki/No_Silver_Bullet" target="_blank">no silver bullets</a> in fact. It's like all the <a href="http://en.wikipedia.org/wiki/Hype_cycle" target="_blank">hype cycles</a> ended up in the Trough of Disillusionment at the same time.</div>
</div>
<div>
<br /></div>
<div>
Are we still in <a href="http://en.wikipedia.org/wiki/Software_crisis" target="_blank">software crisis</a>?</div>
<div>
<br /></div>
<div>
At Baruco 2012, Paolo Parrotta <a href="http://www.youtube.com/watch?v=9IPn5Gk_OiM" target="_blank">diagnosed</a> wittily: <i>Software Crisis? You cannot be in crisis for 20 years! </i></div>
<div>
<br /></div>
<div>
It's been 54 years.</div>
<div>
<br /></div>
<div>
Maybe software development just does not get really better? Maybe we are moving just with the real world, like everyone else, not being "special" or "visionary" by definition but actually just normal people with (or without) a degree in computer science, influenced by the same socioeconomic value shifts and biases? Maybe, as <a href="http://online.wsj.com/news/articles/SB10001424053111903480904576512250915629460" target="_blank">Software Is Eating The World</a>, software just becomes as chaotic as the world itself?</div>
<div>
<br /></div>
<div>
Yes, the field improves but it's a little signal in a lot of noise. I am currently writing a book on <a href="http://www.amazon.de/Was-Software-mit-Architektur-tun/dp/3868021183" target="_blank">building architecture vs. software architecture</a>, which is why I haven't been blogging recently. It's not that I want the ideas to sell, no, a book does not bring you much money. But I see that I have to think ideas through, reading stacks of books, old notes dating back 5 years, re-reading articles and posts. And I see some improvement while doing this. But I also see many themes repeat over and over again. Sadly, instead of agreeing on a common sense, views seem to become more dogmatic and further apart. It's always the same pattern (<a href="https://news.ycombinator.com/item?id=5198187" target="_blank">The Excel Pattern</a>):</div>
<div>
<br /></div>
<div>
Someone has a simple idea. Followers who believe in control systems and analytic engineering turn it into a methodology. Someone else sees the methodology and has a simpler idea. Then, both the old idea and methodology are declared evil.</div>
<div>
<br /></div>
<div>
As long as we are not working on the core biases and expectations towards software engineering and architecture this vicious circle will never stop and work towards real improvement, in technology as well as in team building, fairness and business value. Maybe this will turn IT into an art thereby. That's the risk we should be willing to take.</div>
<div>
<br /></div>
<div>
My current book stack - wish me luck:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3kp7pUWE1mcu76KefCkm3V5sHKRFDcjCkD6x2vuTZ-EOq6IG6EOEFLR7tR75rDAAkVpjUSHk10WAbY86n_a0u95RUVMwH4zRdSxDtStRpS8gtjl5gafVuvGDYHCCpbFnCcUvyyPVbEhqC/s1600/bookstack.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3kp7pUWE1mcu76KefCkm3V5sHKRFDcjCkD6x2vuTZ-EOq6IG6EOEFLR7tR75rDAAkVpjUSHk10WAbY86n_a0u95RUVMwH4zRdSxDtStRpS8gtjl5gafVuvGDYHCCpbFnCcUvyyPVbEhqC/s1600/bookstack.jpg" height="150" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<span style="font-size: x-small;">*) But slowly returning, the original posts were mostly reasonable though</span></div>
Unknownnoreply@blogger.com0