Dave Snowden on coping with complexity

My account of a talk that Dave Snowden of Cognitive Edge gave at a Gurteen Knowledge Cafe event hosted at the Telecom Tower on 17 February 2009. Rejecting ‘scientific management’ and ‘business process re-engineering’, Snowden advocates a new approach for organisations seeking to understand and cope with crisis and complexity.

Dave Snowden at the BT tower, Feb 2009

Dave Snowden at the Gurteen Knowledge Cafe at the BT Tower in London: 17 February 2009. Photo Conrad Taylor.

35 YEARS AGO, I lived in a commune at Wick Court, a Jacobean manor house situated between Bath and Bristol, which was then the head office and conference centre of the Student Christian Movement. Another of the 17 or so communards was the SCM’s Chairman, Dave Snowden, a spare, bearded Welsh Marxist Catholic philosophy and physics graduate. Taking no prisoners in argument, and delighting in winding people up, Dave gave the impression of having slipped from another historical epoch, though whether it was from some future time, 1920s Russia or perhaps the Counter-Reformation was less easy to judge.

Our paths separated after Wick Court. During summer 2008 I was again in the Bristol region, staying with my friend Bob Bater, who works in the area of knowledge management — a field I have strayed into from my lifelong interest in information design and communication. Bob told me about the work of a company called Cognitive Edge and its founder Dave Snowden, a description of whom sounded remarkably familiar. So when I had the opportunity to hear a lecture by Dave Snowden at a Gurteen Knowledge Cafe event at the BT Tower in London, I made an effort to go along.

Quite apart from the joy of re-union (and some grand views of London from the revolving restaurant at the top of the tower), it was also fascinating to discover how Dave’s thinking and mine have been oscillating around some of the same attractors over the years. Mind you, he has made a great deal more use of them than I have, through his many consultancies with major enterprises, and even state security organisations.

The topic for the day’s Cafe session was How can we best keep employees engaged in their work, in the current economic climate? But Dave’s talk addressed a much wider set of issues around what organisations should do to adapt to a complex and crisis-ridden world — one in which the ‘lessons’ of the past are no longer a guide for future action, and ‘management by objectives’ doesn’t equip us with the agility and intelligence to respond to emergent threats and opportunities.

Paradigm shifts in management thinking

The word paradigm has many sloppy uses, but Dave deploys it strictly to mean a theoretical framework: very close to the use established by Thomas Kuhn in his book on the sociology of scientific knowledge, The Structure of Scientific Revolutions. For a period of time, a paradigm dominates because it seems best to organise our thinking about the world; but then, a new approach that flies in the face of received wisdom and which is initially ridiculed gets taken up; and at a certain point, a ‘paradigm shift’ occurs and the former heresy becomes the dominant idea. The revolutions associated with Copernicus and Darwin are two such instances.

In the last 120 years, said Dave, two paradigms in succession have dominated management thinking. The first of these, dubbed Scientific Management (or ‘Taylorism’ after Frederick Winslow Taylor, its progenitor), focuses on the function of the business organisation and takes a command and control approach to the organisation of work. In this period, the emphases are on mass production, time and motion, automation and efficiency. The Machine is the model, and people are viewed as components of the system.

The second paradigm identified by Dave Snowden is Business Process Re-engineering, which took off in the 1990s. The belief in BPR is that one should look anew at the company’s objectives and define the outcomes or goals you want to achieve, the values or behaviours which you want your staff to adopt. Dave notes that this makes several assumptions, among them that you can pre-define your goals, and that one can rely on a form of causality in which results are repeatable, that the past is a guide to the future. The approach also relies on scalable and reliable technology (particularly information and communication technology) through which metrics can be gathered and control can be driven.

And now, said Dave, we are entering a third paradigm, which it was his job to explain. Rather than projecting an idealised future, this approach seeks to understand and manage the evolutionary potential of the present. It is people-focused, exploits mass collaboration and pervasive social computing, and relies on distributed rather than centralised cognition.

Dave also displayed for us a quote from Seneca:

The greatest loss of time is delay and expectation, which depend upon the future. We let go the present, which we have in our power, and look forward to that which depends upon chance, and so relinquish a certainty for an uncertainty.

Catastrophes, niches and predators

Dave argues that times of economic crisis also tend to coincide with paradigm shifts, the times when the previously dominant set of ideas appear weak and the opportunity arises for the new model to take hold. Focusing on evolutionary potential and emergent trends in the present is particularly beneficial in opening our attention to novel possibilities for ways of doing business.

An idea he borrows from evolutionary biology is ‘Dominant Predator Theory’. This holds that after a major disturbance of the ecology, there will be one predator species that will first make the move into the new niche, occupy it and dominate it. The speed with which the predator can move into this niche is key to its dominance; and agile companies will be the ones that emerge from the current crisis in best shape.

Systems and agents: three configurations

To understand how an organisation might exploit the third paradigm, Dave presented three system models: ordered, chaotic and complex. By ‘system’ he means networks that have coherence, though that need not imply sharp boundaries. ‘Agents’ are defined as anything which acts within a system. An agent could be an individual person, or a grouping; an idea can also be an agent, for example the myth-structures which largely determine how we make decisions within the communities and societies within which we live.

  • Ordered systems are ones in which the actions of agents are constrained by the system, making the behaviour of the agents predictable. Most management theory is predicated on this view of the organisation.
  • Chaotic systems are ones in which the agents are unconstrained and independent of each other. This is the domain of statistical analysis and probability. We have tended to assume that markets are chaotic; but this has been a simplistic view.
  • Complex systems are ones in which the agents are lightly constrained by the system, and through their mutual interactions with each other and with the system environment, the agents also modify the system. As a result, the system and its agents ‘co-evolve’. This, in fact, is a better model for understanding markets, and organisations.

Most people understand how to think about order and chaos, but understanding complexity requires a new mind-set. One property of complex dynamic systems is that patterns may take shape within them that can rapidly escalate if they find reinforcement. Many people know of the phrase ‘the butterfly effect’ to express how interactions between agents can sometime build rapidly into something that would have been hard to predict. The key to managing complex systems is therefore to attune to spotting emerging trends as early as you can, sometimes referred to as weak signal detection. If you have the means of detecting signals early, you can move to boost those that you view as positive, and dampen down those which are negative.

The childrens’ party metaphor

I don’t remember the younger Dave as a raconteur, but he’s certainly an accomplished one now. To explain the difference between the above three systems and how to manage them, he told a story about the rules around a birthday party for his 12-year-old son, and his 16-year-old daughter.

What would happen if you were prepared to model a 12-year-old’s party as a chaotic system? Anything could happen, including the house being reduced to ashes. What would an ‘ordered system’, ‘management by objectives’ approach look like? Dave span out a fantasy process starting with declaring the objectives for the party, pinning up motivational posters, writing a project plan with clear milestones at which progress would be measured against declared outcomes, starting the party with motivational video and Powerpoint, and after the party, reviewing the outcomes and updating the best-practice database. (That raised a laugh and ripple of applause.)

Far better to model the kids’ party as a complex system. You set boundaries (‘cross that and you’re dead!’) — but you keep them flexible, you see if you can stimulate the formation of ‘attractor mechanisms’ that will set up some good patterns, and you keep a weather-eye open for bad patterns that you will want to disrupt. Your aim is: to manage the emergence of beneficial coherence. And the benefits in business of adopting this process, Dave asserted, are that you get as dramatic an improvement as business process re-engineering ever got you, and at a low cost if you spot the patterns early and move appropriately.

Dave’s 16-year-old daughter’s party raised the stakes to a new level, as a previous incident involving a couple of friends and a bottle of vodka had given them cause to know. Here, the rules were more carefully set: who could be invited, what kind of alcohol etc. Shortly before the impending event, Dave was working with a well known client organisation in Langley, Va. USA, and sympathetic friends there loaned him an extensive array of surveillance equipment. With the house fully bugged, Dave and his wife stayed in their bedroom, tuned in, and only twice had to make forays so as to be on hand to disrupt possibly negative emergent patterns. ‘In a complex system, what counts is your weak signal detection,’ Dave explained.

Distributed cognition

Swindon's Magic Roundabout

Swindon’s ‘Magic Roundabout’

Distributed Cognition is a key concept in Dave Snowden’ vision of the new way to manage radically networked organisations, and he explained it with reference to the so-called ‘Magic Roundabout’ in Swindon. I’ve been trying to get my head round it (according to Dave American visitors can’t get cars round it!), and my best way of conceptualising the roundabout is as a very compact ring road, with traffic in both directions, linked to five radial roads by a mini-roundabout at each junction.

The Magic Roundabout offloads onto drivers the decisions about how to interpret the traffic flows, when to go and to stop, which way to turn and so on. The roundabout has no traffic lights. That’s what distributed cognition is about: you rely on the intelligence of the agents in the system. Since it was introduced it has never jammed up.

When the Magic Roundabout was set up, the partitions within the roadways were set up on a temporary basis, movable on the basis of observations by policemen and road traffic engineers about how people reacted to them. And the moral that Dave drew from that was: ‘Move away from attempting failsafe design, towards setting up safe-fail experimentation’. The reason? Because you cannot analyse the problem space fully in advance, and you have to be prepared to adjust systems interactively until you find that they work.

Putting distributed cognition to work

Within complex systems, for the organisation to become alerted to early weak signals of pattern formation, Dave argues that it is necessary to work with objects of fine granularity — be these objects organisational ones or informational ones.

As for the granularity of levels of human organisation at which distributed cognition works, we should also be aware of certain numerical thresholds: ‘Organisations work differently as 5 or less individuals, 15 or less, and 150 or less’. 150 is Dunbar’s number, the limit on the number of ‘acquaints’ a normal person can maintain; it is also a typical kind of size for a military fighting unit such as a company, and W.L. Gore & Associates (the makers of GoreTex) divide business units when they exceed this size. Dave described fifteen as a ’limit of trust’ and related to the size of the typical extended family; and five as Miller’ Number, the limit of short-term memory. (Note: George Miller actually defined this limit as ‘the magical number seven, plus or minus two.’)

Spotting the ball, missing the beast

Dave then got us to take part in an experiment which demonstrated both the strengths of distributed cognition, and one of the failings of all human cognition about which we should be on our guard. He prepared us for this by getting us to think about the fairground competition that invites us to guess how many jelly beans are in a large jar. If the first guess is visibly posted up, subsequent guesses tend to bracket this; whereas if each person remains unaware of the other guesses, cognition remains distributed and the average of the guesses is more accurate.

Our task was to watch a video in which three students dressed in white T-shirts walked within a circle passing a basketball between them, while another three dressed in black T-shirts also circled between them, likewise passing a ball. How many times was the ball passed between the students in the white shirts?

Who spotted the gorilla?

Daniel Simons’ attention test: while concentrating on counting the number of times the ball is passed, most people don’t see the gorilla…

This visual attention test wasn’t new to me, as a couple of years before I had read in Scientific American about Daniel Simons’ 1999 experiment. Most people concentrating on this task fail completely to notice that half way through the clip (see it here on YouTube), someone in a gorilla suit shambles in from the right, grins at the camera and beats his chest, then leaves. Many in this audience likewise missed the gorilla — but as a thumbs-up to distributed cognition, the reported number of passes neatly bracketed the correct answer, which is 14. (I anticipated the gorilla, but failed to count three of the passes as a result!)

Given how we were prepped for the task, the gorilla was a typical ‘weak signal’ that slipped under most people’s radar. There is a moral to this: we don’t tend to notice what we are not looking for. According to Dave, people on the mild side of the autistic spectrum do tend to notice the gorilla despite concentrating on the task.

Stories and filters

With both positive and negative effects, human beings and societies are closely attuned to stories. Indeed a book by John D Niles characterises our species as Homo narrans. Families, workgroups, companies and whole cultures have stories, and dominant narratives act as primary filters through which we see (or fail to see) data. (Indeed, I see this as related to the idea of a ‘paradigm’ anyway.)

Cognitive Edge has developed methods and tools which harness the illuminating power of stories, and distributed cognition.

But what if we were able to capture stories and make use of them to build business intelligence? Dave’s clients are coming to value the freeform comment over conventional forms of survey, and also in preference to the output of focus groups (in which the danger of bias from the facilitator is high). At Cognitive Edge they have been developing methods and software tools which harness the illuminating power of stories beside distributed cognition, and make the raw data available directly to decision makers together with useful quantitative summaries and visualisations — what Dave calls disintermediation.

Dave gave several examples of this in practice, but one will suffice for here. In the My Visit project, National Museums Liverpool wanted to collect and analyse feedback from the hundreds of thousands of school children who visit and interact with its staff. The children leave small text comments, and are asked to ‘self-signify’ the comment fragments by placing them on a sliding scale between two negative polar opposites. Examples of scales:

  • From ‘staff patronise the children’
    to ‘staff are too childlike and pathetic’
  • From ‘too much to see, and it’s overwhelming’
    to ‘not enough to keep me interested’
  • From ‘rushed from place to place; missed things’
    to ‘too much time in one place’

The software provided by Cognitive Edge — SenseMaker — presents the spectra of results as histograms. Obviously, what the museum management would like to see would be positive stories that sit comfortably in the middle of these negative extremes. But to detect problems and fix them early, what they keep their eye on is the histogram columns at the negative edges. The long-term pattern displays as blue bars, the last 48 hours’ results show in red. Behind the simple display is a database of all the comment fragments, and the management can drill down quickly to read the individual, unmediated complaints.

Sensemaker software

The SenseMaker display lets museum staff monitor emergence of trends, and drill down to read comments left by children. Image by permission of Dave Snowden, displayed here quarter size (click on image to inpect full-size image in a new browser window).

Feedback from the system has helped the museum to refine and develop over twenty learning activities for children, and has quadrupled the figures for learning visits.

Dave showed other examples covering e.g. attitudes of employees to their leadership, scenario elicitation within a Canadian forestry service, and so on. One of the latest uses of SenseMaker sounds fascinating and I look forward to learning more: the Children of the World project aims to create a cultural map of the world, getting children to gather stories from their families that are reflections about past and present life, future hopes and aspirations. Starting in Liverpool, the project will branch out soon to Australia and Canada, Bangladesh and Africa.

Shifting paradigms

Summing up, Dave argued that while there is nothing wrong with the BPO paradigm for the things it’s good at, it doesn’t deal well with complexity. So what are the practical implications for organisations prepared to make the switch?

  • The demands of weak signal detection imply setting aside periodical analyses based on surveys and focus groups, in favour of narrative-based research that continuously captures and displays a disintermediated evidence base, as described above.
  • Rather than determining outcomes and measuring performance against targets (what’s ruining UK health and education), shift to measuring the impacts of activities and allow for emergence and adjustment.
  • Rather than using centralised scenario planning, set up systems which allow for scenarios to be generated by employees.
  • Break people out of their boxes. If you can assemble ‘crews’ with a membership that spans functional job boundaries and put the crews ‘on watch’, you will have at your service a team whose skills exceed those of any one of its members.
  • Best practice databases are all very well, but they lead you to rely on what you ‘learned’ in the past. Dave recommends knowledge management practices that collect narrative fragments.
  • ‘Practice-Informed Theory’ similarly assumes stability and doesn’t deal well with emergent behaviours and situations. Indeed Dave felt that more ‘Theory-Informed Practice’ is desirable. For example if there were more professional ethical principle applied in banking and accounting, would we be in the mess we’re in now?

A similar Dave Snowden presentation as MP3…

IN APRIL 2009, Dave Snowden gave a talk with similar contents to the UK Chapter of the International Society for Knowledge Organization. I recorded the talk, and the MP3 audio and Dave’s slides can be retrieved from the ISKO-UK Web site.