Notes are not enough! Why relying on your notes will lead you down the garden path.
•
7 gefällt mir•1,979 views
You take great notes, right? Have you ever compared them to a transcript? It’s amazing how much of the important stuff we miss as we take the time to interpret what someone says, formulate what we’re going to write, then go through the physical act of writing before switching back into listening again. In this presentation we’ll walk through the model of communication, exploring the limitations of perception, cognition, attention and memory. By the end I hope you’ll appreciate why your notes are not enough. Presented at Design Research 2017, Sydney
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Ähnlich wie Notes are not enough! Why relying on your notes will lead you down the garden path.
Ähnlich wie Notes are not enough! Why relying on your notes will lead you down the garden path. (20)
Mehr von Ash Donaldson
Mehr von Ash Donaldson (10)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Notes are not enough! Why relying on your notes will lead you down the garden path.
- 1. NOTES ARE NOT ENOUGH! @ashdonaldson Design Research 2017, Sydney
- 3. I decided that we shouldn’t even look at the notes. We’d just go with our gut. Senior Researcher playing back findings
- 4. We can all capture stories… …but are they true? *sniff*
- 5. Communication feels intuitive… …but it’s really complex
- 6. The great enemy of communication, we find, is the illusion of it. William H. Whyte
- 7. Communication can be critical Melbourne tower, this is Alpha Echo Charlie inbound from the South at 5,000. Request descent to join downwind of zero niner at 1,000.
- 8. An interview is a directed conversation. J. & L. Lofland
- 9. How we think about thinking…
- 10. Mental models of the mind Hydraulic
- 11. Mental models of the mind Hydraulic Mechanistic
- 12. Mental models of the mind Hydraulic Mechanistic Information Processor
- 13. Information Processor Model - Fixed focus - Captures everything
- 14. Information Processor Model - Fixed focus - Captures everything - Logically interpreted - Systematically classified
- 15. Information Processor Model - Fixed focus - Captures everything - Logically interpreted - Systematically classified - Stored faithfully - High definition
- 16. Information Processor Model - Fixed focus - Captures everything - Logically interpreted - Systematically classified - Stored faithfully - High definition
- 18. It all starts with an intent Sender
- 19. that intent exists within an environment Sender Interviewer’s environment
- 20. the intent needs to be expressed as an idea Sender Interviewer’s environment Encodes
- 21. through a channel Sender Interviewer’s environment Encodes Channel
- 22. carrying transmitted information Sender Interviewer’s environment Encodes Message Channel
- 23. which has to be perceived Sender Interviewer’s environment Encodes Message Channel Channel
- 24. [ attention is limited ] • We don’t have a fixed attention • Serial switching: Can only attend to one thing at a time • Easily distracted: Directed Attention Fatigue (DAF) • Expensive and rapidly exhausted • Affected by sleep, stress, nutrition, medications, age & exposure to external stimuli
- 25. [ perception is tiny ] • We don’t take everything in as input • We perceive only tiny fragments • Our minds construct a seamless movie • We more readily perceive things we expect • Pilots divide the sky into quadrants to scan for aircraft • Look under a pier: you’ll notice more fish the longer you look
- 26. Inattentional Blindness: The Monkey Business Illusion - https://youtu.be/f94o3B3csYI
- 27. …and interpreted by the receiver Sender Interviewer’s environment ReceiverEncodes Message Decodes Participant’s environment Channel Channel
- 28. [ cognition ] • Our brains do not function like CPUs • Our cognition is based on pattern matching • We draw heavily on past experiences and present cues • We satisfice - stopping when we have enough information to make a conclusion • We are subject to many predictable biases
- 29. Nothing is so difficult as not deceiving oneself. L. Wittgenstein
- 31. [ memory ] • We do not store information like a hard drive • Long-term memory is constructive • The more you remember it, the further from the truth it is • Short-term memory is fleeting • Fills quickly (4 chunks - not 5 to 7) • Gets replaced by the next thing we attend to • Has limited duration (10 to 20 seconds)
- 32. Interviewer’s environment Participant’s environment both parties have a history Sender ReceiverEncodes Message Decodes Channel Channel
- 33. then a signal is sent back Sender Interviewer’s environment Receiver Receiver Sender Encodes Message Message Decodes Participant’s environment EncodesDecodes Channel Channel Channel Channel Feedback Feedback
- 34. Noise Noise Noise Noise …of course, there’s always some type of noise Sender Interviewer’s environment Receiver Receiver Sender Encodes Message Message Decodes Participant’s environment EncodesDecodes Channel Channel Channel Channel Feedback Feedback
- 35. Noise Noise Noise Noise Transactional model of communication Sender Interviewer’s environment Receiver Receiver Sender Encodes Message Message Decodes Participant’s environment EncodesDecodes Channel Channel Channel Channel Feedback Feedback
- 37. CPM-GOMS of a note
- 38. Add a third perspective Interviewer’s environment Participant’s environment Notetaker’s environment
- 39. So what should you do?
- 40. 1. Record your interview
- 41. When reviewing the footage I saw their expressions in the moments when I was looking away. It was intense, and important to re-live some of these interviews seeing new things. Natalie Rowland Qualitative Specialist (redrollers)
- 42. Recording interviews: Tips • Craft a good recording release • Aim for video. If not, use audio • Skype or Facetime: We use eCamm’s Call Recorder • Aim for a quiet, well-lit space in context • Test your recording equipment
- 45. We had 3 notetakers for one of our interviews (most notetakers I’ve ever had) and I just combed the transcript. We missed about 15 critical points that were said. Alyce Lythall Senior Researcher (Tobias & Tobias)
- 46. Transcripts: Tips • Use participant coding e.g. T1P03 • REV.com US$1/min • Clients appreciate research documentation • Clean up your audio
- 47. 3. Build in enough analysis time
- 48. Analysis: Tips • Create a cadence that allows digestion • Allow at least 3x the research time • Build the time into projects up front • If you can’t sell analysis time, hide it • Use Quicktime Pro or similar that will allow you to playback recordings at 3x • Comb transcripts for insights and verbatims
- 49. Notes are not enough! • Recordings and transcripts give objective feedback: • Re-live stumbles and correct next time • Every interview recording is an opportunity to become a better researcher • Recordings and transcripts = Good data in
Hinweis der Redaktion
- The last decade has seen a resurgence of design research. Large organisations are learning from the startups that threaten their business by following the mantra to just “get out of the building” because “talking to even one user will make your product better.” The lean movement has created the need - and many people have ended up in a research role by just being proactive and getting out there - which is great! But like most things in design, this presentation was borne out of frustration. Frustration with seeing this new breed of researchers making fundamental errors in their approach.
- Errors that put into question the validity of their findings. What’s worse, there is no way of them even knowing what they are doing is wrong. Qualitative research is a tricky area. It’s about uncovering and capturing people’s deep stories rather than a surface description of a large sample of a population.
- On one hand, it seems intuitive. Everyone feels as though they can talk to people and reliably capture their stories. But if the research is not done correctly, who can dispute what the researcher reports? Neither the client, nor the researcher are ever the wiser if what is reported turns out to be misleading or just plain wrong. Products and services can be designed based on a faulty premise. Bad data in. Bad data out.
- But qualitative research is so difficult precisely because it seems so simple. We can all converse, right? We do it every day and we do it well. Communication is innate to us. It feels intuitive and accurate, but it’s not. Communication is actually really complex and full of errors. We just don’t notice them.
- Many of you know I come from a background in Human Factors. We design for situations in which people may get injured or die as a result of miscommunication. So it’s impressed on us early how critical it is to account for possible leaks and misinterpretations when people have to work together - and there are many.
- In aviation, that means constraining everything: Using a single language internationally - English; Having a strictly controlled vocabulary with phrases like ‘inbound from the South’; Using phonetics like ‘Alpha Echo Charlie'; Using controlled articulations like ‘fife’ for five or ‘niner’ for nine; Indicating who’s in control of the aircraft by the first pilot saying ‘Handing over’ and the second having to respond with ‘Taking over’ before the first pilot releases her controls; and so forth. There’s a lot of constraint and control employed to decrease the risk of misunderstanding - unlike in the real world.
- So let’s talk about the staple of qualitative research: the interview. Today I want to break down an interview into it’s component parts and walk you through some critical concepts, so you can understand why I’m often heard saying “Notes are not enough!”
- Before we get into the nuts and bolts of an interview, we have to dispel some myths of how our mind works.
- Our mental model of how the mind works is always constrained by our latest technology. In the 5th century BC, Hippocrates advocated an early hydraulic model based on the 4 humours. Aqueducts and hydraulics were the cutting edge technology of the day.
- Descartes then famously proffered the automata / mechanistic doctrine in the 1600s - reflecting the clockwork machine technology of the time
- At school, I was taught the Information Processor Model - which reflects our understanding of the ubiquitous technology of today - computing.
- Now in the information processor model, our senses are like video cameras - faithfully capturing all sights and sounds - and they stay focussed as long as we point our metaphoric camera at the subject.
- This data is then logically interpreted and classified by our CPU-like brain.
- And written faithfully in high definition to the hard drive of our memory - for retrieval at any time.
- Of course, the information processor model, whilst intuitive and easy to grasp, is just plain wrong - as we’ll see. But it’s our mental model - and so the basis for plenty of self deception when it comes to how we understand communication
- There are many models of communication - each with their strengths and weaknesses. Some are extraordinarily complex, so for the sake of expedience, we’ll use the simplest applicable model to explore how an interview works - the transactional model of communication.
- It all starts with an intent. The sender is the message creator that has an intent to communicate. They have their own physical capabilities and constraints: The clarity, resonance and timbre of their voice, Their motor control, cognitive development and so forth
- That Sender exists within their own background and environment: Socially, Culturally and Relationally. For the Social dimension - Society shapes the way a person communicates. There are norms, values, laws and other restrictions of a society to communicate within a specific limit. e.g. greeting people when meeting, thanking, apologizing, etc. For the Cultural dimension, it’s determined by the caste, class, race, ethnicity, religion, gender and sexuality. This is the lifestyle and identity of the person. People who come from the similar cultural contexts can communicate more effectively. For the Relational dimension, it’s based on relationship history and manners. You speak differently with an old friend than you do your parents, an acquaintance, an authority or a complete stranger.
- The intent then needs to be expressed as an idea Encoding is turning intent into messages by generating content and symbols. This is shaped by the mental models, languages and forms of expression the sender knows and understands. A well-read or multi-lingual person may be able to add more nuance to their communication than someone with a limited education.
- Then through the channel The channel is the medium that carries the message. This could be sound waves through the air, light reflecting off an object, electronic signals along a copper wire and so forth
- The message is simply the transmitted information that passes through the channel - the content and symbols in whichever form it is created.
- Then the message has to be perceived. This is where we draw back to our Information Processor mental model. To perceive something, you first have to attend to it.
- Our attention is not fixed like a camera on a tripod. We can only consciously attend to one thing at a time. When we talk to someone whilst driving, we’re serially switching - attending to the conversation mostly, but flickering our attention back and forth to the road. Serial switching is so fast, it makes us feel like we’re doing multiple tasks at once. When we want to concentrate, this is an issue because our attention is so easily distracted. Directed Attention Fatigue sets in quickly and is the result of overuse of the brain’s inhibitory mechanisms, which handle incoming distractions while maintaining focus on a specific task. Attention is also really expensive calorifically. Our brain uses 20% of our consumed calories and most of that is due to attending to things. And our modern environment is designed to constantly grab our attention - depleting our resources through the day.
- As a pilot you are taught to break the sky into quadrants and systematically scan each quadrant, line by line to spot approaching or conflicting aircraft. This is because we only clearly see about the size of a 20c piece. The rest of the picture we see is made up by what our mind expects to be there. Our minds cunningly construct a seamless movie of reality for us. Another fun thing to do to demonstrate this is to look under a pier. The longer you look, the more fish you’ll notice. This is because that clear focus of your eye darts to and investigates the movement it detects, revealing new types of fish the longer you look. I assume that a number of you would have seen the upcoming demonstration of inattentional blindness.
- The point of this demonstration is, when we are attending to something, we miss significant amounts of what’s going on around us. If you are doing research and attending to either facilitating or taking notes, you’ll miss significant portions of what is happening.
- After being perceived, the message then needs to be interpreted by the receiver. Just like the Interviewer, the Participant also has their own physical capabilities and limitations to receive a message. They also have their own social, cultural and relational context Decoding is affected by cognition and memory, so let’s revisit those parts of the information processor model.
- Of course we don’t process thoughts logically like a CPU. We are, however, really good pattern matching creatures. So good, we get many false positives - seeing patterns where none exist. This is why people have superstitions and ‘systems’ for gambling. We also rely on just enough information and draw on past experience and current contextual cues to make decisions - known as satisficing. And with the rise of Behavioural Economics, we’ve started cataloguing the innumerable ways in which we are predictably biased.
- The most meta of the biases we experience is the ‘bias blind spot’ - our ability to recognise the impact of biases on other people whilst failing to see (or denying) the impact of biases on ourselves. This is particularly bad for researchers who hold the strong opinion that they are not biased. Our mind’s job is to constantly fool us - creating that seamless illusion of reality which hides our biases and faulty thinking. Let’s look at an example of not only our pattern matching bias - which sees patterns that aren’t even there - but how external stimuli can fundamentally change our perception.
- Your mind is amazing at generating these false representations of reality without you being aware of them. Some simple cues or pre-conceptions can make you perceive more than is actually there.
- Even though it feels like it, we don’t store memories like videos to be played back later in High Def. Long-term memory is constructive. It recalls snippets and makes up everything in between - what’s known as confabulating. So every time you recall a memory, it’s a bit further from what actually happened. This is important to recognise when doing analysis. You can’t rely on your memory. More appropriate to note taking - short term memory is tiny and fleeting. You can only faithfully hold 4 chunks (Neuroscience has recently dismissed the 5-7 chunks). If something else grabs your attention, you lose what’s in memory and even if you don’t get distracted, it only lasts 10 to 20 seconds unless it’s stored to long term memory through repetition or association.
- So to compound the issues of understanding - both parties have their own histories and environments. Where the Interviewer and Participant's environments overlap allows for a shared understanding. If both share the same culture, profession, age, gender, etc - the overlap is greater and communication will be easier. If not, there’s plenty of opportunity for misunderstanding. e.g. ‘potential’ means very different things to a career counsellor (personal development), an electrical engineer (storage of charge) and a physicist (unused physical energy).
- Once the message is received and decoded, the Receiver becomes a Sender - providing feedback. Of course, feedback can come in as many forms as the sender’s message - from a nonverbal emotional reaction to what the participant says and how they say it.
- and at every stage there is noise 1. Physiological-Impairment Noise: Physical maladies that prevent effective communication, such as actual deafness or blindness preventing messages from being received as they were intended. 2. Semantic Noise: Different interpretations of the meanings of certain words. For example, the word "weed" can be interpreted as a pesky plant in the garden or an illegal substance 3. Syntactical Noise: Mistakes in grammar can disrupt communication, such as abrupt changes in verb tense during a sentence. 4. Organizational Noise: Poorly structured communication can prevent the receiver from accurate interpretation. For example, unclear and badly stated directions can make the receiver even more lost. 5. Cultural Noise: Stereotypical assumptions can cause misunderstandings, such as unintentionally offending observers of Islam by offering them a snack during Ramadhan 6. Psychological Noise: Certain attitudes can also make communication difficult. For instance, great anger or sadness may cause someone to lose focus on the present moment. Disorders such as Autism may also severely hamper effective communication. 7. Environmental Noise: physically disrupts communication, such as interviewing in a noisy café, a dimly lit room, a dark, hot, noisy mine
- So this is looking at communication using one of the simplest, general models - the Transactional Model of Communication. You can see how messy it is. How many opportunities there are for missing or misinterpreting things. And this is just simple conversation - without adding the effort of reading an interview script, laddering on questions or figuring out what is an important area to investigate and how to frame the probe to delve into it. All that takes much more attention away from listening to the participant. 20% or more of your time as an interviewer is taken up with these internal processes. And that’s before we even consider note taking.
- So what does it take to write a hasty note? Let’s look at the ideal situation in which you have a quiet, controlled environment, an interviewer and experienced note-taker working together to get the most out of an interview. There’s a suite of methods we can use to calculate the time it takes for a trained person to complete a task. This is known as GOMS - Goals Objects Methods and Selection Rules. To figure out the effort in note taking, we can use a particular flavour, known as Cognitive, Perceptual Motor or CPM-GOMS
- Using some GOMS modelling software known as Cogulator to calculate the time taken to process and write the note “Carer. Couldn’t keep track of Father’s spending” we see that in ideal conditions, it can take 17.6 seconds. That’s 17.6 seconds of not attending to what is being said. Sure, the note taker could serially switch between writing and listening, but due to our limitations in attention, perception, cognition and memory, what they write will then most likely become inaccurate - changing from the original thought as memory gets dumped and new information gets added. So that’s the best case scenario - an interviewer, a note taker and a participant. Sometimes there’s not even a dedicated note-taker. So how much will get missed there?
- …and let’s also add the fact that they Notetaker may not share the same social, cultural and relational context as the Interviewer. This makes the area of shared understanding even smaller.
- Something that I haven’t put down here but is very important - on the days you do research or analysis, be sure to protect yourself from external stimuli. Turn off your social media and email for the day. Don’t read magazines, watch TV, browse the Internet or listen to podcasts. Avoid environments where there is high advertising. All of these stimuli deplete the small and finite amount of attention you have for the day.
- But down to the mechanics of doing the qual research. Do everything within your power to record the interview. I know sometimes it won’t be possible, but wherever you can, do so. It will make you a better researcher for those times when you can’t record.
- As we saw in the transactional model of communication, it’s not just the spoken content you have to attend to. It’s how people express their ideas - the pitch, pace pause and intonation of their speech. And the non-verbal communication of facial expressions, posture and physical reactions add yet another, rich layer of meaning, which is why video is always a great medium - especially for difficult topics.
- Here’s some tips for recording. Have a good recording release. It should explain in Plain English exactly what the recording will be used for and how the identity of the participant will be protected. The more you protect the participant, the more likely they’ll sign a recording release. It also should provide for recording options and ask permission to follow up if the need arises. Video is the gold standard but people can be shy about having their image captured, so you should always be prepared with an audio only option. If you’re doing remote interviews, we like to use eCamm’s Call Recorder as it splits out the interviewer and participant video and audio and can capture at high-res. Of course, aim for a quiet space in the appropriate context with lighting behind you. Always check batteries, clear storage and test recording equipment the day before the session.
- Because I always like to know what other people use, I thought I’d share what our field kit includes: 2TB external hard drive 64GB iPod as a secondary recording device Belkin 6600 mAh battery backup 2 x Sandisk 128GB Class 10 SD card Telstra 4G mobile hub Panasonic HC-V770M HD Camcorder with 2 spare batteries Røde smartLav+ lapel mic for the iPod VideoMic Pro Directional mic (for the camcorder) Manfrotto Mini tripod
- Transcripts are critical to good analysis
- I make sure my team always gets transcripts of their interviews - no matter how many or how good their notetakers are. The first time a new T&Ter gets a transcript of their interview, they are usually astonished by what gets missed by themselves and their notetakers. (and how many mistakes or misinterpretations they made).
- For research ethics, we anonymise our participants with coding immediately. The code can be simple like this one is T1P02 is simply Team 1, Participant 3. Although it’s better to write the transcripts yourself (if you want to soak yourself in the research), it’s very time consuming and often not practical when working to tight deadlines. We often use a service called rev.com which is cheap and provides great results (as long as you send clean audio). It’s an easy sell to clients because it gives them something tangible from research - a searchable Word document that can be shared and referred to again later. It’s also great for storytelling to get accurate quotes. When we’ve had to record in a noisy environment, I like to use SoundSoap on the Mac to clean up the audio and extract the voice before sending it to rev.
- If you’re going to all this effort to faithfully capture what people have communicated to you, respect the participant’s stories by building in enough analysis time
- We aim to do research in sprints and build in a little analysis between participants. or at the end of the day. Also at the end of each sprint, then finally, a block of solid analysis after the research sprints are completed. This gives us the opportunity to not only digest what we’ve experienced for synthesis later, but identify gaps in the research that we can follow up with research participants later. Minimum rule of thumb is 3x the research time for analysis. Business Anthropologists typically aim for 10x but in my experience, that’s not a viable option for an external consultant. If necessary, analysis time can be easily hidden by employing sprints with weekly playbacks (and agreeing with the client to set aside time to prepare for them). You can also spread out research sessions to provide time between sessions to review prior sessions. Use software like Voice Recorder Pro or Quicktime Pro to playback sessions at higher speeds It’s great to also follow along with transcripts and comb for insights to put on Post-Its for synthesis.
- Listening back to recordings and reading transcripts provides you with objective feedback, so you can improve your practice. It gives you the ability to pick up on your own mistakes in technique - whether that be talking over the participant, cutting them short, using leading questions, misinterpreting what they mean, or even noticing gaps in your understanding that you can follow up on. You can also compare your recording analysis to your notes. I can’t impress this on you enough: Without re-living your interviews, you can never improve as a researcher. Every interview recording is an opportunity to improve your research skills for those few times that you won’t be able to record. Proper analysis, reviewing recordings and transcripts allows you to re-live the interview without the distraction of thinking of the next question or referring to topic guides - so you have the opportunity to really listen. It also allows other people with other perspectives to weigh in on analysis and synthesis. Most importantly, it increases the likelihood that your findings will be accurate - and that you’ll be able to extract the real gold from the research - not just pre-conceptions, shallow findings or quotes. You need good data in to get reliable insights out. Notes are not enough.