Naturalistic Decision Making

Naturalistic Decision Making

By Brian Moon and Laura Militello
Brian Moon and Laura Militello interview leading NDM researchers, who study and support people who make decisions under stress.
Where to listen
Breaker Logo
Google Podcasts Logo
Overcast Logo
Pocket Casts Logo
RadioPublic Logo
Spotify Logo
Episode #6: Interview with Julie Gore
Date: 5/15/2020 Show Description: “Today we welcome Julie Gore. Julie Gore is a Reader in Organizational Psychology, at the School of Management, University of Bath in the UK. She is a Chartered Psychologist and Fellow of the British Psychological Society. Her research focus is on the psychology of expertise and Naturalistic Decision Making (NDM) across a range of professions working under uncertainty. She is Associate Editor for Journal of Occupational and Organizational Psychology and serves on the boards of the British Journal of Management and Frontiers in Organizational Psychology. Julie is also an academic advisor for Nasdaq’s Behavioural Science Lab. Dr Gore received one of the world’s first NDM PhD degrees in Applied Cognitive Psychology from Oxford Brookes University, UK.” Where to find Julie’s work: The Oxford Handbook of Expertise  The University of Bath  Naturalistic Decision Making and Uncertainty Naturalistic decision making: navigating uncertainty in complex sociotechnical work SPRITE+  Learn more about NDM: NaturalisticDecisionMaking.org Journal of Cognitive Engineering and Decision Making Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Timestamps: Experience as one of the first NDM PhD program participants [1:46] How Julie initially heard about NDM [3:03] Julie's PhD research question [3:55] Discoveries made during PhD research [5:18] Advice for young NDM PhD students [6:15] Common barriers students need help managing [8:02] Surprises and concerns about today's generation of NDM researchers [10:10] Examples of PhD students whose perspectives have been transformed by NDM models [12:05] Making space for NDM research papers in journals and publications [14:52] Nature of Julie's work with NASDAQ [16:32] Exciting research on decision-making ethics and accountability in the digital space [19:23] Organizations' and outsiders’ attitudes toward NDM [23:17] Showcasing the utility of NDM methods [25:18] A particularly rewarding project [27:10] An explanation of "adaptive expertise" [29:12] Opportunities to advance NDM in the context of today's novel circumstances [32:37] Three people who influenced Julie's career [33:47] Influences from outside NDM [34:43] Changes in the NDM community [36:35] One question that can determine if someone is an NDM researcher [37:30] Directions for future research [39:37] Two truths and a lie [41:02]
44:08
May 24, 2020
Episode #5: Interview with Robert Hutton
Date: 5/7/2020 Show Description: “Today we welcome our friend, Rob Hutton. Rob is co-founder and director of Trimetis Ltd, a four-person cognitive systems engineering consultancy in Bristol, England. Rob and I started our careers together as young research assistants at Klein Associates in the early 1990s. His thinking contributed to early cognitive task analysis methods and he helped articulate models of macrocognition. Rob is a Fellow of the Chartered Institute of Ergonomics and Human Factors. He is also a part-time lecturer in Nottingham Trent University’s psychology department.” Where to find Robert: Trimetis Nottingham Trent University Learn more about NDM: NaturalisticDecisionMaking.org Journal of Cognitive Engineering and Decision Making Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Timestamps: Starting an NDM business in England [1:30] Types of client organizations [3:30] A project that Rob is excited to be working on [6:05] Challenges conveying the value of NDM theory to students [8:45] What’s the craziest situation you’ve found yourself in, in the name of science? [16:40] Challenges of cognitive task analysis field research [23:55] A rewarding impactful project [25:10] Success of Co-authored Applied Cognitive Task Analysis paper [29:10] What is the most valuable proposition that you offer your customers? [29:50] Cultural differences of NDM research in the US vs. the UK [33:05] Who are three people who have influenced your approach? [41:40] What question would you ask someone to determine whether they practice NDM? [46:30] Two truths and a lie [50:00]
56:18
May 19, 2020
Episode #4: Interview with Jennifer Phillips
Date: 5/1/2020 Show Description: “Today we welcome our pal, Jennifer Phillips. Jenni is the CEO of the Cognitive Performance Group, which she cofounded in 2006. Her work focuses expert decision-making, primarily in the military training community. She and her colleagues at CPG have pioneered research into the development of skilled performance, and designed innovation in the areas of decision-centered training and assessment, including the development and application of mastery models. Among her many publications is a chapter in the Oxford Handbook of Expertise exploring mastery models and their value in supporting the acquisition and assessment of expertise.” Learn More About Jenni: Cognitive Performance Group Developing Mastery Models to Support the Acquisition and Assessment of Expertise Learn more about NDM: NaturalisticDecisionMaking.org Journal of Cognitive Engineering and Decision Making Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Timestamps: Jenny describes her focus of expertise, including tactical thinking in the marine corps and mastery models [1:30] Jenni breaks down the term “mastery model” and its stages [3:45] Jenni gives a mastery model example from her work with marine corps instructors [6:05] Work on assessment of decision skills, performance rubrics, and the process of adaptability [9:50] Major influences in Jenni’s career in NDM [14:05] Other people and organizations outside the NDM community who have impacted Jenni’s work [16:30] What makes Jenni’s approach and expertise unique in her field [19:15] Challenges and nuances of working in a military environment [21:30] Difficulty getting buy-in to conduct research in law enforcement [23:45] A favorite anecdote from research in IED defeat work [24:45] Some distinct and difficult nuances of IED defeat work in Afghanistan and Iraq  [29:45] Meaning and application of “cognitive fidelity” [32:50] A surprising characteristic of young researchers [35:20] One single question that can determine if someone is an NDM practitioner [39:05] Two truths and a lie [39:45] Excitement about research on making mastery models work at a large scale [42:10]
47:00
May 8, 2020
Episode #3: Interview with Emilie Roth
Date: 4/24/2020 Show Description: “Today we are talking with our friend and colleague, Emilie Roth. I first met Emilie at the second Naturalistic Decision Making meeting in Dayton, OH in 1994, but she had been studying how people manage complexity long before that meeting. In fact, I remember at that meeting thinking: ‘I want to be like her.’ Emilie is one of the people that inspired me to think I really could be a scientist. And I know Emilie has inspired many others over the course of her career. She is one of the architects of the cognitive systems engineering movement. She had an important role in redesigning nuclear power plant control rooms in the wake of Three Mile Island. And that was just the beginning; her contributions have been felt far and wide. She now lives in Palo Alto, CA where she runs Roth Cognitive Engineering.” Where to find Emily: Roth Cognitive Engineering Resources: Oxford Handbook of Expertise Designing collaborative planning systems: Putting Joint Cognitive Systems Principles to Practice Cognitive work analysis Facets of complexity in situated work Learn more about NDM: NaturalisticDecisionMaking.org Journal of Cognitive Engineering and Decision Making Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Timestamps: Emily's first published paper [1:30] Events that shifted Emilie’s focus from experimental psychology to NDM [7:30] Developing the observation tools and methods for developing systems of expertise [12:25] Challenges in creating expert decision-making models for nuclear operations at Westinghouse [17:15] Experience navigating a world of engineers as a female psychologist [20:00] The benefits of work culture at Westinghouse [21:45] Observations of railroad industry culture, practices, and informal procedures [23:00] Optimistic outlook on specific research and applications of NDM [28:45] Two things about Emilie’s personal life that others might not know [34:20] Three books that influenced Emilie’s work [38:00] Associative thinking exercise [44:15] Elaborating on automation pros and cons [45:25]
48:02
May 6, 2020
Episode #2: Interview with Robert Hoffman
Show Description: “Robert is a luminary in the NDM community, a widely recognized world leader in cognitive systems engineering and Human-Centered Computing, and a senior member and fellow of numerous international science and engineering societies. Since 1999, he has called the Institute for Human and Machine Cognition his organizational home, and he organized the NDM 6 conference in 2003 in his hometown of Pensacola, FL. He has co-authored and co-edited 18 scholarly books – including two with your hosts! – and is co-author on over 100 publications in peer-reviewed journals. For twenty years, he served as Co-Editor for the Department on Human-Centered Computing in IEEE: Intelligent Systems. He was a co-founder of The Journal of Cognitive Engineering and Decision Making, a major publication outlet for NDM research.” Where to find Robert: Institute for Human and Machine Cognition Books by Robert: Minding the weather: How expert forecasters think.  Cognitive Systems Engineering: The Future for a Changing World.  Perspectives on Cognitive Task Analysis: Historical Origins and Modern Communities of Practice. Applied concept mapping: Capturing, analyzing, and organizing knowledge. Learn more about NDM: NaturalisticDecisionMaking.org Journal of Cognitive Engineering and Decision Making Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Timestamps: Robert's first published paper [1:45] Event’s that led to Robert’s interest in NDM [5:15] Robert’s shift from psychometrics [14:00] NDM research project that Robert is most excited about [17:00] The falsifiability of NDM and research limitations surrounding the coronavirus [22:40] Shortcomings of graduate psychology and computer programs [25:30] Advice for people interested in exploring the NDM field [27:45] Two experiences and the lessons they imparted about eliciting expert knowledge [29:45] Some things about Robert that others might not know [35:30] Reasons why Robert loves football so much [37:45] Athletes and cognitive task analysis [40:17] Word association exercise [42:00]
44:53
April 28, 2020
Episode #1: Interview with NDM Pioneer Gary Klein
Show Description: “Today we are talking with my friend and business partner, Gary Klein. Gary is one of the pioneers of the Naturalistic Decision Making movement. He is known for moving the study of decision making out of the laboratory and into the wild. In 1989, he invited a small group of international researchers to meet in Dayton, OH to talk about decision making in natural environments and how to study this phenomenon. It was that group of 32 researchers who came up with the term Naturalistic Decision Making. Gary's many contributions include the recognition-primed decision model that has greatly influenced how we all think about decision making and expertise. He and his colleagues developed early cognitive task analysis methods that have been a core set of tools for studying decision making in real world environments. He has mentored countless researchers, including both of us. Gary now lives in Boston, MA. He runs a company called ShadowBox LLC, and is also part-owner and advisor to Applied Decision Science, LLC.” Where to find Gary ShadowBox LLC Website LinkedIn Psychology Today Column Learn more about NDM: Website: https://naturalisticdecisionmaking.org/ Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Resources mentioned during the show: Sources of Power by Gary Klein Seeing What Others Don't: The Remarkable Ways We Gain Insights by Gary Klein Timestamps: Gary's first published paper [1:36] How Gary got started in Naturalistic Decision Making [4:48] Gary's research on Israeli aircraft maintainers' coping methods during Yom Kippur War [7:55] Current NDM-related research that Gary is most excited about [12:10] Two of the biggest mistakes of Gary’s career and how he recovered from them [14:53] Gary shares his favorite research stores: -A firefighter whose quick thinking saved several of his colleague’s lives [22:53] -A British naval officer whose quick, life-saving decision was at first mistaken for ESP [28:00] -A neonatal nurse's expert instincts helped save a baby's life [33:33] Suggestions for how the NDM community can help during today's coronavirus crisis 37:15] Where Gary would prioritize funding and research investment right now [44:05] Associative thinking exercise [46:18] How the scientific method is “often a trap” that can limit researchers' learning opportunities [48:55]
53:47
April 20, 2020