Article Text
PDF
Abstract
Background Video recorded and in-person observations are methods of quality assessment and monitoring that have been employed in high risk industries. In the medical field, observations have been used to evaluate the quality and safety of various clinical processes. This review summarises studies utilising video recorded or in-person observations for assessing clinician performance in medicine and surgery.
Methods A search of MEDLINE (PubMed) was conducted using a combination of medical subject headings (MeSH) terms. Articles were included if they described the use of in-person or video recorded observations to assess clinician practices in three categories: (1) teamwork and communication between clinicians; (2) errors and weaknesses in practice; and (3) compliance and adherence to interventions or guidelines.
Results The initial search criteria returned 3215 studies, 223 of which were identified for full text review. A total of 69 studies were included in the final set of literature. Observations were most commonly used in data dense and high risk environments, such as the emergency department or operating room. The most common use was for assessing teamwork and communication factors.
Conclusions Observations are useful for the improvement of healthcare delivery through the identification of clinician lapses and weaknesses that affect quality and safety. Limitations of observations include the Hawthorne effect and the necessity of trained observers to capture and analyse the notes or videos. The comprehensive, subtle and sensitive information observations provided can supplement traditional quality assessment methods and inform targeted interventions to improve patient safety and the quality of care.
- Quality measurement
- Performance measures
- Human factors
- Patient safety
- Qualitative research
Statistics from Altmetric.com
Introduction
Observations have been used for quality assessment and improvement in non-medical fields, such as teaching and aviation.1 ,2 In medicine, low frequency preventable complications can have high costs. Observations are one of the most sensitive methods of error detection and can be used to evaluate individual performance and group care processes in settings ranging from the operating room to the pharmacy, informing changes in practice and future research.3 ,4 While patient outcomes are certainly affected by a physician's technical skills, organisational, situational and communication factors also play important roles.5 Therefore, assessments of quality in healthcare must reliably capture information in dynamic and complex care situations. Observations are valuable to identify conditions of risk involving human performance and organisational and cultural dynamics.
Observation in medicine is attentive watching of a clinical process by individuals not involved in the delivery of care being observed. Observations can be in-person (direct) or video recorded (indirect). In-person observations require a trained observer to enter a clinical setting and take notes on the process of interest (ie, length of postoperative handover or number of interruptions during surgery). Following the observation, the observer notes must be analysed and coded to provide qualitative and quantitative data on the clinical process. Observers must be trained in evaluation and analysis to give consistent and meaningful results. Video recorded observations generally use a mounted camcorder to capture the video and audio of a clinical situation. These videos can then be reviewed multiple times to analyse and code the data captured. Occasionally, observations utilise only audio recording if the process of interest is strictly verbal.
Observations have particularly valuable uses for quality and clinician performance assessment in clinical settings. Observations can be utilised to evaluate clinician teamwork and communication, identifying breakdowns in information transfer. Additionally, they can be used to detect clinical risks and medical errors, to quantify the occurrence and determine the most frequent types. Observations are also beneficial to assess compliance with guidelines and monitor quality improvement interventions.
Despite the documented success of observations as a method of quality assessment in clinical settings, findings have not yet been summarised comprehensively. We performed a literature review to summarise the use of observations for assessing clinician performance and quality of care in complex healthcare processes.
Methods
Search strategy
A literature search of MEDLINE (PubMed) was conducted on 1 May 2015. The search was conducted by pairing the medical subject headings (MeSH) term ‘observation’ individually with the MeSH terms ‘quality improvement’ (12 results), ‘medical errors’ (174), ‘process assessment’ (23), ‘communication’ (408), ‘patient safety’ (17), ‘risk assessment’ (177) and ‘quality assurance, healthcare’ (273), totalling 1084 articles. The MeSH term ‘videotape recording’ was then paired with the MeSH terms ‘quality improvement’ (4 results), ‘medical errors’ (432), ‘process assessment’ (8), ‘communication’ (1776), ‘patient safety’ (3), ‘risk assessment’ (60) and ‘quality assurance, healthcare’ (168), totalling 2451 articles. A total of 3535 articles resulted from the 14 searches.
Study inclusion and exclusion criteria
Peer reviewed studies available in English and published after 1990, the dawn of patient safety research, were included in the review. Commentaries, letters or comments on prior articles were excluded. Included studies used observations to investigate clinician performance in one of three ways: (1) assessing teamwork and communication; (2) assessing the frequency, cause or consequence of errors and weaknesses; or (3) assessing adherence to a quality improvement intervention or established guideline. Articles were excluded if ethnographic observations described the context or situation of a clinical event but did not directly assess clinician performance. Articles were also excluded if the observations focused on clinician–patient communication, were of simulated clinical practice, used for education or assessment of medical trainees, to test the reliability or validity of a tool, or were in an animal study.
The initial search returned 3535 articles. The searches with the MeSH term ‘observation’ returned 1084 studies, and the searches with video recording returned 2451. After removing duplicates, 3215 articles remained. Of the 3215 studies, 321 studies published prior to 1990 were excluded. Sixty-four articles not available in English were excluded, leaving 2830 for title and abstract review.
Two researchers independently reviewed the titles and abstracts based on the inclusion and exclusion criteria. After reconciling inclusion decisions, 223 articles were advanced to full text review. The researchers independently reviewed the full texts, and the first author made final inclusion and exclusion decisions. Details of excluded records can be found in figure 1. According to the criteria, 69 articles were included in the final review.
Search strategy.
The final set of studies was grouped into three categories of clinician performance assessment: (1) teamwork and communication assessment, (2) error and contributing factor detection and (3) intervention and guideline compliance evaluation. Details of individual studies can be found in table 1.
Details of the final set of studies, grouped into three categories of clinician performance assessment: (1) teamwork and communication assessment, (2) error and contributing factor detection and (3) intervention and guideline compliance evaluation
Results
Of the 69 studies reviewed, 16 used audio, video or combined audio-video recording, 51 used in-person observations and 2 used both methods. Observations were commonly used to assess error prone situations involving high risk to patients or high information transfer between clinicians. In the reviewed studies, the method of observation was used most frequently to assess performance in surgical procedures (16 studies), followed by medication administration (7 studies) and hand hygiene (6 studies). In these settings, observations were used mainly to identify clinician teamwork and communication patterns (27 studies), errors and contributing factors (18 studies), and intervention and guideline compliance evaluation (24 studies).
Teamwork and communication assessment
Several studies used observations to assess clinician communication and teamwork.6–32 Fairbanks et al6 described communication patterns and links in the emergency department, finding that the most common communications were face to face and that bedside and charge nurses play a central role in information sharing. Mackintosh et al observed delivery suites to identify the main mechanisms supporting team situation awareness. The whiteboard and delivery suite coordinator were important components supporting the team's decision making and performance.18
Multiple studies used observations to capture interruptions and multitasking, both of which can diminish the quality and safety of care. Using trained observers, Healey et al quantified distractions and interruptions during urological surgeries, finding an average interruption rate of 0.45/min. By far the most frequent source of distraction or interruption was conversations, followed by work environment problems, telephone calls and equipment. The least frequent sources were movement in front of video monitors and beepers.24 Gillespie et al used in-person observations of 160 procedures across 10 specialties to characterise intraoperative teamwork and interruptions. Approximately two-thirds of surgeries (107; 66.9%) were characterised by interruptions, with a mean of 2.3 (range 1–9) interruptions per procedure. Surgeons, the most frequently interrupted clinicians, were interrupted in over 60% of cases (98/160; 61.3%).23
Observations of surgical procedures can also help characterise communication patterns and detect threats to patient safety. Espin et al used observations to examine the nature and patterns of communication between perioperative nurses and surgeons, finding that issues of time were a catalyst for tension. Nurses were more likely than surgeons to use commands (14% of communications related to time vs 9%) and surgeons were more likely than nurses to ask questions (34% of communications related to time vs 7%).22 Lingard et al used observations to detect and categorise communication failures in the operating room. Approximately 30% of team communications were failures, classified as ‘occasion’ (poor timing), ‘content’ (missing information), ‘purpose’ (unresolved issues) and ‘audience’ (key individuals excluded). More importantly, over one-third (36.4%) of failures jeopardised patient safety by interrupting routine, increasing OR tension and increasing the cognitive load of clinicians.27
Error and contributing factor detection
Both in-person and video recorded observations have been utilised to detect errors in various clinical settings, such as surgical procedures, resuscitations, medication administration, nursing care, equipment usage and patient transfers.33–50 Tang et al used video recordings of laparoscopic surgeries to investigate the incidence of surgical errors. The study identified 2242 errors in 200 procedures, with a mean total error per case of 11. Using observations, the researchers were able to further classify the errors as consequential (30%) or inconsequential (70%).48 Tissot et al used trained observers to identify medication related errors and adverse events in geriatric and cardiothoracic surgery units. The observers noted nurses’ exact actions in the preparation and administration of medications. To identify errors, these notes were subsequently compared with original prescriptions, drug manufacturer recommendations, data in the literature and unit protocols. The medication administration error rate was 14.9%, with the most frequent errors related to dosing and wrong time administration (41% and 26%, respectively).37 Elpern et al used observations to investigate the incidence of errors in application of mechanical devices in intensive care units. The study found that 49% of observations involved an error in the application of intermittent pneumatic compression devices, with 15% of patients not receiving any intermittent pneumatic compression prophylaxis.33
Observations can also identify factors that can contribute to or help prevent medical errors. Arora identified stressful events that compromise patient safety in surgery and calculated the frequency by category. Of the 323 factors acting as potential stressors in 55 cases, technical problems, patient problems and equipment issues were the highest stress, highest frequency events.45 Rothschild et al used in-person observations to understand the role of the intensive care unit nurse in preventing errors from reaching and harming patients. Observations detected that nurses intercepted most errors (69%; 98/142) before they led to an adverse event. Additionally, nurses mitigated 13% of errors that reached patients before they resulted in harm and ameliorated 18% before more severe harm occurred.41
Intervention and guideline compliance evaluation
Several studies demonstrate the value of observations for designing and assessing the success of quality improvement interventions in improving clinician performance.51–63 Lyons et al first used handover observations to assess the relationship between handover setting/timing and quality (measured by clinical content scores). These observations established a positive correlation between handover time and clinical content scores, while also identifying that the shortest handovers occurred in the coffee room. This prospective observational assessment helped the development of a standardised handover protocol. The protocol was then implemented in an educational intervention that improved the clinical content scores of the trained doctors.56 Joy et al also used handover observations to design a handover protocol, resulting in improvement of information transfer to intensive care unit clinicians following cardiac surgery. The protocol implementation resulted in decreased technical errors (deviations from protocol involving personnel or equipment; 6.24 to 1.52, p<0.0001) and critical verbal information omissions (6.33 to 2.8) per handover, while also improving clinician perception of teamwork and communication.55
Observations have assisted the evaluation of compliance with clinical practice guidelines.53 ,64–74 Using a video recorder in the emergency department, Evanoff et al measured compliance with universal precautions for treating trauma patients. The observations found one or more major breaks in 191 of 304 invasive procedures. Clinicians failed to wear a mask in 32.2% of procedures and used inadequate eyewear in 22.2%.64 Adherence to a hand hygiene protocol was a frequent focus of observations. Chau et al69 used observations to investigate compliance with infection control practices among hospital staff and, after finding several specific breakdowns in compliance (gown usage, medical device disinfection), designed an intervention targeted at increasing compliance. Talbot et al conducted a study to measure the impact of a similar hand hygiene improvement initiative. The initiative, which focused on leadership buy-in, goal setting and financial incentives, improved hand hygiene adherence from 52% to 75%.53
Discussion
The existing literature on observations as a method for clinician assessment in healthcare demonstrates the value of observations as a quality assessment tool, especially for complex, dynamic and high risk processes. Observations have been effectively used to assess teamwork and communication, detect errors and contributing factors, and evaluate guideline compliance and the efficacy of interventions. Due to the resource requirements, observations may be most valuable and cost-effective in high risk or data dense clinical situations. In high information contexts, such as surgery and handovers, observations can capture information that traditional methods of assessment cannot. In high risk situations, such as resuscitations and medication administration, the resources invested in performing observations can be mitigated by the prevented harm to patients and the costs saved by avoided medical errors.
In data dense environments such as the emergency department, the reviewed studies have shown observations to be an effective assessment method of specific sub-processes, such as information transfer, communication patterns, distractions and teamwork. Because communication is a dynamic process with many components (verbal, eye contact, body language and gestures), observations can be a sensitive method for gaining awareness of subtle clinician behaviours. The studies effectively utilised observations to capture and understand elements of clinician communication performance, ranging from interruptions in handovers to tension in the operating room. Poor communication can have profound effects on patient outcomes and can incur tremendous costs over time, through wasted resources (eg, unnecessary tests, delay in interventions, incorrect interventions or missed diagnoses). Using observations to assess clinician communication can inform interventions to improve information transfer and team performance.
Observations are also a valuable tool for identifying errors in complex, high risk environments such as the operating room. The reviewed studies demonstrate how observations can be used to provide quantitative and qualitative information about medical errors–information that can be missed by more formal measurement methods.75 In a study comparing the error detection rate of video recorded observations to medical record review for trauma resuscitations, medical record review detected only 20% of the errors that were found with observations.76 With the potential for medical errors in any stage of care or clinical setting, the reviewed studies showed observations to be a versatile method of error detection in surgery, medication administration, application of mechanical devices, prescription filling, airway management and pain management.
Additionally, observations are effective in assessment of clinician compliance to established guidelines for practice and can identify specific deviations that may compromise patient safety. Multiple reviewed studies took advantage of guideline deviations exposed by observations to specifically target vulnerable clinical processes. Another usage of observations is to compare observational data before and after an intervention to demonstrate its effectiveness in improving quality of care. Initiatives to improve clinician performance in hand washing, handovers, intensive care unit teamwork, interdisciplinary rounds, medication rounds and trauma care were among some of the interventions successfully evaluated in the reviewed studies, using both in-person and video recorded observations.
Video recorded observations show several advantages over in-person observations.77–79 With video recorded observations, the physical presence of observers is not necessary and the records can be revisited at any time. This may increase the likelihood of capturing subtle cues, brief errors and communication patterns. Additionally, video recorders can be mounted so as not to get in the way of clinician processes.
Video recorded observations have the added benefit of providing direct feedback to clinicians on their practices. As practices over time become routine and repetitive, observations can restore situational awareness and help clinicians pinpoint their own strengths and weaknesses. By revealing clinicians’ habits and exposing discrepancies between intended purpose and actual practice, video feedback gives clinicians the capacity to intervene in their own behaviour.80 This practice has been used in schools to allow teachers to reflect on their interactions with students. Although teachers were reluctant to use observations for critiques, they ultimately found the process empowering.1 In Gawande's report on surgical observations for physician self-improvement in the New Yorker, he states, “No matter how well trained people are, few can sustain their best performance on their own”.81 Thus allowing clinicians to view their video recorded practices may be of help in sustaining best performances. Additionally, video recording can allow analysis of relationships and interactions without being impacted by the memory of an individual.76 ,82 ,83
Both types of observations are limited by observer variation and the time for training in-person observers and analysing field nodes and videos. Standardising observation notes and consistently coding data can present a challenge for accurately describing clinician performance. Additionally, observations of clinical practice call some ethical considerations into question. While the observers do not participate in the clinical process, if they witness a threat to patient safety, they may be ethically bound to intervene.84 Prior to the observations, researchers must consider the potential ethical issues that may arise in their particular clinical situation and decide how observers should react in each circumstance.
On the other hand, video recorded observations can be limited by the fixed position of the camera and the quality of the audio and video equipment used. As such, in-person observations can be preferable in some circumstances. No technical equipment is necessary and the full field of vision allows observers to notice events that happen outside the frame captured by the camera.
The Hawthorne effect, the influence of a subject's knowledge of observation on behaviour, is a factor to be considered in all types of observational studies. Under observation, clinicians can become more cognisant of their behaviours and may subsequently change them. Often, the Hawthorne effect improves performance and quality measures, as providers are aware that they are being judged and scored.85 ,86 When using observations to evaluate intervention effectiveness, the Hawthorne effect may exaggerate improvements. On the other hand, some clinicians may feel uncomfortable and their performance could be impaired.87 With a properly performed video recording, the risk of inducing a Hawthorne effect is low, especially during clinical crises when the focus is directed elsewhere.88 The Hawthorne effect presents a larger limitation during in-person observations, due to the visibility of the observers. Participants under observation may feel judged and inspected or become hostile and uncooperative. The researcher/observer can overcome these challenges by identifying with or gaining the trust of the group being studied. The observational study may also be limited if the observer begins to feel voyeuristic or exploitative.75
This review was limited by the restriction to articles available in PubMed and the choice of MeSH terms. In particular, usage of the MeSH term ‘videotape recording’ instead of the more recent MeSH term ‘video recording’ likely excluded several newer studies that utilised disks to capture videos of clinician performance. Another limitation of the review resulted from the priority to create a transparent, straightforward and replicable search. By strictly including studies returned by the 14 MeSH term searches, some relevant studies were excluded. One example is a study on surgical observations published by Birkmeyer et al89 in the New England Journal of Medicine. While the study satisfies the inclusion criteria and would have been included in the review, the article was not indexed by the MeSH term ‘observation’ or the term ‘videotape recording’ and thus was not returned by the search. Although other studies included in the review cover observations of surgical skill for various procedures, the particulars of the Birkmeyer et al study (eg, observations of bariatric surgery) were not among our described uses of observations. Without capturing all of the relevant studies, it is possible that the review overemphasises certain utilisations of observations without describing other valuable applications.
While not every published study on observations of clinicians is described, the included set of literature still demonstrates the value and versatility of this quality assessment method. The review aims to provide an overview of uses of observations and highlight the potential to describe clinician performance and inform improvement interventions.
Conclusion
Useful in a wide variety of clinical settings, observations are an informative and often cost-effective approach to assessing and improving quality of care, identifying shortcomings, devising interventions and sustaining clinical improvements. In-person and video recorded observations have demonstrated efficacy in quality assessment of clinician performance, specifically to evaluate teamwork and communication, medical errors, guideline compliance and intervention impact on performance. Observations can augment other methods of assessment and generate a comprehensive understanding of systems and processes in complex care environments. Even with secondary data of large patient cohorts, several important aspects of care, such as team communication or the impact of multitasking, cannot be captured. Thus observations should be considered for use when performing clinician quality assessments, in addition to more traditional methods.
References
Footnotes
Contributors AFY conducted the literature review, independently reviewed articles, and drafted and edited the manuscript. LMM generated the idea for review and search terms, independently reviewed articles for inclusion, and revised multiple drafts. ZAA revised multiple drafts and inserted the references. JH revised multiple drafts. DW revised multiple drafts. DPL revised multiple drafts and guided the direction of the manuscript.
Funding The study was funded by the Northwestern University Transplant Outcomes Research Collaborative (T32DK77662-7) and the Agency for Healthcare Research and Quality (5T32HS78-15).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.