University of Virginia Health System

14. General surgery morning report: a competency-based conference that enhances patient care and resident education; Brendon M. Stiles, T. Brett Reece, Traci L. Hedrick, Robert A. Garwood, Michael G. Hughes, Joseph J. Dubose, Hilary A. Sanfey, Reid B. Adams, Bruce D. Schirmer, Robert G. Sawyer (The Department of Surgery)

Hypothesis:  After moving to a night float consult system, our residency program initiated a daily morning report (MR) to discuss all new consults and admissions.  This is attended by all residents and students on three separate general surgery services and the transplant and trauma services.  The conference was originated to sign out the admissions and consults from the previous day to the services that would assume care. While initially oriented towards transfer of patient information, we hypothesize that MR now also serves as a core competency-based resident education tool. 

Research Design:  An anonymous survey was distributed to residents currently on-service.  Questions were asked regarding the value of the current MR, how it addresses the core competencies, and how it could be improved with regard to patient care and resident education.  Answers on a 5-point Likert scale included "strongly agree" (SA), "agree" (A), "don't know or neutral" (N), "disagree" (D), and "strongly disagree" (SD).  We asked respondents to rank the following conferences in terms of educational benefit derived:  MR, morbidity and mortality, grand rounds, and specialty conferences. 

Results:  The majority of residents (n=25) agreed that MR is an efficient method to sign- out patient care (84% SA or A) and that it provides an excellent educational experience (88% SA or A).  Importantly, they agreed that it is presented in an evidence-based format (88% SA or A).  Regarding the core competencies, residents all asserted that MR addresses "Patient care" (100% SA or A) and "Medical knowledge" (100% SA or A).  Most agreed that it addresses "Professionalism" (60% SA or A), "Interpersonal skills and communication" (76% SA or A), and "Practice based learning and improvement" (92% SA or A).  The four most important components identified with respect to continuing to improve both patient care and resident education were the presence of the on-call attending, a review of relevant radiology, provision of follow-up on select cases, and critical review of the literature.  On average, MR was seen as our most educational conference, with 52% of residents ranking it first.

Conclusions:  While MR is ubiquitous in most primary care residency programs, such a conference has not typically been held on surgical services and is not described in the surgical literature.  After adopting a night float system, we developed MR at our institution as a necessity for patient sign-out.  As this conference has continued to evolve, it has become an excellent tool for resident education.  It now serves the dual purpose of enhancing patient care and medical education.  Importantly, we are also using MR to provide evidence of learning and assessment of the core competencies.  This conference provides an example for program directors of how to tailor existing resident work sessions or conferences to meet ACGME competency requirements. 

Accepted for oral presentation at the Association of Program Directors in Surgery, Annual Meeting March, 2006 and for poster presentation at the ACGME Annual Meeting, March, 2006.

Back to top

15. The Pediatric Home Visit Program: Addressing the ACGME Competencies; Payne NJ, Waggoner-Fountain LA, University of Virginia School of Medicine, Department of Pediatrics, Charlottesville, VA

Hypothesis:  The Pediatric Home Visit Program provides a natural environment for the instruction and assessment of residents' clinical skills and professionalism.  As such, it is hypothesized that the program will prove useful in the assessment of the six core ACGME competencies of patient care, medical knowledge, interpersonal and communication skills, professionalism, practice based learning and improvement, and systems-based practice.

Research Questions: 

1-Which competencies do pediatric residents perform well or perform poorly?

2-Does level of resident training influence performance?

Research Design:  Four pediatric residents and one faculty member visit pediatric patients in their homes and school settings.   All members of the home visit team directly observe a rotating lead resident physician conduct a patient care visit.  Overall performance ratings, 360º evaluations (self, peer, faculty, and patient's family member), portfolios, skills checklists, and patient logs will be used to assimilate data to answer a number of research questions related to each of the six ACGME core competencies.  Qualitative data analysis coupled with statistical analysis was used for initial assessment of overall performance ratings and 360º evaluations. 

Results and conclusions:   All six core ACGME competencies can be addressed in the home visit setting.  Trends reveal that residents perform well overall.  Level of training may not be the strongest predictor of strong performance rating or achievement of the more complex competencies.  Future analysis will introduce a longitudinal component to the study and incorporate other assessment tools.  Ongoing analysis will provide more data to guide additional changes regarding the residency program's curriculum for teaching and evaluating the ACGME competencies.  

Funding Source:  Academy of Distinguished Educators Research Grant, awarded July 2004

Back to top

16. RESIDENT ATTITUDES TOWARD A SYSTEMS-BASED PATIENT SAFETY EDUCATION PROGRAM; NB MAY,  JD VOSS, JM SCHECTMAN, M PLEWS-OGAN, (University of Virginia, Charlottesville, Virginia).

RESEARCH QUESTION:  If the patient safety movement is to succeed, we must incorporate participatory training early in physicians' careers while simultaneously addressing the culture of safety and quality improvement.  This is proving to be a daunting task.  As part of an ongoing evaluation of our new residency patient safety curriculum, we began conducting annual interviews with residents to improve the curriculum and to better understand its impact on residents' practice of medicine.

RESEARCH DESIGN:  All 2nd-year medicine residents participate in an 8-session curriculum in systems-based practice and practice-based learning and improvement during their ambulatory rotations.  Three sessions are devoted to a hands-on patient safety experience that includes didactic instruction, individual investigations of a reported error or near-miss, and presentation of their findings to the clinic's Patient Safety Committee. All residents who completed the root cause analysis and investigation were contacted for the interview study.  Of the 15 eligible residents, 13 participated in confidential, in-depth interviews.  Each interview with Dr. May lasted between 40 and 70 minutes.  All interviews were audio taped, transcribed, and analyzed using an adaptation of Spradley's domain analysis (Spradley JP.  The Ethnographic Interview.  New York:  Holt, Rinehart and Winston, 1979).

RESULTS AND CONCLUSIONS:   Although residents were generally able to discuss patient safety from a systems rather than an individual blame-and-shame perspective, their actual behavior rarely incorporated the systems approach to patient safety.  Few residents completed incident reports before or after the training, although they did feel the training would be useful when they were in their own practices with more control over processes of care. The training was intended, in part, to empower residents to make change, yet it often resulted in frustration that change was neither easily initiated nor lasting.  Residents identified barriers to behavior change that included (1) the physicians' innate sense of individual responsibility to address the problem themselves; (2) flaws in the mechanics of the reporting system; (3) the belief that someone else, generally a nurse, will report the event; (4) fear that even an investigation of a near miss might reveal the residents' own mistakes; (5) the complexity of a systems approach  and (6) diminished willingness to participate in the safety process because their stay at the institution will be limited.

The study suggests that successful change implementation must be part of the educational experience if residents are to truly embrace the systems approach to patient safety as learners and physicians.  Culture change must continually be a focus of institutions committed to teaching and improving patient safety.  Resident physicians can be a valuable resource as we examine new curricular innovations to address patient safety and the new ACGME competency requirements.  We have used results from this study to improve the curriculum, adding a change implementation project for all learners.

FUNDING:  Support for this study came from two sources:  Academy of Distinguished Educators, and Innovations in Medical Education, Peggy Plews-Ogan, Principal Investigator. 

Back to top

17. Observation of Precepting Study; John Gazewood, MD, MSPH, Departmetnt of Family Medicine, Tovia Martirosian, SMD 2006

Background and research question: By 2030, there will be 70 million people over the age of 65, twice the number in 1999. Family physicians provide the majority of care to older patients in outpatient settings, and will continue to do so. It is thus imperative that family physicians are adequately trained to care for these patients. Unfortunately, recent surveys of graduating physicians indicate that they do not feel well prepared to care for the complicated older patient. In order to gain a better understanding of the current state of geriatric teaching in family medicine residency programs, we sought to describe precepting encounters in a University-based family medicine residency program.

Methods: We directly observed precepting encounters between residents and attending physicians, which all occurred in a central precepting room. Resident physicians presented findings from their initial patient assessment to the attending physician who answered or asked questions when appropriate. The observation of these interactions was performed by a medical student who sat in the precepting room and silently observed consecutive interactions. In addition to patient age, gender and visit type, the observer recorded the content of precepting interactions at 15-second intervals using a data collection instrument that contained 16 domains describing content of precepting encounters. Sample domains included discussion of history, exam, and functional status. The observer recorded whether a geriatric issue was discussed. Prior to beginning formal data collection, the observer and principal investigator reviewed 4 hours of video-taped precepting encounters until agreement was reached on coding. Total time for each domain for the resident and faculty were calculated. Descriptive statistics, chi-square test, t-tests and ANOVA were performed as appropriate using SPSS v. 10. For multiple comparisons, we set statistical significance at a level of p=0.01. The study was approved by the Human Investigation Committee and all participants provided informed consent.

Results: A total of 259 precepting encounters were observed, including 33 encounters with patients over age 64 years. The average patient age was 38, the median 36, with a range of 0 to 86 years, and 69 percent were women. Fifty-four percent of precepting encounters involved visits for acute care, 28 percent chronic care and percent health maintenance visits. Mean duration of all precepting encounters was 5.7 minutes, and an average of 2.5 conditions were discussed. Eighty-five percent of the 33 precepting encounters involving patients over the age of 65 included discussion of a geriatric issue. Fifteen percent of all precepting encounter time was spent discussing geriatric issues. Precepting encounters for patients over age 64 and those between 35 and 64 were of similar mean length (6.6 vs 6.2 minutes, p is NS), In precepting encounters of older patients, residents spent more time relating history (2.3 vs 1.7 minutes, p<0.01); more time was spent discussing functional issues (0.37 vs. 0.14 minutes, p<0.01); and more conditions were discussed (3.9 vs 2.9, p<0.01).

Discussion: This observational study shows that precepting encounters involving geriatric patients occur infrequently in this residency program. Residents and faculty do address geriatric issues in the large majority of these patients, and spend a significant proportion of the precepting encounter discussing geriatric issues. Strengths of this study include the use of direct observation to record in detail the content of precepting interactions. The principal limitation of this study is limited generalizibility, as it involved observation of precepting encounters in a single setting. We did not attempt to assess the quality of geriatric teaching. This study demonstrates that geriatric issues can be addressed during precepting encounters in a busy family medicine training program, and that the family practice center should be an important site of geriatric teaching.

Funding: Harrison Teaching Chair Funds.

Back to top

18. A Research Program to Increase Participation of Pediatric Residents in Scholarly Activities and to Improve Training in Practice-Based Learning Competencies; Martha Hellems MD, MS, Linda Waggoner-Fountain MD, Richard Stevenson MD (University of Virginia Department of Pediatrics).

Hypotheses:   A comprehensive pediatric resident research program can

1) increase resident participation in scholarly activities

2) enhance resident education in principles of evidence-based medicine, epidemiology, and skills necessary to conduct and present research projects

3) facilitate documentation of advanced proficiency in ACGME competencies in medical knowledge, practice-based learning and improvement, interpersonal and communication skills, and professionalism.

Description of Research Design: Exposure to research opportunities early in medical education may influence career choices towards academic general pediatrics or pediatric fellowship training.  More generally, research during medical training may foster an interest in scientific inquiry, enhance fluency in scholarly discourse, and teach crucial skills for life-long learning as a physician. Over the 2004-2005 academic year, the University of Virginia's Department of Pediatrics implemented a resident research program to enhance the education and preparation of pediatric residents for careers in academic medicine and in primary care, and to help achieve and document resident competencies in practice-based learning.

 The University of Virginia's Pediatric Resident Research Program was created by and is under the leadership of two Co-Directors for Resident Research.  The program is comprised of several complementary components.  The practicum component of the research program guides participants through the stages of implementation and completion of a mentored research project.  The didactic component includes a monthly journal club, and monthly small-group sessions, a series of noon conference lectures, and an annual research seminar.  Coordination of resources is critical to the success of a resident research program. Such resources include a pediatric IRB coordinator, a PhD statistician, literature-searching and computer skills training through the Health Sciences Library, lecture series through the fellowship and MD/PhD programs, a monthly newsletter, and an informational website. 

Results and Conclusions: In the first 18 months of the research program, program activities reached a large proportion of residents with 20-26 (of 34) residents and 2-6 faculty attending Journal Club each month, 4 interns and 1 senior resident attending each month's small-group session, and 17 residents from UVa and an associated pediatric residency program attending the annual Research Symposium seminar.  Thirteen residents initiated scholarly projects since the summer of 2004 and are currently in various stages of their work. While graduates of pediatric programs cite lack of time as one major barrier to conducting research during residency, only 8 of these residents chose to take protected elective time in which to conduct their research.  Research mentorship is concentrated within a subset of enthusiastic faculty:  nine faculty members serve as mentors for these projects, with one faculty mentoring 4 separate projects.  Two residents presented their research findings at national meetings.  Two have had papers accepted for publication.  One has been awarded a CATCH grant for her project.  One presented the results of her analysis to a local community group.  Skills and knowledge self-assessment and faculty evaluation tools have been developed for Journal Club.

On-going discussions with resident participants and research mentors will provide insight into their research experiences, individual goals, and potential barriers to participation which will aid us in further program development and outcome assessment.  It is unclear at this time if the resident work-hour limits enhance or undercut research participation. 

Source of funding:  University of Virginia Department of Pediatrics

Back to top

19. Impact of a Web-based Diagnosis Reminder System on Errors of Diagnosis; Stephen M Borowitz, M.D., Larissa R Amy, M.S., Jason A Lyman, M.D., Patrick A Brown, M.D. and Mark J Mendelsohn, M.D. (Department of Pediatrics and Department of Health Evaluation Sciences, University of Virginia, Charlottesville, VA).

Background: Errors of diagnosis account for 10-30% of medical errors. The most common cause of diagnostic error is failure to consider reasonable alternatives after an initial diagnosis has been reached (premature closure). A diagnosis reminder system can provide a checklist of possible diagnoses, complementing the initial differential diagnosis and lessening the risk of premature diagnostic closure. Isabel is a Web-based pediatric diagnosis reminder system. Using unstructured language, clinicians enter a list of clinical features and Isabel provides a list of relevant diagnoses. .

Objective: Determine whether the Isabel diagnosis reminder system reduces diagnostic omissions.

Design/Methods: Resident physicians were presented with a set of six simulated cases of differing difficulty. For each case, participants developed a list of likely diagnoses and an initial management plan before and after using the Isabel system. The quality of responses was compared to responses of a panel of three expert pediatric clinicians. Primary outcome measures were a change in the number of clinically important diagnoses included in the differential diagnosis, and a change in a previously validated diagnostic quality score (DQS).

Results: 16 pediatric, 7 family medicine, and 2 emergency medicine residents participated in the study. All 25 residents completed all six cases. In 15 of the 150 cases completed (10%), Isabel caused the user to include a major diagnosis they had not considered and should have. For each of the six cases, the mean diagnostic quality score increased significantly after residents consulted the Isabel system (0.028 + 0.049, 95% CI 0.020 - 0.036, p<0.001). Among residents at a comparable level of training, there was a statistically significant effect on DQS scores between pediatric and non-pediatric residents (0.36 + 0.01 vs 0.027 + 0.001, p<0.001). When they were surveyed, 60% of users agreed or strongly agreed that the Isabel system provided useful information and nearly 100% reported the system was easy or very easy to use.

Conclusions: By providing a checklist of likely diagnoses, a web-based diagnosis reminder system reduced diagnostic omissions by residents for a set of simulated cases. Incorporating this type of system into education and clinical practice may decrease the number of medical errors.

Source of funding:  None

Back to top 

20. Development of an Ideal Standard for Interpersonal and Communication Skills for Psychiatric Medicine Resident Physicians; J. Kim Penberthy, Ph.D., Edward M. Kantor, M.D., Zachariah C. Dameron, M.D., and Carolyn Runyon, B.A. (Department of Psychiatric Medicine, UVA Health System).

Introduction/Hypothesis:  Interpersonal and Communication Skills, one of the six General Competencies defined by the ACGME/ABMS are crucial to the education of residents, but are difficult to quantify, teach, and assess via traditional methodologies.  Furthermore, there are no standard set of principles of good interpersonal and communication skills by which to assess resident/patient interactions.  By using data collected over the past eighteen months, we propose to create a standardized profile of effective interpersonal communication style.  We believe that the residents who more stringently adhere to this proposed standard during the intake evaluation will also have higher patient satisfaction scores.

Research Design:  An interpersonal and communication skills training program has been implemented to psychiatric medicine residents, utilizing Contemporary Interpersonal Theory (CIT), and an objective measure of interpersonal communication style, the Impact Message Inventory (IMI).  The IMI data collected from the patients about their resident physician will be evaluated in several ways.  First, we computed the residents' behaviors (as determined by patient responses) complementarity to a standard set of good communication skills as defined by the literature.  Second, we compared the resident adherence to the patient satisfaction scores (ranked on a scale of 0 to 100) in order to determine the accuracy to the ideal interpersonal communication skills assessment rubric.

Results/Conclusion:  Preliminary results suggest that those residents who received the highest levels of patient satisfaction also exhibited the greatest complementarity to the ideal standard of interpersonal communication skills.  This data allows for the standardization of an interpersonal communication skills assessment, an important component of the ACGME/ABMS guidelines.

Funding Source:  Research funded by a UVA GME Innovations award.

Back to top

21. Measuring an "Intermediate Effect" in the NICU; Patrick Brown and Phillip Gordon (Dept. of Pediatrics), Stephanie Guerlain and David Bauer (Systems and Information Engineering).

Background: Preliminary studies of physicians' information-seeking behaviors in an academic NICU indicate that physicians use the bedside flowsheet to gather much of their information about patients in this setting.  Despite the complexity of the interpreting data from the flowsheet, residents are not usually taught how to "read" the flowsheet in any formalized way.  Instead, residents learn to interpret data from the flowsheet informally by modeling the behaviors of more senior colleagues.  Since the flowsheet has never been studied from an educational perspective, it is unclear how quickly this competence is achieved, whether this skill is achieved in a purely linear fashion, or how much exposure is required in order to achieve a minimum level of competency in this skill.

Objective: To assess the relationship between NICU experience and residents' abilities to correctly interpret patient data from the bedside flowsheet.

Design/Methods: 5 hypothetical cases were prepared by two pediatric faculty members.  The cases represented 3 common and 2 less common diagnostic scenarios in the NICU.  The data for each case was presented on a standardized flowsheet - a sheet of paper with 81 data elements presented over an 8-hour time period.  After validating the cases with NICU faculty and fellows, the cases were presented to 37 pediatric residents.  Performance was based on residents' ability to select the correct diagnosis for each case as well as by identifying the items from the flowsheet that residents considered relevant for a particular case (a measure of diagnostic reasoning).  Residents' NICU experience was assessed by reviewing the call schedules for the past 3 academic years.

Results: Results revealed no relationship between a resident's level of experience in the NICU and their ability to correctly diagnosis a particular case.  Regardless of NICU experience, most residents correctly identified the 3 common cases but missed the less common diagnoses.  Assessment of diagnostic reasoning revealed an apparent "dip" in performance around 4-6 weeks of NICU experience.  This trend was demonstrated whether or not a resident achieved the correct diagnosis.  In other words, residents at intermediate levels of training seem to have more trouble identifying clinically relevant information from the flowsheet than novices (residents with no NICU experience) and residents with more than 15 weeks of experience.  For reasons that are not clear, performance seems to wane again after 19-20 weeks of experience.

Conclusions:  Decreases in performance among intermediates has been described in previous studies of novice to expert development.  This so-called "intermediate effect" is posited to occur as individuals gain more knowledge about a particular domain.  During this period intermediates are able to consider more alternatives than novices,  but lack the ability to process their knowledge as efficiently as experts.  The results of this study are limited by the small sample size.  However, the results suggest that the manner in which residents learn to interpret data from the NICU flowsheet is not linear.  This conclusion has implications for resident education as well as the idea of competency-based benchmarks.  With respect to education, it would be important to know whether educational modules could alter the slope of the current learning curve or if the intermediate effect is a necessary part of novice to expert development.  On the subject of competency-based testing, it will be important to understand what the learning curve of a particular task looks like prior to developing a competency measure.  Depending on the sensitivity of a particular test, novices may outperform more experienced residents without necessarily having true competence in a particular domain.  We are in the process of testing pediatric residents at other academic centers to establish the validity of these findings.

Funded by: GME Innovation Award and a grant from the Academy of Distinguished Educators.

Back to top 

22. Resident Sign-Out - A Precarious Exchange of Critical Information in a Fast Paced World; Stephen M Borowitz, M.D., Linda A Waggoner-Fountain, M.D. (Department of Pediatrics), Ellen J Bass, Ph.D. and Richard M Sledd, B.S. (School of Engineering, University of Virginia, Charlottesville, VA ).

Background: During sign-out, information, responsibility, and authority are transferred from one set of caregivers to another. As a result of resident duty hour restrictions, the number of sign-outs has increased. Nevertheless, sign-out generally remains an informal and unstructured process without standardization of the information exchanged. Few residencies formally teach residents how to sign-out or assess a resident s ability to sign-out. Despite its critical importance, little research has been performed examining the sign-out process.

Objective: Characterize the sign-out process on an acute care ward.

Design/Methods: Residents rotating on an acute care ward participated in a prospective study. Immediately following call nights, they completed a confidential survey characterizing their night on call, the adequacy of the sign-out they received, and where they went to get information they did not receive during sign-out.

Results: On 18 of 54 surveys (33%), residents indicated something happened while on call they were not adequately prepared for. For 13 of these 18 instances (72%), they indicated there was information they did not receive during sign-out that would have been helpful, and in 11 of the 18 instances (61%), they indicated the situation should have been anticipated and discussed during sign-out. When nights when something happened the resident was not adequately prepared for were compared to nights they felt adequately prepared, the quality of sign out assessed with a 5 point Likert scale (1 = ‘inadequate to answer call questions" to 5 = "adequate to answer call questions") was significantly different (3.67 + 1.03 vs 4.22 + 0.80, p=.03). There were no differences in how busy the nights were as assessed with a 5 point Likert scale (1 = slow to 5 = busy) (3.17 + 0.86 vs 2.76 + 1.09, p=0.17), patients on service at the beginning of call (14.22 + 4.57 vs 14.79 + 4.33, p=0.66), number of admits (4.47 + 2.03 vs 4.56 + 2.32, p=0.89), number of transfers to an ICU (p=0.25), or whether residents were "cross covering" (p=0.16).

Conclusions: Although residents frequently sign-out to one another, there are many times when important information is not transmitted. Analysis of these "missed opportunities" can be used as part of an educational program teaching residents how to sign-out.

Source of Funding:  Buchanon Graduate Medical Education Innovation Grant, University of Virginia Health Sciences Center and Grant from Association of Pediatric Program Directors

Back to top 

23. Preliminary Evaluation of a Web-Based Tool to Support Housestaff Competency in Practice-Based Learning and Improvement; J Lyman*, J Schorling**, N May**, K Scully*, N Sarafian*, M Nadkarni**, J Voss** (*Dept. of Public Health Sciences, **Dept. of Internal Medicine, University of Virginia School of Medicine).

Objectives: We developed Systems and Practice Analysis for Resident Competencies (SPARC), a WWW-based tool to support teaching and assessing the practice-based learning and improvement (PBLI) ACGME competencies. SPARC allows housestaff in the Department of Medicine to explore de-identified, population-based data about their patient panels with peer comparisons.  Reports describe demographic and clinical characteristics of resident patient panels and specific disease screening and management rates.  SPARC has been integrated into the existing curriculum for residents to identify areas for personal quality improvement projects.  While a comprehensive evaluation of SPARC is underway, we report here our preliminary results of a pre-post assessment of residents' self-rated competencies related to PBLI, conducted during an ambulatory rotation with 2nd and 3rd year Medicine residents.

Methods: Two faculty members (JV, JL) developed a 23 item instrument to assess self-rated competency in PBLI based on previously identified learning objectives.  Residents assess their ability to perform general PBLI tasks on a 5 point scale ranging from novice to master. Learners also indicate their agreement with statements relating to specific PBLI tasks using a 5 point Likert scale.  The survey was administered immediately before using SPARC and two weeks after using SPARC to explore their panel data and presenting plans for personal quality improvement projects. 

Results and Conclusion: In our first month of use 7 2nd year and 2 3rd year residents used SPARC and completed our instrument. We used a distribution free test (sign test) to evaluate how often scores within subjects improved after using SPARC.  In all assessment areas, both mean and median scores improved, and in all 23 questions the improvement noted was statistically significant with p values of .004 to .031. Table 1 shows the response distribution for each category "pre"(x) and "post" (o) for a sample of questions. For example, one respondent indicated a "novice" level in the post assessment for their ability to compare their practice with a larger population. In Table 2, the frequency of "agree" responses for a sample of survey questions are shown before and after using SPARC.

Table 1: Response Distribution for Self-Rated Ability of General PBLI Tasks, Pre (x) and Post (o)

Table 2: Frequency of "Agree" Responses for Specific PBLI Tasks, Pre vs. Post

Our preliminary results suggest that SPARC is a valuable resource for helping residents to learn about PBLI, but interpretation is limited by our sample size as well as self-assessment bias. We plan to study SPARC for a cohort of 60 residents this spring using a comprehensive evaluation that includes both faculty and housestaff perceptions of the usefulness of SPARC for teaching and assessing PBLI competency as well as faculty assessment of resident competency in this important area.

Source of Funding: Development and evaluation of SPARC were supported through a grant from HRSA for which John Schorling, MD, MPH is the principal investigator.

Back to top 

24.  ASSESSING AND IMPROVING KNOWLEDGE IN CYTOPATHOLOGY TRAINING; W K Brix and  K A Atkins (Department of Pathology) and J Jackson (Office of Medical Education).

Background: Knowledge based assessment is one of the six domains of competency for residency training developed by the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project.  In response to these mandates, The American Society of Cytopathology and Association of Directors of Anatomic and Surgical Pathology have outlined expectations for fund of knowledge for cytology at three skill levels.

Objectively assessing fund of knowledge and expansion of knowledge base is challenging secondary to: 1) lack of available universal testing (none available aside from the annual Residency In- Service Exam); 2) the large breadth of information; 3) the lack of continuity between the faculty and residents. Currently UVA resident knowledge and diagnostic ability are graded by faculty on a 5 point scale.

This study is designed to: correlate faculty assessment of resident knowledge with objective test scores, evaluate the utility of a mid-rotation test and a mandatory study slide set, and to provide an objective means to comply with the mandates set by the ACGME.

Design: A web based test of 26 questions was developed for a beginning skill level residents (first 2 months of cytopathology training), covering general topics in medical, gynecologic, and fine needle aspiration cytology. The test was composed of multiple choice questions similar to the anatomic pathology boards. 

Eleven residents rotating for the first 2 months on cytology have participated in the study. Five residents took the end of the rotation test (ERT) after rotating as usual on the service. Six residents were given a mid rotation test (MRT) and a mandatory study slide set composed of 20 commonly encountered lesions. They also took the ERT. Faculty evaluations were recorded and compared to the test results.

Results: The average score on the end of the rotation test for residents with no MRT was 54%, while the average score for residents with MRT was 77% (a difference of 6 questions). The average score for the mid rotation test was 50% for both groups. Faculty evaluations ranged from 3 to 5 for all residents.

Conclusions: 

• In this preliminary study, a mid rotation test and mandatory slide study set improved performance on the ERT by an average of 23%. Although the numbers are too small to establish a causal relationship, they are encouraging regarding the intervention.
• Subjectively, residents report that they were more attentive to discussions involving topics they had seen on the MRT.
• There was little correlation between faculty evaluation and test performance. The results do not necessarily suggest that one evaluation method is superior; rather, they highlight the utility of triangulating evaluations.

Future directions:

• Increasing web based exams and training modules

As more residents take the exam, the questions will be analyzed for discrimination of knowledge base and level of difficulty. It is anticipated that a series of quizzes will be available for all trainee levels.  Residents will progress through the on-line exams and slide sets at their individual pace. Residents will progress to the next skill level when they have achieved 70% or better on the on-line exam and slide sets.  It is anticipated that this format will provide an objective means of assessing resident knowledge as well as highlight topics the faculty need to emphasize.

Incorporating OB/Gyn residents into the study.  A training module and a delayed follow up exam regarding the interpretation and management of cervical Pap smears and biopsies will be developed.  A training group and control group will be evaluated.

Funding: The "future directions" are being funded by a grant from the UVA GME office.

Back to top 

25. Enhancement of Resident Education and Clinical Efficiency through Workflow Analysis; Stephen M. Borowitz, M.D. and Linda A Waggoner-Fountain, M.D. (Department of Pediatrics).

Background: Duty hour restrictions, night floats, and hospitalists have all changed the care of hospitalized children and yet the teaching structure of resident education on pediatric inpatient wards has changed little in most academic hospitals. Little research has focused on how residents spend their time in the hospital, or, how to make that time most productive for education and patient care.

Objectives: 1. explicitly define resident workflow on the inpatient wards, 2. characterize the objectives of each activity in the workflow, 3. Develop a new workflow that promotes clinical efficiency and educational activities.

Design/Methods: During facilitated sessions, the pediatric housestaff, program directors, selected faculty, and a workflow analyst mapped resident workflow. During the discussion, workflow was graphically represented with flow-charting software. Once the workflow had been agreed upon, educational and patient care objectives of each activity were explicitly delineated. A small group developed a straw model of a new streamlined workflow that accomplishes all major educational and patient care objectives. During additional facilitated sessions, the straw model was reviewed and refined.

Results: At baseline, resident workflow was very disjointed. Many activities were redundant and others did not accomplish stated objectives. Contact with consultants was sporadic and bedside teaching opportunities were frequently lost. Housestaff were often paged out of teaching rounds. There were no extended stretches of time dedicated to patient care. The revised workflow combined several didactic conferences and rounding activities and provided housestaff with a defined daily schedule. Implementation of the new workflow has significantly increased resident satisfaction with their inpatient ward experience. The changes promote face-to-face contact with consultants, therapists, and nurses who all participate more regularly in management discussions and bedside teaching activities.

Conclusions: As the complexity of caring for hospitalized children increases, activities are added to the residents' workday that can result in convoluted processes that do not maximize resident learning experiences or their ability to efficiently care for patients. Process flow analysis and work redesign can improve the clinical efficiency of residents and their educational experience on the inpatient wards.

Source of Funding:  None

Back to top