Abstract
Objective To explore the use of Web-based resident practice profiles (RPPs) as a means of tracking the clinical experiences of residents to ensure an adequate educational experience.
Design Quantitative analysis of recorded patient encounters with residents.
Setting The Department of Family and Community Medicine at St Michael’s Hospital in Toronto, Ont.
Participants Twenty-seven residents enrolled in the department’s training program between July 1, 2006, and June 30, 2007.
Main outcome measures The clinical experiences of residents with respect to patient demographic information, procedures performed, and diagnoses. Resident data were stratified by age, sex, training status, and source of medical degree, and RPPs were compared with patient profiles of physicians at the study site, at the university, and in provincial practices.
Results A total of 9108 patient visits were recorded by the 27 residents during the academic year. Patient visit characteristics were very similar across all the resident variables except with respect to sex. The top 8 diagnoses encountered by residents were very similar to those of the comparison groups; anxiety or neurosis was the most common problem. Injections and Papanicolaou smears were the most common procedures, with 17.9 and 11.6 procedures, respectively, performed on average per resident during the study period.
Conclusion The RPP is an excellent Web-based tool to capture the clinical experience of postgraduate trainees. The practice profiles of the resident group were very similar to those of physicians in the study site, the university, and the province, demonstrating that common diagnoses made in practice correlate well with the clinical experience in residency.
Clinical exposure forms the basis of postgraduate medical education. Therefore, it is important for training programs to ensure that residents have a sufficiently rich clinical experience to deal with most of the problems they are likely to encounter in practice.1,2
Ambulatory care training has become standard in primary care physician education3–6; however, effective monitoring of residents’ experiences with ambulatory care has posed serious challenges to training programs.2 It has been noted that the setting of training programs influences the types of patients and problems encountered by the physician-in-training.7
Several studies have raised concerns about whether the quantity and type of clinical experience in residency adequately prepare trainees for practice. One study assessed the clinical experience of emergency medicine residents who were followed over a period of 9 months and 2152 shifts. Residents did not encounter more than 50% of the diagnoses required by training guidelines in sufficient frequencies; 22.7% of required diagnoses never presented at all.8 The clinical experiences of pediatric residents rotating through a pediatric emergency department similarly did not offer sufficient clinical exposure to several clinical problems for residents to achieve competence.9 The experiences of 8 general internal medicine residents also varied with respect to age, sex, and distribution of diagnoses encountered in continuity clinics.10
The resident practice profile (RPP), “a compilation of information allowing definition and evaluation of several parameters of health care delivery,”11 is a potentially useful tool to determine the breadth of clinical experience in residency training. The RPP allows for identification of practice deficiencies, which can guide reassignment of patients among residents11 and allow for more informed selective and elective learning experiences during the residency program.
This paper describes the results of a live ongoing intrahospital Web-based program to monitor the ambulatory care experiences of residents, including procedures performed, in a large residency program based in the Department of Family and Community Medicine at St Michael’s Hospital in Toronto, Ont. In order to assess whether our residents were being adequately prepared for future practice, we also compared the most common diagnoses encountered by residents with the diagnostic profiles of physicians at our site, elsewhere within the university, and throughout the province.
METHODS
Setting
The Department of Family and Community Medicine at St Michael’s Hospital, a fully affiliated teaching institution within the University of Toronto, comprises 4 distinct geographic sites in the inner city. Each site has a unique patient profile determined by the geographic location and the expertise of the faculty. In a given academic year, the department is the training division for 24 to 32 residents who are assigned to a site for the entire 2-year residency. In addition to 8 months spent in family medicine ambulatory clinics during the residency, all residents return for a continuity clinic on a weekly basis to provide ongoing care to the patients in their practices. Owing to the decentralized nature of the program and the differences in the patient populations served by each site, the RPP system was developed to capture the ambulatory care experiences of each resident and ensure that each resident was meeting the learning objectives of the program.
Data collection
The RPP Web-based system can be accessed at St Michael’s Hospital and all ambulatory clinics associated with the hospital. The program is password protected and is accessible only to the residents, program director, and program administrative assistant. Residents are expected to complete their records of patient activity during all their ambulatory care clinics in family medicine, either after each patient encounter or at the end of the day.
Data for this study were generated during one academic year from July 1, 2006, to June 30, 2007. All residents were expected to code each patient visit according to patient age, sex, procedures performed, and general system-based and specific diagnoses. The specific diagnoses were based on the College of Family Physicians of Canada’s Priority Topics and Key Features for Assessment in Family Medicine.12 Residents were able to enter up to 4 diagnoses and multiple procedures per patient visit.
Data analysis
Simple descriptive statistics were calculated based on frequencies of anonymous resident and patient visit–related data generated from the RPP system. Frequencies and percentages were generated to describe the resident cohort in terms of age, sex, residency training status (first or second year), and medical school attended (Canadian vs international); cross-tabulations were performed to describe patient age and sex by resident characteristics. Frequencies and percentages were also calculated to describe resident experience in terms of general system-based and specific diagnoses and of procedures performed.
The most common specific diagnoses recorded by residents were mapped to Ontario’s most common diagnostic codes and compared with primary care physician visits between April 1, 2006, and March 31, 2007, at St Michael’s Hospital, at other Toronto academic sites, and for all of Ontario. Comparisons were made with 113 538 patients at 11 University of Toronto–affiliated academic sites and 1 841 670 patients belonging to blended capitation models in Ontario. These comparison data were derived from the physician billings database from the Ontario Health Insurance Plan (OHIP) and obtained through a comprehensive research agreement with the Ontario Ministry of Health and Long-Term Care. The OHIP covers physician and hospital services for Ontario residents and includes approximately 94% of ambulatory physician visits in the province. The study was approved by the Research Ethics Board of St Michael’s Hospital in Toronto, and analysis of Toronto and Ontario comparison data was approved at Sunnybrook Health Sciences Centre in Toronto.
RESULTS
Between July 1, 2006, and June 30, 2007, a total of 9108 patient visits were recorded by 27 residents; the median number of visits per resident was 350. Table 1 describes the characteristics of the resident cohort in terms of sex, age, and type of medical degree (Canadian or international). Most residents were women (70.4%), in their first year of residency training (59.3%), and graduates of a Canadian medical school (81.5%). The median resident age was 27 years.
Resident characteristics
Characteristics of all patients stratified by residents’ sex, age, and training status are presented in Table 2. Patients’ characteristics were very similar across all the resident variables except for sex. Male residents had a greater proportion of visits by male patients compared with female residents (55.9% vs 39.0%), and female residents saw more female patients than male residents did (61.0% vs 44.1%).
Residents’ experience with various types of patients
Figure 1A shows the frequency distribution of diagnoses according to body system. There were 12 168 diagnoses entered, for an average of 1.34 diagnoses per visit. Psychiatric diagnoses were the most common, accounting for 13.4% of diagnoses entered. Preventive care and musculoskeletal diagnoses were second and third, accounting for 12.8% and 9.9% of all diagnoses, respectively.
Diagnoses encountered by residents: A) Distribution of general diagnostic groups. B) Top 8 diagnoses.
Figure 1B compares the top diagnoses reported by study site residents (37.8% of all visits) with the top 8 diagnoses encountered in capitation practices in Ontario, at University of Toronto–affiliated sites and by physicians at our study site. Overall, the most common diagnoses seen by residents were very much in line with those of the comparison groups; residents saw slightly higher percentages of common diagnoses like hypertension, common cold, and well-person care. As expected, anxiety and neurosis diagnoses were encountered more frequently in Toronto than in the rest of the province. Other top diagnoses among residents that were not among the top 8 in Ontario were eczema or benign skin lesions and nonspecific dermatologic complaints, accounting for 3.7% and 2.5% of all patient visits, respectively.
Figure 2 shows the distribution of 17 frequently performed ambulatory care procedures documented in the RPP. Injections and Papanicolaou smear tests were the most common procedures, with 17.9 and 11.6 procedures, respectively, performed on average per resident during the study period. Other procedures performed at least once on average were digital rectal examination, cryotherapy, and ear syringing.
Procedures performed by residents
DISCUSSION
This study reports the results of a successfully implemented intrahospital Web-based program to monitor the practice profiles of residents. The RPP is an important tool in the education of primary care residents. Ideally, residents should receive as broad a clinical experience as possible with patients who have various diagnoses as evenly distributed as possible.11
Periodic assessment during residency could more accurately define a resident’s learning needs and allow for in-training adjustments of the educational experience in the event of an imbalanced practice profile. The tool has indeed been used to address educational gaps in our residency program in several ways. First, by selectively scheduling patients with specific demographic or diagnostic characteristics with a given resident. This has been particularly effective in increasing the proportion of pediatric and obstetric patients in a resident’s practice. The profile has also been used to balance the proportion of male and female patients in a practice and to populate a resident’s practice with patients suffering from certain conditions, such as cerebrovascular disease, when the RPP indicated a deficiency of such patients.
Because the RPP allows the resident and program director to review practice profiles, it has been used to guide selective and elective choices in order to improve the breadth of a resident’s clinical experience. The fact that the system is ongoing allows for more timely modifications of the learning experience than systems using periodic generations of practice profiles,13,14 especially in a 2-year residency program.
In this report, we provide data on RPPs within the largest primary care training site at the University of Toronto. Our results show that the RPPs of residents did not differ substantially in any resident characteristics except with respect to the proportion of male and female patients per resident, in that both male and female residents saw more patients of their own sex. Neither residents’ age nor status in the program (first or second year) affected the patient visit distribution by age or sex. This finding is consistent with previous reports on the differences in practice characteristics of male and female physicians.10,15
Population health services data can be used as a means to verify the evaluation objectives and competencies developed by the College of Family Physicians of Canada and various training programs. When the practice profiles of residents in our study were compared with those of physicians at the study site, the university, and provincial capitation practices, all groups followed the same general ordering. Anxiety and neurosis diagnoses were the most common for all groups, but were proportionately higher in Toronto than in the province as a whole. Overall, residents encountered a slightly higher percentage of the most common diagnoses except for “well-person care” and “nonspecific musculoskeletal complaints,” for which they had almost double the proportion of encounters.
Examination of procedures documented by residents in their ambulatory care setting revealed that many procedures were rarely or never performed; sigmoidoscopy and diaphragm fitting were not performed by any residents; endometrial biopsy and breast cyst aspiration were performed once by 2 residents; electrodessication, wedge resection of toenails, intrauterine device (IUD) insertion, and joint aspiration were performed fewer than 6 times each. Even though the data were captured over one academic year only, we expect that the general pattern of the procedures never or infrequently performed is fairly consistent across years. If we consider that a resident needs to perform at least 10 procedures to feel confident in performing that procedure,16 our data indicate that trainees will require supplementary experiences that will allow them, upon graduation from the program, to perform the procedures with confidence. In our study the only 2 procedures that met this criterion were injections and Pap smears, performed on average 17.9 and 11.6 times per resident, respectively.
Limitations
The most significant limitation of the RPP system is the reliance on residents to enter patient encounter data, which is an additional task residents must complete in an already busy residency program. A measure of compliance, such as reconciliation of the RPP data with billing data, is desirable. However, given the structure of our billing system (in which residents’ encounters with patients were billed under the staff physician), we were unable to verify the completeness of residents’ recording of patient encounters. Billing data, which are readily available for patient encounters and do not increase the workload of the resident, also have not been shown to reflect the clinical problems addressed during a patient visit completely when compared with the clinical record.17
Although data collected for the RPP were not identical to the OHIP billing codes used in the comparison data and residents typically entered more than 1 diagnosis per visit, the analysis results in very similar diagnoses encountered across all the groups.
These data are generated from a single inner-city teaching site in a university-based program, which does not necessarily reflect the practice profiles of sites in suburban or rural centres. It is reassuring, however, that the types of patient problems encountered by residents in the study were similar to those encountered by fully trained physicians in the study site, university-affiliated sites, and provincial capitation practices.
Conclusion
The RPP is an excellent tool to document the clinical experiences of residents in the ambulatory care setting. Our study has demonstrated that it is possible to achieve practice profiles for residents that reflect profiles that residents will likely encounter upon graduation from the program.
Acknowledgments
The Institute for Clinical Evaluative Sciences is supported by the Ontario Ministry of Health and Long-Term Care. The results and conclusions are those of the authors; no official endorsement by the Ontario Ministry of Health and Long-Term Care is intended or should be inferred.
Notes
EDITOR’S KEY POINTS
-
In a study of the largest primary care training site at the University of Toronto in Ontario, the types of disorders residents encountered were similar to those encountered by physicians at the study site, other sites affiliated with the university, and sites throughout the province.
-
Among the residents, practice profiles differed only in the proportion of visits by male and female patients (male residents saw more male patients and vice versa).
-
The top 8 conditions encountered were similar between resident and comparison groups; anxiety or neurosis was the most common diagnosis among all the groups.
-
Resident practice profiles can reveal whether or not residents are receiving adequate training that is representative of established practice in terms of patient demographics, procedures performed, and diagnoses.
POINTS DE REPÈRE DU RÉDACTEUR
-
Lors d’une étude effectuée dans le plus grand site de formation en soins primaires de l’université de Toronto, en Ontario, les types d’affections rencontrés par les résidents étaient semblables à ceux que rencontraient les médecins pratiquant au site de l’étude, dans les autres sites affiliés à l’université et à dans divers sites dans la province.
-
Les profils de pratique des résidents ne différaient que par la proportion des visites de patients masculins ou féminins, les résidents masculins voyant plus de patients masculins et vice versa.
-
Les 8 conditions les plus fréquemment rencontrées par les résidents étaient semblables à celles des groupes de comparaison, les diagnostics les plus fréquents étant l’anxiété et les névroses chez tous les groupes.
-
Grâce aux profils de pratique des résidents, il est possible de savoir si les résidents reçoivent une formation adéquate, représentative de la pratique habituelle pour ce qui est des caractéristiques démographiques des patients, des interventions pratiquées et des diagnostics.
Footnotes
-
This article has been peer reviewed.
-
Cet article a fait l’objet d’une révision par des pairs.
-
Contributors
All authors contributed substantially to the data analysis and preparation of the manuscript.
-
Competing interests
None declared
- Copyright© the College of Family Physicians of Canada
