Abstract
Problem addressed The Continuing Professional Development (CPD) department of the College of Family Physicians of Canada evaluated the Mainpro+® and CERT+® programs from June 2016 to May 2018 to determine users’ awareness of the changes made to each program and to determine user engagement and satisfaction.
Objective of program To assess changes in CPD program quality, the effects of changes on member understanding of credit reporting and engagement, data accuracy of credit reporting, and the perceptions of the CPD program certification process among providers.
Program description Surveys and interviews were conducted with stakeholders from both groups, and consultations occurred with a third-party consultant. Administrative data and program files were also analyzed. More than 33 000 users (about 95% of all Mainpro+ participants) have accessed Mainpro+ since its launch. Satisfaction varies, with 31% of members and 39% of non-member Mainpro+ participants expressing difficulty entering activities. Most users (79%) understand the changes implemented. Among CERT+ users, half (50%) find the platform easy to use, whereas 23% find it difficult; 86% find the CPD program submission requirements somewhat or very clear. Project limitations include difficulty comparing data between phases and a lack of qualitative data.
Conclusion The College of Family Physicians of Canada anticipates these program enhancements will lead to higher-quality CPD programs and greater clarity and efficiency for members and CPD providers. All collected data will be used to inform ongoing improvements to both platforms to improve the experience of all users.
In June 2016, the College of Family Physicians of Canada (CFPC) updated the Mainpro® program with the aim of improving the quality of continuing professional development (CPD) programming and providing greater clarity and efficiency for users. The new program, known as Mainpro+®, includes updates to Mainpro’s framework and guidelines, and revisions of its online credit reporting (Mainpro+) and program certification (CERT+) platforms.
Objective of program
The CPD department recognized the need to measure the effect of these changes and engage CFPC members in the evaluation process. To ensure neutrality in the evaluation process, the CFPC engaged a third-party vendor to conduct a formal evaluation of the Mainpro+ program. The aim of the evaluation was to assess whether the changes implemented in the new Mainpro+ system resulted in higher-quality CPD program development, increased member engagement and understanding of the changes, more accurate data regarding members’ CPD activities, and an improved certification process for developers of CPD. The specific objectives of the evaluation program were as follows:
to measure the effectiveness of the new technology platforms for credit reporting and certification of programs;
to determine the effects of the changes made to Mainpro+ standards and guidelines on members and non-member Mainpro+ participants (NMMPs); and
to assess the extent to which the Mainpro+ program has increased the quality and standards of CPD programming for Canadian family physicians.
The evaluation of the roll-out of Mainpro+ is a critical component of the ongoing monitoring and assessment of CPD policies, processes, and technology to support guidelines, standards, and participation of Canadian family physicians in practice-relevant CPD activities.
The responses were separated by respondents’ member class, as we sought to evaluate whether CFPC members would have a better understanding of the new requirements, credit categories, etc, compared with their NMMP colleagues. Member classes are defined as follows:
Members encompass several membership classes. The largest cohort is active members, who are licensed physicians in good standing who are engaged in the practice of family medicine and who pay annual fees as set by the board, fulfil CPD requirements, and belong to both the national College and a Chapter.1
The NMMPs comprise Mainpro+ program participants who do not receive the benefits of CFPC membership (eg, the use of CFPC designations, a print subscription to Canadian Family Physician, and reduced rates for CFPC programs and conferences). These users are assigned the same 5-year cycle and annual requirements that active CFPC members are.2
Program description
Lines of inquiry.
This article is a summary of findings from the Mainpro+ evaluation undertaken between July 2016 and May 2018. The evaluation draws on several lines of inquiry:
Surveys: Respondents included Mainpro+ participants and CPD providers. Two phases of the surveys were administered, with phase 1 in November 2016 and phase 2 in November 2017.
Interviews with stakeholders: Mainpro+ participants and CPD providers (universities and CFPC Chapters) were interviewed.
Administrative data and program file review: This review included program fees, CPD application data, and Mainpro+ user data.
Consultation with the CPD department: An interactive discussion was held with those closely involved in the project implementation.
Evaluation.
The evaluation was carried out from June 2016 to May 2018. A mixed-methods evaluation design was used to answer research questions. Each of the following lines of inquiry were individually analyzed, with findings summarized in technical reports:
Surveys of Mainpro+ participants and CPD providers: Two online surveys were administered for each respective group in fall 2016 and fall 2017. The surveys captured perspectives on the new platform and the program enhancements. Mainpro+ survey response rates averaged between 8% and 15%, while CERT+ survey response rates averaged between 10% and 12%.
Mainpro+ administrative data and documentation: Data exported from the Mainpro+ platform were analyzed to obtain information about Mainpro+ users, activities, and Impact Assessment Questionnaire results. Additionally, data from the CERT+ platform and program financial records were reviewed to understand effects of changes to the certification fee structure.
Interviews with Mainpro+ participants: Ten telephone interviews (8 in English, 2 in French) were conducted with participants who had indicated willingness to participate in an interview to further explore their experiences with the Mainpro+ platform and to understand their perspectives on enhancements.
Interviews with Chapter representatives and university CPD offices: Ten telephone interviews (8 in English, 2 in French) were conducted with CFPC Chapters and university CPD offices to explore perspectives about the certification program enhancements and the CERT+ platform.
Consultation with the Mainpro+ project team: A 1-hour telephone consultation examined the project team’s perspectives about the program enhancements.
Strengths and limitations of the evaluation
Overall strengths.
Strengths of both the Mainpro+ program and the CERT+ program evaluations included the fact that feedback was obtained from all important Mainpro+ and CERT+ stakeholder groups; the patterns of the survey results were consistent over time and with the interview findings; and the survey results reflected both positive and negative perceptions of the program.
Limitations
Mainpro+: A limitation of the Mainpro+ program evaluation is that younger members are comparatively underrepresented. The composition of the survey respondent group is similar to the overall population; however, women are slightly underrepresented. Respondents’ geographic representation is similar to family physicians in Canada as a whole, but Quebec residents and French speakers are comparatively underrepresented. The samples from phase 1 and phase 2 had reduced comparability; NMMPs made up almost 50% of the phase 1 sample compared with phase 2, which was 25% NMMPs. This was to ensure more CFPC members were included in the phase 2 survey, while keeping the total sample (10 000) the same in both phases.
CERT+: Limitations of the CERT+ program evaluation include possible reduced comparability of the samples between phases 1 and 2 because of the addition of the session contact information list to the sample in phase 2 (which also includes more individuals working at for-profit CPD providers). Information about activity category, delivery format, and credits per hour was available only for Mainpro+ entries with a valid session ID (about half of all relevant activities) and was not available for any CPD activities that were certified by university CPD providers. Thus, these findings should be used with caution, as they do not include all CFPC-certified activities.
Other limitations: Owing to budget limitations, only a small sample of open-ended responses were analyzed. As the surveys were completed by a small pool of participants, self-selection bias is likely. As both platforms were introduced at the same time as new CPD and certification requirements, attitudes about one might be influencing attitudes about the other. Finally, Mainpro+ users are not required to enter their activities right away, so the numbers likely underestimate the actual level of CPD activity among Mainpro+ participants.
Discussion
Do Mainpro+ participants understand the Mainpro+ requirements?
When changes are made to established programs, it often takes some time for participants to understand the changes. As misunderstandings can undermine participant experience and program effectiveness, it becomes important to monitor participant understanding in the months following the changes. We assessed Mainpro+ participants’ understanding of the changes through surveys conducted at 5 months (November 2016) and 18 months (November 2017) after Mainpro+ was launched, and through interviews in January and February 2018.
Most member participants (79%) are aware of the new requirements of the Mainpro+ system (Figure 1). Details about the new cycle dates and 5-year cycle requirements are clearly understood by most. However, more than a year after implementation, there are some lingering areas of confusion (Table 1).
How do Mainpro+ participants experience the new platform?
Mainpro+ administrative data shows that more than 33 000 participants have logged on to Mainpro+ since its launch (representing up to 95% of all Mainpro+ participants), and most users (87%) have logged at least 1 CPD activity. About 2 in 5 have generated a transcript detail report (38%) and about 3 in 10 (29%) have generated a credit summary report. Only 1% of users have used the planning goals feature. These tools are available to members to help them track their progress during a reporting cycle and to plan their learning (Table 2).
The Mainpro+ user experience has been variable. Some survey respondents praised the platform’s general simplicity and ease of use and appreciated that it allows them to claim their credits online. According to one survey respondent, “I love that it is so simple, easy to use, and fast to update.” However, a substantial proportion of members (31%) and NMMPs (39%) continue to find entering an activity difficult, and 21% of members and 30% of NMMPs experience challenges generating reports (Figure 2).
Most interviewees indicated they have had to seek help (usually from CFPC staff) when using the platform. The greatest challenge is with entering activities and finding the correct CPD activity in the drop-down menu. The credit summary page is the Mainpro+ feature most widely regarded as useful, according to 94% of members and 90% of NMMPs. Smaller proportions of participants find other features useful: 69% of members and 59% of NMMPs regard the Holding Area as at least somewhat useful; 52% of members and 44% of NMMPs say the Impact Assessment Questionnaire is at least somewhat useful; and 48% of both members and NMMPs find the CPD goal planning feature at least somewhat useful.
How engaged are Mainpro+ participants in the Mainpro+ program?
While login and other platform use data show us how often Mainpro+ users are accessing the platform, these cannot serve as a measurement of true engagement. We sought a clearer picture by adding information about which features participants are using, as well as qualitative information from interviews and open-ended survey responses about participants’ attitudes toward the platform.
These data points indicated that Mainpro+ users are logging in approximately twice every 3 months, which implies a high level of engagement. However, interviewees indicated they are reporting mainly to stay in good standing rather than to track CPD for their own purposes. Some even indicated they are not logging all of their eligible activities; they either forget to do so, have enough credits already, or have had technical difficulties in logging them. We also inferred from qualitative segments of the survey that NMMPs in particular feel disengaged from the program. These concerns will help inform future strategies to improve communications and outreach to Mainpro+ participants to build on current engagement levels.
There was also recognition on the part of some respondents that aspects of the Mainpro+ enhancements were positive; in particular, the new credit categories were perceived to be more intuitive, the online platform was improved, and the Self-Learning credit category was more flexible.
Do Mainpro+ program providers understand the requirements under the Mainpro+ system?
Most CPD providers (86%) find the certification requirements to be clear. The application fee structure was well understood right away (94% of respondents found this to be clear when asked about in the phase 1 survey). The requirements for ethical review, quality criteria, and documentation to be submitted were also clear to most CPD providers surveyed, but more than a year after program launch, a minority still found these unclear (Figure 3).
Half (50%) of program providers who responded say the CERT+ online platform is easy to use, with 23% finding it either somewhat or very difficult. Some who experienced difficulties with the platform expressed appreciation for the technical support provided by CFPC staff.
What evidence is there that CPD program quality is improving?
Mainpro+ introduced increased rigour in the certification requirements for the 2-credits-per-hour and 3-credits-per-hour programs, which aim to engage in individualized learning with a focus on outcome measurement. It was anticipated that this would translate into high-quality CPD offerings. To understand whether quality is really improving, we asked stakeholders for their perceptions. We also reviewed the administrative data to discern any details about the kinds of programs that are being certified.
We found that the total number of submissions was similar to previous years, from which we inferred that the difficulty of the submission process was not much more difficult after the launch of the new requirements. Twenty-seven percent of providers expressed that the newly implemented requirements had a positive effect on the quality of the 2- and 3-credit-per-hour programs. In contrast, 21% of providers saw no change to the quality of submissions and 23% could not say either way.
Additionally, CFPC Chapter and university CPD reviewers observed improvements to programs in areas that the CFPC sought to improve, namely interactivity, responsiveness to learner needs, health care and pharmaceutical industry bias, evidence base, measurement and reinforcement of learning, effective application of knowledge to practice, and needs assessment.
Conclusion
Physicians participating in Mainpro+ demonstrate a commitment to lifelong learning. The CFPC strives to ensure physicians have the tools and resources in place to support participants of the Mainpro+ platform. The feedback sought from this project yielded key information that aided in improving Mainpro+ policy and the platform in general.
Based on the feedback from the Mainpro+ evaluation, resources were created for members including communication on some of the key areas of confusion in the program such as activity and credit categories. Continuing Professional Development department staff have reduced some of the duplicated administrative requirements of certification and will continue to explore how to further streamline this process. A working group in the assessment credit category has been created, with the aim of providing further clarification and requirements for both program providers and Mainpro+ participants. With 95% of Mainpro+ participants logging in since the launch, the CFPC will continue to explore ways to engage members and is enhancing the process of finding activities in the drop-down menu to improve user experience.
The CFPC anticipates these program enhancements will lead to higher-quality CPD programs and greater clarity and efficiency for members, providers, and other users. We will be using all collected data to inform future improvements to both platforms, including enhancements addressing ease of use concerns, streamlining credit reporting, and communication strategies to better educate Mainpro+ users and CPD providers.
Acknowledgments
We thank all stakeholders who participated in the Mainpro+ evaluation.
Notes
Editor’s key points
▸ Most participants are aware of the new requirements of Mainpro+®. However, there is still some uncertainty surrounding new credit categories and activity categories.
▸ Users generally find Mainpro+ easy to use, but most do not use more advanced features such as report generation and continuing professional development (CPD) planning tools despite describing them as useful. There is still a substantial proportion of Mainpro+ users who find basic functions of the platform difficult to use.
▸ Although participants log in more frequently than anticipated, there is still a high degree of disengagement from all users, especially non-member Mainpro+ participants. However, the implemented enhancements seem to address many of the participants’ concerns regarding ease of use and flexibility.
▸ Most CPD providers understand the new requirements. The chief areas of concern are with the nature and amount of documentation required for the application process, and that the certification process is time-consuming and onerous, especially for organizations with limited resources.
▸ Before the changes to Mainpro+, CPD program quality was perceived as high by participants. Similar ratings were observed in the Mainpro+ platform after the new certification criteria were introduced. Those close to the certification process have seen evidence of improvement, but at this point, few CPD participants have noticed a change in quality. Responses to the survey from other CPD providers contained some scepticism that the new requirements would improve quality.
Points de repère du rédacteur
▸ La plupart des participants connaissent les nouvelles exigences de Mainpro+®. Toutefois, une certaine incertitude persiste entourant les nouvelles catégories de crédits et d’activités.
▸ Les utilisateurs estiment généralement que Mainpro+ est facile à utiliser, mais la plupart d’entre eux ne se servent pas des caractéristiques plus avancées, comme la production de déclarations et les outils de planification du développement professionnel continu (DPC), même s’ils les qualifient d’utiles. Il y a encore une proportion considérable d’utilisateurs de Mainpro+ qui trouve difficile de se servir des fonctions de base de la plateforme.
▸ Même si les participants se connectent plus souvent que prévu, un fort degré de désengagement perdure chez tous les usagers, surtout de la part des nonmembres qui participent à Mainpro+. Par contre, les améliorations apportées semblent répondre à un bon nombre des préoccupations des participants concernant la facilité d’utilisation et la flexibilité.
▸ La plupart des fournisseurs de DPC comprennent les nouvelles exigences. Ils sont davantage préoccupés par la nature et la quantité de la documentation requise dans le processus de demande de certification, et par le temps requis et la lourdeur de ce processus, surtout pour les organisations dont les ressources sont limitées.
▸ Avant les changements apportés à Mainpro+, la qualité du programme de DPC était perçue par les participants comme étant élevée. Des évaluations semblables ont été observées à propos de la plateforme Mainpro+ après l’instauration des nouveaux critères de certification. Les intervenants associés de près au processus de certification ont vu des preuves d’amélioration, mais peu de participants au DPC ont remarqué des changements sur le plan de la qualité. Les réponses d’autres fournisseurs de DPC au sondage exprimaient un certain scepticisme quant à une amélioration de la qualité en raison des nouvelles exigences.
Footnotes
Contributors
All authors contributed to the concept and design of the study; data gathering, analysis, and interpretation; and preparing the manuscript for submission.
Competing interests
All authors are paid employees of the College of Family Physicians of Canada.
This article has been peer reviewed.
Cet article a fait l’objet d’une révision par des pairs.
- Copyright© the College of Family Physicians of Canada