Abstract
-
Objective
Workplace-based assessment (WBA) plays a crucial role in assessing entrustable professional activities (EPAs) in the competency-based medical education era. This pilot study explored the perceptions of residents and assessors of two WBAs for three Korean Society of Emergency Medicine EPAs.
-
Methods
Eight emergency medicine (EM) residents underwent WBAs, with mini-clinical evaluation exercises (mini-CEX) conducted by nine EM faculty members and multisource feedback (MSF) provided by two internal medicine faculty members and four emergency room nurses, for a total of 69 assessments. We conducted an anonymous online survey to gather feedback on experiences, perceptions, and recommendations for improving WBA, such as mini-CEX and MSF, with responses scored on a 5-point Likert scale.
-
Results
Of the 23 initial participants, 15 (65.2%) responded, including 5 residents and 10 assessors. EM faculty viewed mini-CEX favorably, noting its strong integration of supervision and effectiveness in assessing resident performance. EM residents reported comfort issues during assessments, preferring immediate feedback and multiple assessors. MSF was generally perceived positively but showed discrepancies in the utilities of rating scales and feedback types, indicating potential areas for improvement.
-
Conclusion
Two WBAs for three Korean Society of Emergency Medicine EPAs were found to be feasible and acceptable in the context of Korean EM residency training. However, perceptions varied between assessors and residents, necessitating clear communication about WBA objectives and processes. Our findings are useful for shaping future EPA-based training programs, balancing traditional and WBA methods, and enhancing feedback quality.
-
Keywords: Medical education; Graduate; Internship and residency
Capsule Summary
What is already known
Workplace-based assessment (WBA) has become a vital element of competency-based medical education globally, especially for evaluating entrustable professional activities (EPAs) in clinical environments. WBA methods, including mini-clinical evaluation exercises (mini-CEX) and multisource feedback (MSF), have been adopted in numerous countries and demonstrate effectiveness in enhancing medical training by providing realistic evaluations of trainees' skills.
What is new in the current study
This study is a pilot trial of WBA using EPA-based assessments in Korean emergency medicine. It provides insight into assessor and resident perceptions of WBA and highlights the challenges and opportunities of integrating these methods into emergency medicine training in Korea. The study also explores the variations in perceptions between different assessors and suggests the need for improved communication and feedback mechanisms to enhance the effectiveness of WBA.
INTRODUCTION
An entrustable professional activity (EPA) is a unit of professional practice that constitutes the daily work of clinicians [
1,
2]. In other words, EPA stands for a basic unit of competency that must be possessed at the point of performing medical treatment with independent responsibility without a supervisor in actual medical treatment [
2,
3]. The evaluation of EPAs has rapidly become immensely popular, and EPAs are now being introduced or adopted in postgraduate medical specialty training programs and undergraduate medical education programs in several countries [
4–
6].
There are several approaches to assessing EPAs, such as workplace-based assessment (WBA). This method involves assessing learners in their regular working environments, specifically in clinical settings. WBA can take various forms, such as direct observation of patient care processes or procedures, discussions based on medical records or cases, and feedback from colleagues and patients [
7]. In competency-based medical education (CBME), WBA supplements traditional educational learning and enhances learner performance by providing feedback and coaching [
8]. WBA plays a crucial role in CBME for assessing EPAs and ensuring that learners are equipped to deliver quality patient care. Compared to traditional paper-based or educational examinations, WBA provides more comprehensive and accurate assessments of trainee capabilities, enabling continuous improvement and evaluation throughout the educational program.
The American emergency medicine (EM) residency program has implemented CBME and WBA with 23 milestones for over a decade, with evaluations by the Accreditation Council for Graduate Medical Education (ACGME) categorizing resident competency levels on a scale of 1 to 5 [
9]. Canada recently revamped its training programs to incorporate the EPA concept and introduced WBA in EM specialist training [
10]. The perspectives of trainers and trainees of WBA and EPA-based assessments vary depending on individual experience, background, and learning style, and may significantly impact assessment results [
11–
13].
The Research on the Development and Systematization of the Training Curriculum for Residency project implemented by the Korean Ministry of Health and Welfare has mandated the Korean Society of Emergency Medicine (KSEM) to develop EPAs and enhance training of residents. KSEM developed a plan to systematize training and programmatic evaluation in 2021 including 10 EPAs (
Supplementary Table 1). Detailed interpretations of each EPA and corresponding assessment method guidelines have been presented [
14]. As this was an extensive renovation of EM training in Korea and the assessment tools were brand new, the task force felt it was necessary to evaluate the program in real-world clinical environments in a preliminary fashion. Pilot implementation in one hospital was conducted voluntarily, achieving this purpose.
Despite the fact that residency training programs are based on EPA in many specialties in Korea, there is ongoing difficulty in effectively implementing WBA. A previous literature review revealed that engagement with WBA in medical workplaces varies significantly, and that there is widespread negativity towards WBA among trainees and trainers [
12]. Most previous research on the implementation of EPA or WBA has been conducted in western countries such as the United Kingdom, Canada, or the United States, with a dearth of literature from Asia [
12,
13,
15,
16]. Some WBA studies in Asia have reported results for user satisfaction and benefits, but there is still limited experience with the full-scale implementation of WBA. Additionally, Asian teaching perspectives, which focus on knowledge transfer, differ from western approaches that prioritize individual competence, differences that should be taken into account [
17,
18]. In Asia, the teacher-student relationship typically exhibits high power distance and low individualism. These factors may discourage residents from actively seeking feedback, as assessments are often seen as authoritative. Personalized feedback based on individual competency levels is also challenging within the hierarchical and standardized structure of many Asian training environments. To effectively implement WBA, it is essential to shift from group supervision to providing individual feedback [
16,
19]. There is a need to examine perceptions surrounding WBA based on EPAs to understand its implementation in the context of educational culture in Asia.
In this pilot study we employed an observational, mixed methods design to investigate the feasibility, acceptability, and stakeholder perceptions of implementing WBA aligned with EPA in a Korean EM residency training program.
METHODS
Ethics statement
This study was approved by the Institutional Review Board of Hallym University Dongtan Sacred Heart Hospital (No. HDT 2023-05-011-001). All participants provided online informed consent before the survey. No personal information was collected during the study to ensure privacy.
Study setting and design
The study was conducted in an 800-bed teaching hospital in a suburban area of Korea. This emergency department serves over 60,000 patients annually and is staffed by 8 EM residents, 13 EM specialists, and 74 nurses.
The WBA, based on the KSEM EPAs, was implemented from September to December 2022. Before starting WBA, a workshop was conducted to help EM residents and assessors who would participate in WBA understand the concepts of EPA and WBA and learn how to provide appropriate feedback. WBA was applied to evaluate EPA 2 (perform a history, physical examination, and differential diagnosis on an emergency patient, and develop an examination and treatment plan), EPA 5 (communicate well with emergency patients, their caregivers, and collaborating healthcare providers about the process and outcomes of care), and EPA 6 (share information and collaborate with other healthcare providers for continuity of care) of the 10 EPAs. The WBA evaluation methods were the mini-clinical evaluation exercise (mini-CEX) and multisource feedback (MSF) using a form developed by the KSEM task force team [
14]. Mini-CEX was conducted during residents' regular shifts. Residents requested to be assessed by faculty members, and the faculty then assigned appropriate patient cases to residents and observed their clinical processes. After each shift, the faculty would provide written feedback through the assessment form. While verbal feedback was not mandatory, some assessors offered it immediately. In the mini-CEX, faculty used tools developed specifically for each EPA. Nurses and faculty used different versions of the MSF tool. For nurses, the focus was on teamwork in the emergency department and attitudes towards other professions, while faculty assessed residents' ability to assess patients and how well they communicated with specialists.
While the WBA was being conducted, all participants were notified about a post-WBA survey to obtain feedback from participants. We surveyed participants anonymously in July 2023, about 3 months after finishing the entire WBA process and providing feedback to trainees. An advertisement for the WBA feedback survey initiative was delivered to each participant in person and posted at the nurse and doctor work stations in the emergency room to make participation as convenient as possible. The advertisement included information about the survey's objectives and how participants could access and respond to it. Three QR codes were provided to each of the trainees, EM faculty, and MSF assessors to access the survey (Google Forms, Google). Each participant was informed to answer only one survey according to their position.
Development of survey tools
The questionnaire investigated the experiences, perceptions, and recommendations surrounding WBA conducted in the emergency department for this study. Authors (KHP and JOP) reviewed previous studies and developed the survey [
20–
22]. To develop mini-CEX-related questionnaires, we modified the survey tool utilized in Bashir et al. [
20] and Joshi et al. [
22]. Similarly, we revised the questionnaire to assess experiences and perceptions of MSF, based on Burford et al. [
21]. Questionnaires were developed separately for assessors (EM faculty, other faculty, and nurses) and residents. The questions on the survey utilized a 5-point Likert scale anchored by strongly agree/always (5 points), neutral/sometimes (3 points), and strongly disagree/rare (1 point).
After questionnaires were developed, all questions were aggregated and reorganized into three types of questionnaires. The first was for residents, and included the mini-CEX and MSF. The second was for mini-CEX assessors (EM faculty), and the third was for MSF assessors. We added open-ended questions to the ends of all questionnaires to collect opinions on improving WBA.
The survey used Google Forms with an appropriate add-on technique to ensure anonymity and participants had the right to withdraw at any time to protect privacy and confidentiality. The questionnaire's introduction provided detailed explanations of these measures, and participants were required to provide online informed consent before they started the survey.
Data analysis
The results of all questions answered on a 5-point scale were summarized as mean and standard deviation. Open-ended comments were not specifically analyzed but were used for reference. Due to the small number of respondents, responses were not normally distributed, and statistical significance could not be assessed. Based on a review of the survey results, we concluded that a mean score ranging from 4.5 to 5 indicates strong agreement or that the task is usually done; a score ranging from 4 to 4.5 indicates moderate agreement or that the task is often done; a score ranging from 3.5 to 4 indicates neutrality or that the task is sometimes done; and a score less than 3.5 indicates disagreement or that the task is rarely done.
RESULTS
A total of eight EM residents participated in the WBA. Nine EM faculty conducted mini-CEX, while MSF was completed by two faculty members from internal medicine and four emergency room nurses. In total, 69 assessments were conducted, with an average of 8.6 assessments per resident. One resident who underwent 13 assessments received the highest number of assessments, while the resident with the fewest assessments had only 7. Among the assessors, EM faculty conducted 26 assessments, internal medicine faculty members conducted 11 assessments, and nurses conducted 32 assessments. Fifteen survey responses were collected: five from residents, six from mini-CEX assessors (EM faculty), and four from MSF assessors (other faculty or nurses) (
Supplementary Table 2).
Table 1 shows EM faculty experiences and perceptions of mini-CEX as assessors. High scores for experiences (4.8±0.4) in response to "Most mini-CEX were completed during clinical supervision shifts with almost no extra time," indicate strong agreement, suggesting this was a usual practice. "Direct observation of resident's performance" and "ease of carrying out the exercise" both scored 4.3±0.5, indicating moderate agreement. Regarding assessor perceptions, "mini-CEX can supplement traditional internal assessment" scored 4.7±0.5, showing strong agreement. However, "mini-CEX improved my attitude towards residency training" scored 3.8±1.1, indicating neutrality. These results suggest positive experiences and perceptions among faculty, with suggestions for integrating mini-CEX alongside traditional methods while acknowledging its time-consuming nature.
In
Table 2, EM resident experiences, perceptions, and suggestions with mini-CEX are detailed. Scores for "assessment conducted in practice" (4.4±0.5) and "directly observed for a specific patient" (4.4±0.5) indicate that an assessment was often performed in a usual clinical setting. "Comfortable while being examined" scores were low (2.8±1.0), suggesting feelings of discomfort. Feedback-related items such as "Duration of feedback was appropriate" and "The feedback I received was relevant" both scored 4.2±0.8, indicating moderate agreement or that these aspects were often well received. However, "Opportunity to express views during feedback" received a score of 3.2±0.8, indicating this goal was only occasionally achieved, and suggesting a need for improvement. In perception questionnaires, satisfaction with mini-CEX (3.6±1.2) and skill enhancement in medical interviewing (3.8±1.2) scored moderately, indicating neutrality. In the proposal questionnaire, “multiple evaluators” (4.4±0.5) and “separate time for feedback” (4.0±0.6) received moderate levels of agreement, indicating a desire for evaluation from various perspectives and a need for time to focus on feedback and reflection.
Tables 3 and
4 outline perceptions of MSF among assessors and EM residents. Assessors neutrally agreed that MSF is a good idea (3.5±1.5) and residents moderately agreed, scoring 4.2±0.4.
In
Table 3, assessors indicated that the MSF form was not easy to complete (3.3±0.4). However, they strongly agreed that feedback provided to residents through the MSF would positively impact behavior or attitudes (4.5±0.5). In the questionnaire regarding the response format, assessors strongly agreed on the usefulness of the rating scales (4.5±0.5) and moderately agreed on the usefulness of the written comments (4.3±0.8).
In
Table 4, residents expressed neutrality regarding the overall usefulness of MSF for identifying a doctor in difficulty (3.6±0.5). However, they moderately agreed that the feedback received from assessors was useful for reflecting their manner of working with colleagues (4.4±0.5), which aligns with assessor perceptions. Residents favored written comments in MSF over a rating scale (4.2±0.8 vs. 3.8±0.8). They also agreed that the feedback provided by the assessors was reliable (4.4±0.5) and that the assessors were knowledgeable about their work (4.4±0.5), reinforcing the validity of the assessment.
DISCUSSION
This study presents an important first step in implementing WBA in Korean EM residency training by exploring the experiences and perceptions of assessors, including EM faculty, other faculty members, nurses, and EM residents, regarding mini-CEX and MSF. We found that assessors generally had positive experiences and perceptions of mini-CEX as an assessment method. They expressed particularly strong agreement with its integration during clinical supervision and held moderate beliefs in its utility for internal assessment, although it was acknowledged to be time-consuming. EM residents generally expressed concern about psychological comfort during the mini-CEX and preferred having multiple assessors and dedicated time for feedback. Both assessors and residents recognized the usefulness of MSF in enhancing collaboration with colleagues. However, there were differences in the perceived utility of MSF rating scales and feedback types.
CBME is growing in popularity worldwide as it produces better-prepared physicians. In CBME, learners advance when they meet fixed competency standards based on EPA assessments by WBA. WBA provides authentic assessments of a physician's day-to-day activities and can offer feedback to learners, supporting learning [
11,
23–
26]. Numerous tools and programs for WBA have been introduced across various fields and specialties. One of the main obstacles to effectively implementing WBA tools is a need for more engagement during the design phase and time constraints in clinical environments. Additionally, there are challenges in translating the technical language of competency-based assessment into easily understandable terms [
11].
The positive responses from both assessors and residents towards mini-CEX and MSF indicate readiness to embrace these methods. The other CBME in the internal medicine residency training program with WBA showed an overall satisfaction score of 3.24 [
27]. However, varying perceptions between these groups highlight the need for clear communication about the objectives and processes of WBA. Practically, user perceptions of WBA varied and often depended on implementation factors. The most important concern for assessors is how much effort and time WBA requires [
28,
29]. There may be negativity among trainees as well as, to an extent, their trainers. This negativity is caused by three dominant problems: poor understanding of the purpose of WBAs, insufficient time available for undertaking these assessments, and inadequate residency training [
12].
In our study, mini-CEX often took 10 to 15 minutes and was completed during the clinical supervision shift. Sometimes, EM faculty put in more time and effort for their assessments compared to traditional internal assessments, which were usually in the form of quizzes or written tests, because using the tools provided in the mini-CEX to observe and provide feedback was more time-consuming than usual bedside teaching. MSF assessors disagreed that the MSF form was easy to complete, indicating that they might feel pressured to provide feedback during daily communication with residents. This perception can significantly impact assessor engagement, especially in time-pressured emergency department settings. To reduce this problem, some researchers have suggested simplifying frontline assessments to be more intuitive for assessors to complete [
30]. Assessor training was also recognized as important and necessary for supporting a faculty-driven collection of learner assessments, and others have advocated the use of digital technology to provide real-time feedback and documentation [
8].
Regarding the operation of mini-CEX, residents strongly preferred having mini-CEX assessed multiple times, preferably by multiple assessors rather than repeatedly by the same assessor. As an assessment, WBA was not free from assessor or learner-driven sampling bias in data collection [
8]. To ensure unbiased sampling and overcome the limitations of individual assessment tools and raters' shortcomings, collecting WBA data with multiple methods using multiple assessors and across various contexts is recommended [
31,
32]. This approach helps to reduce the risk of biased sampling and addresses issues such as halo or leniency effects caused by individual raters.
Notably, residents reported discomfort during mini-CEX assessments, which could affect their performance and overall effectiveness despite being told it was a formative assessment. This perception is important when utilizing WBA, including mini-CEX, as a formative assessment. Although WBA is fundamentally a tool to facilitate resident learning, if used as a formative assessment, it may cause stress surrounding assessments by impacting performance and generating a staged environment, and thus will not fulfill its original purpose [
12]. Creating a comfortable and supportive assessment environment is essential. When assessors create a supportive assessment environment with opportunities for learner agency, assessment is perceived as lower stakes and more useful for learning [
13]. Addressing these comfort concerns improves the assessment experience and ensures that assessments accurately reflect residents’ capabilities and learning progress.
This study underscores the importance of timely and relevant feedback about mini-CEX during the learning process. The resident preference for dedicated feedback time in mini-CEX assessments is a critical insight, emphasizing the role of feedback in real-time learning and skill development. This finding suggests that feedback mechanisms should be an integral part of WBA assessment, tailored to provide constructive, timely, and actionable insights. The ability to receive and apply feedback effectively can significantly enhance residents' learning experience and professional growth. The key to achieving buy-in is that WBA should be designed and implemented in a way that is meaningful to users [
10]. For residents, meaning was enhanced when the WBA process was seen as having educational value, so immediate and appropriate feedback is critical [
30].
Regarding the response format of MSF, we observed perception gaps between assessors and residents, particularly in the rating scales. While assessors were very positive about using rating scales to give scores, residents disagreed, suggesting that residents preferred more qualitative feedback through MSF. Bridging this gap in perceptions and ensuring that assessors and residents both understand the purpose of MSF and work in harmony are essential for success.
Limitations
An assessment form for WBA was initially developed by EM residency training experts on the KSEM task force. However, further nationwide pre-implementation validation of the WBA assessment form is required to ensure external validity and internal consistency. This study can serve as a starting point, and we hope that it will create opportunities to develop and validate more sophisticated items tailored to the Korean educational environment.
Although this study was conducted in a single training hospital, it provides valuable insights into the perceptions of trainers and trainees regarding WBA and EPA-based assessments in EM residents. Previous studies of user perceptions were often limited to a single institution or specialty [
11]. While it may not reflect the overall training environment in Korean clinical settings, the results of this study serve as a strong foundation for building future multicenter studies and pilot projects.
This study was conducted 7 months after the WBA, which may have affected participants' ability to remember the experience accurately. To minimize recall and social desirability biases, the survey should have been conducted immediately after the WBA, and changes in perceptions should have been assessed before and after implementation. Finally, only quantitative data from the questionnaire were analyzed, which is a potential limitation of the study. Qualitative research methodology is widely used in research exploring participants' perspectives and experiences with assessments [
33,
34]. All participants were interviewed, and qualitative feedback was gathered immediately after completing the WBA. Unfortunately, interviews were not constructed for research purposes, resulting in an interview format that was too informal to be useful for qualitative research. As a result, we were unable to employ the mixed-method approach, but we attempted to incorporate feedback into quantitative surveys and for survey interpretation. Therefore, future research could mitigate these limitations by recruiting larger sample sizes, conducting multicenter studies, and applying qualitative or mixed methods to understand the perspectives of participants better. By addressing these constraints, it will be possible to produce more dependable and precise results to guide best practices in WBA and EPA assessments.
The insights from this study are pivotal for shaping future EPA-based training programs in Korea. Our findings suggest the necessity of balancing traditional and WBA methods, addressing comfort and perception issues, and enhancing feedback quality. Future training programs should consider these aspects to create more effective, learner-centered environments. This study serves as a foundational guide, offering a nuanced understanding of the current landscape and clear directions for future improvements in medical training.
NOTES
-
Author contributions
Conceptualization: YTO, JOP; Data curation: YTO; Formal analysis: JOP; Investigation: KHP; Methodology: KHP; Project administration: JOP; Supervision: JOP; Writing–original draft: KHP; Writing–review & editing: all authors. All authors read and approved the final manuscript.
-
Conflicts of interest
The authors have no conflicts of interest to declare.
-
Funding
The authors received no financial support for this study.
-
Data availability
Data analyzed in this study are not publicly available due to privacy restrictions, but are available from the corresponding author on request.
Supplementary materials
Table 1.Emergency medicine faculty’s experiences and perception of the mini-CEX
Table 1.
|
Question |
Score |
|
Experience |
|
|
I directly observed the resident's performance. |
4.3±0.5 |
|
It took me 10–15 minutes to complete the exercise. |
4.2±0.4 |
|
Most mini-CEX were completed during clinical supervision shifts with almost no extra time. |
4.8±0.4 |
|
The exercise is easy to carry out. |
4.3±0.5 |
|
I provided the feedback in a constructive way. |
4.0±0.8 |
|
I provided appropriate feedback to the residents discussing both their weaknesses and strengths. |
4.0±0.8 |
|
Perception |
|
|
Mini-CEX requires more time and commitment than the usual method of internal assessment. |
3.8±1.1 |
|
I feel that mini-CEX can sample more areas for assessing residents' competence than the traditional internal assessment. |
4.2±0.7 |
|
Evaluation of a candidate by mini-CEX is better than the traditional way of internal assessment. |
4.0±0.6 |
|
Mini-CEX can supplement the traditional way of internal assessment. |
4.7±0.5 |
|
Mini-CEX has improved my own attitude towards resident training. |
3.8±1.1 |
Table 2.Emergency medicine residents’ experiences, perception, and suggestions to the mini-CEX
Table 2.
|
Question |
Score |
|
Experience |
|
|
The assessment was conducted while I was in practice. |
4.4±0.5 |
|
I knew that the assessor was evaluating me. |
4.0±0.9 |
|
It took me 10–15 minutes to complete the exercise. |
4.2±0.8 |
|
Directly observed for a specific patient. |
4.4±0.5 |
|
Aware of the competencies being assessed. |
4.2±0.8 |
|
Comfortable while being examined. |
2.8±1.0 |
|
Duration of feedback was appropriate. |
4.2±0.8 |
|
The feedback I received was relevant. |
4.2±0.8 |
|
Feedback made me aware of my strong points. |
4.0±0.9 |
|
Feedback made me aware of my weak points. |
4.0±0.9 |
|
Opportunity to express my views during feedback. |
3.2±0.8 |
|
Perception |
|
|
Satisfied with mini-CEX as a method of assessment. |
3.6±1.2 |
|
Mini-CEX enhanced my skills in medical interviewing. |
3.8±1.2 |
|
Mini-CEX enhanced my skills in physical examination. |
3.4±1.0 |
|
Mini-CEX enhanced my professionalism skills. |
3.6±1.2 |
|
Suggestions |
|
|
It is better to have multiple assessors evaluate each resident. |
4.4±0.5 |
|
It is better to have one assessor repeat the assessment. |
3.2±0.8 |
|
It is advisable to comprehensively assess multiple EPAs in one patient. |
3.8±1.0 |
|
It is recommended to select appropriate patients for each EPA and assess them separately. |
3.6±0.8 |
|
Combining work and assessment helps improve your expertise as an emergency medicine doctor. |
3.6±1.2 |
|
It is best to provide immediate feedback from the assessor. |
3.8±1.0 |
|
Separate time should be given for feedback. |
4.0±0.6 |
Table 3.Assessors’ perception of the MSF
Table 3.
|
Question |
Score |
|
Overall view |
|
|
MSF is a good idea in principle. |
3.5±1.5 |
|
How appropriate did you find the level of detail or focus of the questions? |
3.5±1.1 |
|
Workload and ease of use |
|
|
The MSF form was easy to complete. |
3.3±0.4 |
|
Usefulness |
|
|
The feedback provided by this form would successfully identify a doctor in difficulty. |
3.8±0.8 |
|
MSF will lead to positive changes in residents’ behavior and/or attitudes. |
4.5±0.5 |
|
How useful do you think the feedback you gave through this form was in each of these areas: |
|
|
Relationships with patients |
4.0±0.7 |
|
Working with colleagues |
4.3±0.4 |
|
Attitude and approach to job |
3.8±1.1 |
|
Professional skills (record-keeping, time management, etc.) |
3.8±1.1 |
|
Response format |
|
|
How useful were the rating scales (tick boxes) for giving the feedback you wanted? |
4.5±0.5 |
|
How useful were the spaces for writing a comment for giving the feedback you wanted? |
4.3±0.8 |
|
Validity |
|
|
I had sufficient experience of the doctor’s work to give accurate ratings. |
4.0±1.0 |
Table 4.Emergency medicine residents’ perception of the MSF
Table 4.
|
Question |
Score |
|
Overall view |
|
|
MSF is a good idea in principle. |
4.2±0.4 |
|
How appropriate did you find the level of detail or focus of the questions? |
4.0±0.0 |
|
How positive is the feedback that you have received? |
3.8±0.4 |
|
Usefulness |
|
|
The feedback provided by this form would successfully identify a doctor in difficulty. |
3.6±0.5 |
|
How useful do you think the feedback you received from this form was in each of these areas: |
|
|
Relationships with patients |
4.0±0.0 |
|
Working with colleagues |
4.4±0.5 |
|
Attitude and approach to job |
3.8±0.8 |
|
Professional skills (record-keeping, time management, etc.) |
3.6±0.8 |
|
Response format |
|
|
How useful were the rating scales (tick boxes) for giving the feedback you wanted? |
3.8±0.8 |
|
How useful were the spaces for writing a comment for giving the feedback you wanted? |
4.2±0.8 |
|
Validity |
|
|
I think the feedback I was given on this form was reliable and trustworthy. |
4.4±0.5 |
|
I am concerned some ratings or comments were not based on actual experience of my work. |
3.4±1.0 |
|
I know assessors have experience of my work. |
4.4±0.5 |
|
I get on with them as a person. |
4.4±0.5 |
|
I expected to get positive feedback from them. |
4.0±0.6 |
|
I expected to get critical feedback from them. |
3.8±0.4 |
REFERENCES
- 1. ten Cate O. Entrustability of professional activities and competency-based training. Med Educ 2005;39:1176-7.
- 2. Cate OT. A primer on entrustable professional activities. Korean J Med Educ 2018;30:1-10.
- 3. Ten Cate O, Taylor DR. The recommended description of an entrustable professional activity: AMEE Guide No. 140. Med Teach 2021;43:1106-14.
- 4. Hart D, Franzen D, Beeson M, et al. Integration of entrustable professional activities with the milestones for emergency medicine residents. West J Emerg Med 2019;20:35-42.
- 5. Englander R, Flynn T, Call S, et al. Toward defining the foundation of the MD degree: core entrustable professional activities for entering residency. Acad Med 2016;91:1352-8.
- 6. Prudhomme N, O’Brien M, McConnell MM, Dudek N, Cheung WJ. Relationship between ratings of performance in the simulated and workplace environments among emergency medicine residents. CJEM 2020;22:811-8.
- 7. Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ 2010;341:c5064.
- 8. Chan TM, Sebok-Syer SS, Cheung WJ, Pusic M, Stehman C, Gottlieb M. Workplace-based assessment data in emergency medicine: a scoping review of the literature. AEM Educ Train 2020;5:e10544.
- 9. Kessler CS, Leone KA. The current state of core competency assessment in emergency medicine and a future research agenda: recommendations of the working group on assessment of observable learner performance. Acad Emerg Med 2012;19:1354-9.
- 10. Sherbino J, Bandiera G, Doyle K, et al. The competency-based medical education evolution of Canadian emergency medicine specialist training. CJEM 2020;22:95-102.
- 11. Anderson HL, Kurtz J, West DC. Implementation and use of workplace-based assessment in clinical learning environments: a scoping review. Acad Med 2021;96(11S):S164-74.
- 12. Massie J, Ali JM. Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings. Adv Health Sci Educ Theory Pract 2016;21:455-73.
- 13. Acai A, Li SA, Sherbino J, Chan TM. Attending emergency physicians’ perceptions of a programmatic workplace-based assessment system: the McMaster Modular Assessment Program (McMAP). Teach Learn Med 2019;31:434-44.
- 14. The Korean Society of Emergency Medicine (KSEM). [Research on the development and systematization of the training curriculum for residency]. KSEM; 2023
- 15. Shorey S, Lau TC, Lau ST, Ang E. Entrustable professional activities in health care education: a scoping review. Med Educ 2019;53:766-77.
- 16. Tan J, Tengah C, Chong VH, Liew A, Naing L. Workplace based assessment in an Asian context: trainees’ and trainers’ perception of validity, reliability, feasibility, acceptability, and educational impact. J Biomed Educ 2015;2015:615169.
- 17. Wong AK. Culture in medical education: comparing a Thai and a Canadian residency programme. Med Educ 2011;45:1209-19.
- 18. Pratt DD, Kelly M, Wong WS. Chinese conceptions of ‘effective teaching’ in Hong Kong: towards culturally sensitive evaluation of teaching. Int J lifelong Educ 1999;18:241-58.
- 19. Suhoyo Y, Schonrock-Adema J, Rahayu GR, Kuks JB, Cohen-Schotanus J. Meeting international standards: a cultural approach in implementing the mini-CEX effectively in Indonesian clerkships. Med Teach 2014;36:894-902.
- 20. Bashir K, Arshad W, Azad AM, Alfalahi S, Kodumayil A, Elmoheen A. Acceptability and feasibility of Mini Clinical Evaluation Exercise (Mini-CEX) in the busy emergency department. Open Access Emerg Med 2021;13:481-6.
- 21. Burford B, Illing J, Kergon C, Morrow G, Livingston M. User perceptions of multi-source feedback tools for junior doctors. Med Educ 2010;44:165-76.
- 22. Joshi MK, Singh T, Badyal DK. Acceptability and feasibility of mini-clinical evaluation exercise as a formative assessment tool for workplace-based assessment for surgical postgraduate students. J Postgrad Med 2017;63:100-5.
- 23. Henry D, West DC. The clinical learning environment and workplace-based assessment: frameworks, strategies, and implementation. Pediatr Clin North Am 2019;66:839-54.
- 24. Irby DM, Cooke M, O’Brien BC. Calls for reform of medical education by the Carnegie Foundation for the Advancement of Teaching: 1910 and 2010. Acad Med 2010;85:220-7.
- 25. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007;29:855-71.
- 26. van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Med Teach 2012;34:205-14.
- 27. Yoon CH, Myung SJ, Park WB. Implementing competency-based medical education in internal medicine residency training program: the process and impact on residents’ satisfaction. J Korean Med Sci 2019;34:e201.
- 28. Nair BK, Parvathy MS, Wilson A, Smith J, Murphy B. Workplace-based assessment: learner and assessor perspectives. Adv Med Educ Pract 2015;6:317-21.
- 29. Sharma S, Sharma V, Sharma M, Awasthi B, Chaudhary S. Formative assessment in postgraduate medical education: perceptions of students and teachers. Int J Appl Basic Med Res 2015;5(Suppl 1):S66-70.
- 30. Caretta-Weyer HA, Gisondi MA. Design your clinical workplace to facilitate competency-based education. West J Emerg Med 2019;20:651-3.
- 31. Chan TM, Sherbino J, Mercuri M. Nuance and noise: lessons learned from longitudinal aggregated assessment data. J Grad Med Educ 2017;9:724-9.
- 32. Perry M, Linn A, Munzer BW, et al. Programmatic assessment in emergency medicine: implementation of best practices. J Grad Med Educ 2018;10:84-90.
- 33. Branfield Day L, Miles A, Ginsburg S, Melvin L. Resident perceptions of assessment and feedback in competency-based medical education: a focus group study of one internal medicine residency program. Acad Med 2020;95:1712-7.
- 34. Ahn E, LaDonna KA, Landreville JM, Mcheimech R, Cheung WJ. Only as strong as the weakest link: resident perspectives on entrustable professional activities and their impact on learning. J Grad Med Educ 2023;15:676-84.
