AMA Journal of Ethics^®

In recent years, as the Accreditation Council for Graduate Medical Education (ACGME) and the Liaison Committee on Medical Education (LCME) have implemented formal requirements for education in professionalism, medical schools around the country have been fine-tuning their professionalism curricula. The teaching of professionalism used to take place within the hidden curriculum of "rules, regulations and routines," that is, only by medical students learning through observation of housestaff and attending physicians on the wards and through patient and peer interaction [1]. Instruction has now been implemented within the formal curriculum. There are probably as many different types of formal professionalism education as there are medical schools within the United States.

The cornerstone of professionalism education at New York University (NYU) School of Medicine has been the professional development portfolio. The portfolio was designed by the Professional Development Committee (PDC), consisting of medical students advised by Dr. Adina Kalet. In 2000, the PDC was charged by the LCME self-study committee on student assessment to design a fair and meaningful professionalism evaluation process for the school of medicine. Along with the student editors of this issue of Virtual Mentor, I am a fourth-year medical student at NYU and a member of the first class to be taught and evaluated on professionalism through the professional development portfolio. We thus have a unique perspective on the implementation and evolution of this curriculum.

The portfolio consists of an online collection of essays, diagnostic write-ups and (most importantly) student reflections from all four years of medical school. Every submission is cultivated from an experience required in each course. Some examples of submissions during the first year are reflections on standardized patient encounters, such as counseling a patient on smoking cessation or taking a sexual history; narrative essays; and a reflection on peer evaluation within the gross anatomy course. Second-year assignments include specific diagnostic write-ups and reflections on boundaries and on learning the physical exam. The portfolio during the clinical years focuses on experiences within each clinical clerkship, and the fourth year then provides opportunities to reflect upon the transition to the resident level, within the experiences of subinternship and the residency interview process. The Web page for each reflection contains lists of defined professional values and challenges to those values; these lists have check boxes so that the student can recognize and assign the challenges to his or her most recent assignment.

At the end of every school year, each student writes a one- to two-page end-of-year assessment, reflecting on individual portfolio assignments and his or her professional development throughout the year and defining three specific and practical goals for the upcoming year. The student then reviews this assessment with his or her faculty mentor and together with the mentor evaluates his or her professional development. The evaluation of portfolio expectations includes completeness of submission requirements, depth and quality of self-reflection, and responsiveness to feedback.

Self-reflection: a key component

A pivotal aspect of the NYU portfolio is the central role of self-reflection in the professional development of physicians. Ronald Epstein and Edward Hundert write that "because experience does not necessarily lead to learning and competence, cognitive and emotional self-awareness is necessary to help physicians question, seek new information, and adjust their own biases" [2]. They go on to say that "reflection allows practitioners to examine their own clinical reasoning strategies" [3]. Amanda Howe echoes this:

…professional development opportunities must at minimum be constructed to engage students directly with experiences that mimic their future roles, create opportunities that allow them to reflect and rehearse the skills involved in managing such experiences, and require them to take personal responsibility for outcomes of both their experiences and learning [4].

Thus the portfolio is designed to facilitate reflection upon how experiences within the curriculum affect the student's current and future professional development. The anatomy peer review exercise, for example, recognizes the values of collegiality and teamwork and how they are best facilitated within the anatomy lab partnership, in addition to exposing preclinical students to the practice of peer review. Reflecting upon the experience of working as a team and receiving peer feedback is relevant not only for first-year medical students but for clinical students and resident team members on the hospital wards. In this way, and with further impact as the student enters the clinical years, the portfolio seeks to provide a middle space between the explicit curriculum and the hidden curriculum by encouraging reflection upon the values seen and ascertained in the "rules, regulations and routines" of the wards. And because portfolio submissions are based upon course-specific requirements, the professional values and conflicts that arise pertain to the students' actual experience at their current level of training. Stern and Papadakis write that evaluating professionalism is optimal when a "professional dilemma that is relevant to everyday lives" is resolved using "real world contexts" [5].

Along with its relevance to current experience, self-reflection truly underscores the idea of professional development in general. Epstein and Hundert state simply that "competence is developmental" [3]; different values and conflicts become more relevant at different stages of training. For example, awareness of the nuances of an appropriate and meaningful patient-physician relationship grows significantly from the first to fourth year of medical school with increased volume of and autonomy in patient interactions. Starting to contemplate these relationships as a first-year medical student sets a tone for continued professional development, not only throughout medical school but for the rest of medical training and practice.

Reliability and validity in the assessment of professionalism education

Continued professional development occurs within the context of the portfolio at the end-of-year assessment, particularly through the learning goals outlined by the student. These goals are meant to be specific, measurable and formulated by reviewing the conflicts and values exposed by the reflections during the entire year. Epstein and Hundert assert that a strong mentoring system should complement the formulation of an individual "learning plan in which trainees chart their learning needs [and] the means of achieving them…as a required outcome of an assessment" [6].

In an Academic Medicine article, Shiphra Ginsburg et al. emphasize the importance of "reliable, valid, and appropriate evaluation" in professional education [7]. A deliberate attempt was made by the PDC to pursue this goal when training the faculty mentors. Each faculty mentor is required to participate in professional development faculty training, which outlines the expectations for portfolio submissions and the quality of self-reflections and end-of-year assessments, offers examples of appropriate learning goals, and explains how to gauge evaluations. These faculty training workshops provide specific examples of student portfolios that exemplify the three evaluation levels—below expectations, meets expectations and exceeds expectations.

Medical students often have the mind-set that the only acceptable grade is the best grade, so a concerted effort is made to instill the idea that to "meet expectations" is truly excellent and that only a handful of students in an entire class who made contributions to their own and others' professional development above and beyond expectations should receive the mark "exceeds expectations," if this rating is to be meaningful. Students immediately began to wonder what these so-called non-grades meant—who would see the assessment, for what purposes would the assessments be used and who would have access to their portfolios. It was decided that the portfolio was mainly for the students and their faculty mentors, but available at the student's discretion for the dean's letter used in residency applications. In this way the portfolio's central purpose of self-reflection and assessment of professional development was preserved while also providing sufficient motivation for excellence. Amanda Howe writes that assessment should be "high profile, both to ensure competency and to motivate learning" [4].

Debugging the initial portfolio

In actual practice, one of the more challenging aspects of implementing the portfolio system was standardizing the roles and expectations of each of the faculty mentors. Students are assigned their faculty mentors when they affiliate with one of the six theme-based societies that make up the mentoring program of our medical school, for example, the Lewis Thomas Society for Arts and Humanities in Medicine or the Severo Ochoa Society for Medical Informatics and Biotechnology. As a consequence, while students and mentors should share some interests, the actual assignments are made at random. Due to a few instances of conflicts either of personality or in defining expectations and goals of the portfolio or mentorship, a handful of students protested their mentor assignments during the first year of the portfolio. The PDC decided that students could switch mentors at the end of their first year if they desired.

The PDC held many open forums with the students during the initial year of the portfolio's use and, as a result, changed or enhanced some features of the portfolio. For example, while some students embraced the opportunity for self-reflection, others felt like they were being forced to reflect upon certain situations or assignments without educational merit. Therefore, the portfolio was changed to include a few specific required submissions in addition to a group of optional situations or encounters, from which the students could pick a certain number that they felt were most influential in their development. Some of the students thought they had too little guidance in focusing the self-reflections, so every assignment was enhanced with specific questions and thought topics. Epstein and Hundert assert that "curricular change…requires a parallel process of institutional reflection, feedback, and remediation" [6], which has been recognized by NYU. In fact, a committee has just been formed by the new dean of student affairs to assess the proceedings of the professional development committee and the portfolio initiative thus far.

And the inevitable technical bugs showed up during that first year, inasmuch as the portfolio was a new interactive online module. These were corrected in a timely manner but caused some frustration among the first portfolio users.

Forward focus

Although most of the technical glitches in the interactive portfolio have been repaired, some of the more philosophical issues regarding this specific method of evaluating and facilitating professional development remain unresolved. No best method for teaching professional development has emerged. In a recent New England Journal of Medicine article, Ronald Epstein reviews the pros and cons of many different methods of assessment, from multiple choice exams to peer assessment and portfolios like those implemented at NYU [8]. Although no single method predominates in usefulness, Epstein supports the idea that "competence should be assessed in an integrated, coherent, and longitudinal fashion with the use of multiple methods and provision of frequent and constructive feedback" [9]. It is exciting to think that our NYU class now possesses a collection of essays spanning the entire four years of medical education, charting our professional development in a way that we hope is meaningful and truly reflective. The portfolio attempts to encompass the ideals of assessment that Epstein defines, but whether this system exemplifies the optimal method to explore and evaluate professional development remains unknown. Educational studies and outcomes need to continue exploring the ideal way of integrating these goals into professional development.