AMA Journal of Ethics^®

Case

Brent, a fourth-year medical student applying to residencies in internal medicine, has worked diligently during medical school, achieving high grades, garnering commendations for empathic bedside care, assuming leadership positions in several organizations, and volunteering in underserved care programs. He is a competitive applicant for the specialty and has already received a number of interview requests. His first interview is at a small, exclusive, but well-known program, and Brent is feeling nervous but excited after having achieved what he had thought was his best.

After the program director introduces himself and the program, a personality test is administered to all of the applicants. Brent begins to feel apprehensive; why is this test being administered? He is worried that his chance of matching at such a program could be ruined if he doesn’t give the “right” answers to these questions that, as far as he can tell, aren’t even pertinent to medicine. Brent’s discomfort about the personality test rattles him to the point that it affects his mood and candor during the interview.

Upon returning home, Brent and his classmates discuss what they have encountered on the interview trail, from colorful interviewers to student faux pas. He tells them about the personality test: “I don’t see the importance of this personality test. Shouldn’t my application speak for itself? It makes me so mad that I might not match there because of how I did on some ridiculous personality test. Seriously, were they interviewing me to see if I’d make a good doctor, or to see if they wanted to date me?”

Commentary

In this scenario, Brent, a fourth-year medical student, is confronted with an unanticipated request to complete a personality inventory as a part of his interview day for a residency position. Fearing that a ten-item personality “test” may undermine his acceptance into a program that, based on his application, he is well qualified for, he laments that the interview questions should be sufficient. Unfortunately, as any program director will attest, the typical interview process is not all that effective in consistently finding the best fit.

Generally, applicants’ medical knowledge and patient care skills are well addressed in their clerkship performance and USLME scores, both generally well summarized in the medical student performance evaluation (MSPE) provided by the dean of students’ office. Much more difficult to determine are a student’s noncognitive qualities, such as conflict management, communication with ancillary staff, and professionalism. While clerkship grades are assumed to include an assessment of these attributes and some medical schools now provide a separate professionalism summary, these are difficult skills to assess well without specific training and focused attention. The halo effect, or the influence of one aspect of performance on the grading of others, undermines the validity of such ratings.

Does is it make sense to even try to assess these characteristics of applicants? Grades and standardized test scores should in theory be sufficient to identify qualified and competent applicants, and certainly meeting with faculty and residents during the interview process should reveal most serious potential problems. The value of data from these sources in predicting performance, however, is modest [1, 2]. Further, the medical education literature suggests that a significant minority of residents have serious problems during their training. The estimate was 7 percent of residents in one national survey of all program directors [3] and another, by the American Board of Internal Medicine, estimated that between 8 and 15 percent of residents had serious problems during training [4].

So why are interviews not sufficient for assessing these characteristics, and why do some programs to use personality inventories? For one, interviewing applicants for any high-stakes position requires specific skills that interviewers are not often taught and experience they don’t often have (faculty may interview only a few candidates in a year).

Second, commonly asked questions do little to reveal whether someone is a good fit with the program or will be successful. Third, the context of the interview is far removed from the setting in which the applicant will work. While interviewing can be stressful, it does not replicate the hospital environment’s multitasking, life-threatening emergencies, conflict, and fatigue. As a consequence, the traditional interview process provides little more than an opportunity for candidates to put on their best face and answer what they think the interviewer wants to hear and for the program to sell itself.

Aware of the large investment made in new hires, the business world has studied performance prediction for years. Despite the popularity of personality inventories, their validity remains controversial [5, 6].What has gained traction with a fairly robust evidence base is the behavioral-based interview (BBI) technique, grounded in the logic that past behavior predicts future behavior. Described by Janz in 1982, the BBI seeks to discover how the candidate acted in specific employment-based situations focusing on experiences, behaviors, skills and abilities that are job-related [7]. As opposed to asking hypothetical questions (e.g., “what would you do if…”), which allow candidates to provide the response they think is expected, the interviewer asks the applicant to discuss an actual experience and how he or she responded to it. For example, in looking at a candidate’s ability to manage conflict, he or she may be asked to “give an example of a situation in which you had difficulties or conflict with a team member and how you resolved it.” To explore a candidate’s ability to respond to negative feedback, he or she may be asked to “tell about a time when you were criticized for your performance and how you responded.” Further details of the behavior and outcome are elicited to facilitate understanding about the respondent’s motivation and the final outcome. Situations discussed may not be medically related but utilize relevant skills.

Interpreting and assessing behavioral interviews is a systematic process: the interviewer considers how relevant the skills described are to the job description, how recent the situation occurred and most importantly, what the applicant’s reasoning was, and what he or she learned from that experience [8]. An applicant’s description of what, in hindsight, he or she would have done differently may provide better insight into a candidate’s commitment to reflection and self-improvement.

While supportive research exists in the nonmedical literature on the reliability and validity of the BBI [7], there is a paucity of medical education literature on the subject. Only two studies look at the BBI in residency recruitment. One radiology program looked at the predictive ability of behavioral interviewing by comparing scores to program director’s assessments 4 years later, showing predictive utility for conscientiousness and interpersonal skills [9]. An anesthesia program looked at how this approach was received and found that the BBI process was acceptable to both the interviewers and the candidates [10].

Is behavioral interviewing going to match the perfect candidate with the perfect program every time? There is certainly no evidence to support that outcome at this time. Can it provide a better understanding of a candidate and make for a more interesting experience for both interviewer and candidate? Anecdotally, the experience of this writer and others says yes. Conversations that encourage thoughtful dialogue are much more likely to conclude with both participants more fully appreciating what the other has to offer. There is more to learn from discussing a specific experience that led to growth than from questions about why a candidate chose a given specialty.

Interviews are an important part of the application process. In the most recent survey by the National Residency Matching Program, program directors reported that an applicant’s interaction with faculty during the visit and interpersonal skills were among the top three factors contributing to his or her ranking [11]. A behaviorally based interview or personality test is not likely to undermine a candidate’s chance of matching. What is probably most important for a candidate to do on a visit to a program is to treat everyone (particularly program staff) with courtesy and respect. Program directors readily acknowledge that, each year, candidates are taken out of the pool simply for their poor treatment of staff during their visit. The feeling is if someone can’t behave respectfully during the one-day visit, how will he or she be over the next 3 to 5 years? Finally, showing genuine enthusiasm and interest will convince most interviewers that you could be a good fit in their program.