“Look at every path closely and deliberately, then ask ourselves this crucial question: Does this path have a heart? If it does, then the path is good. If it doesn't, it is of no use.”
~Carlos Castaneda

Pages

Sunday, January 6, 2008

Are we too nice?

“High achievement always takes place in the framework of high expectation.”
~Charles F. Kettering

When I went through all of our applications, it was hard to separate one application from another. The white pages and black lettering blended together after the about 15th application. For a majority of the applications, all I had was a name, school, and their basic statistics.
John Doe
  • Medical School: State University of X or X University
  • USMLE step 1: 235
  • Clinical Rotations: 1/2 clinical honors
  • Orthopaedic Rotations: honors
  • LOR's: good to excellent with some stating he is in the top 10% of students rotating this year
  • Personal Statement: "... Since I injured my knee playing football, I have always wanted to be an orthopaedic surgeon. ... I have played sports all of my life ..."
I can't count the number of times that I have read this type or similar application. The names of the applicants are sometimes interchangeable. What aspect of their application tells me that this is going to be a stud or a dud? I have received a number of emails and have read plenty of forums that ask the same question, "what do I need to get into orthopaedics?" If I could tell you the exact recipe, I would, but I don't think there is one. In truth, we all know the recipe, good grades, good scores, good rotations, and a little luck. I think what is more important is the special ingredient or special sauce you bring to make you different.

Many applicants get caught up in the numbers of the game. Time and time again the question is asked, "what score do I need to get into orthopaedics" or "what grades do I need" or "how much research do I need"? I can give numbers of the typical solid application, as I have listed some above, but does that guarantee a spot? Not so much. The quoted figures on scores, grades, research, and AOA are just guidelines, not a guarantee. Every year there are applicants with strong numbers that don't get into a residency. So, why didn't they match? Typically, when I have reviewed these non-match applicants, the reason was either glaringly obvious (USMLE score 205, bad letters, failing a subject in medical school) and on other occasions, it was not. When there was no glaring flaw, the applicant looks like every other applicant. There is nothing in their application that makes him/her stand out.

I have asked myself why is it that that most of the applicants so similar. Is it because they have all used performance enhancing drugs to make them all academic superstars? All jokes aside, I think it may have more to do with the way we grade. I blame it on kids soccer, where everyone gets to play and in the and they all get a trophy. Many parents display their honor role student sticker on the rear window of their SUV. All the children are A and B students. Many go to a 4 year university and expect the same. They argue for a better grade and petition for grade changes when it doesn't meet their expectations. The professors that grade on a true Bell Curve are not liked or considered "hard" because they give out fewer A's and B's. Over the past 10 years, I have noticed this trend and I have wondered if we (educators) are too kind in our evaluations?

With a student's application, we receive a copy of the grade distribution for that medical school. When comparing applicants from one school to another, there are definite differences in grading philosophies. I don't know if this is a problem with the grading set up or that we are "too nice." In my experience of clinical grading, unless the student does something drastically wrong (like never show up or cursing out a patient), the student will at least receive a pass. The question is who receives the highly coveted Honors. Each School varies in their grading system. The grades can range from pass/fail only to honors/pass/fail to honors/high pass/fail to honors/high pass/marginal pass/fail and the always popular A/B/C/D/F (with +/-). What puzzles me about all of these systems is that the average tends to be a B or high pass. There are schools with greater that 50% honors in some subjects. You may say, well are these the "lesser schools"? Not so fast young patawan. In my limited research (okay not really research but observation), it is more common for the "very competitive schools" to have more of a top heavy grade distribution and the "less competitive schools" to have a more even grade distribution. It is not uncommon for a school to have grading distribution (in the clinical years) with greater than 50% honors and less that 30% passes. How does this allow for us to assess these applicants? If you score only gives out 20% honors and you received a high pass, should you be penalized? On the other hand, if you went to a school that gave greater that 50% honors, should you be given bonus?

With competitive specialties' concentrating on USMLE scores, students have been crushing this test. The USMLE is one of the only tools we have to compare applicants from different schools and areas of the country. Because the USMLE "powers that be" don't want the test to be used in the manner we use it, they do not provide us with the distribution of scores. In the old days, the mean was in the low 200's (205 when I took it) with a standard deviation of 20. Today, the mean is in the mid 210's. Therefore, a score in the 90's of 225 is equivalent to a score of about 235 in today's scoring (I am guessing). Most of the applicants I have reviewed have an average of a 230 (just a guess, again no true data). Again, when trying to create separation like Randy Moss from a corner back, it doesn't happen. The applicant's are all bunched together like 6 year old children playing soccer.

You may say, "then look at the letters of recommendation (LORs)." This is less helpful than the grading. Most folks have the prerequisite letter from their program chair that says he or she is a supernova or has star like qualities. There are usually 1-2 letters from surgeons that are not known by most interviewers and 1 from a well known surgeon. Although the letters are helpful when pointing out top end and lower end, they to not create the needed separation to differentiate one applicant from another. There have been occasions where I have read the same recommendation on 2 or 3 applicants from the same physician. Although we think we know the code words, I think we kid ourselves at thinking we can read into another's recommendation like it is Morse Code. Usually the true meaning is missed, except when comments are blatant like, "we recommended that he look into other specialties ... "

What is the answer? I have recently begun to reevaluate my own grading system. How is my grading? Am I too nice? The answer is yes. I believe that many of us don't want to be the bad guy. Who wants to be the professor who fails most of his/her students? I don't think that there are many who would answer yes. I believe we do need to re-center. In the clinical setting, the average grade should be a pass. The excellent grades should be give to those who truly stand out for the rest of the students. As an educators, we must communicate our expectations are and explain what passing grade means. Is this a student problem, I would propose it is not. It is a educator problem. We have evaluate honestly. No more just checking the 4 out of 5 box. If they have met expectations, then they should get a pass. You may read this and think I am arguing for more strict grading, but I am not. I think that our grading should be fair. Lumping the average around above average is not fair to those who are truly above average.

“Success is simple. Do what's right, the right way, at the right time.”
~Arnold H. Glasgow

10 comments:

  1. While I agree with some of your points, I think your perceptions are clouded by the fact you are looking at orthopaedic applicants who are by and large the cream of the crop. If you could look at the average or even star applicants to a psychiatry or family medicine residency, you would see quite the difference. A single honor would make many of them stand out, along with a Step I score above 210.

    While certain applicants will certainly make better residents and surgeons, for the most part there are not many truly bad ortho applicants, I'll put the 10 random ortho applicants up against even 10 random gen surg applicants any day in terms of residency performance. Who I feel really bad for is the attendings in the less desired fields who really have to find diamonds in the rough.

    ReplyDelete
  2. anon: thank for your comment. i think you are right. we do have a skewed sample. that being said, when i speak to my college professor friends, they do have similar opinions. may be it is a generational thing, i am not sure.

    in our program, we look for the diamonds in the rough. we look more for a fit than a number. i think by the numbers many orthopaedic applicants blow away those from other specialties, but the question is who will do well in your program. who has the complete package?

    the sad thing is to see good applicants not match. why? usually because they didn't stand out enough.

    ReplyDelete
  3. Your statement is so true. Lumping everybody around the above average mark does a real disservice to those who are truly above average.

    ReplyDelete
  4. Hmm… this entry frustrates me (due to it’s accuracy). Of course, there is a problem with the grading system. However, I’m not sure the answer to rectify it is as clear as “grading harder”. There are only 5 grading categories. Due to the shear quantity of students you will only “shift” wherein the majorities lie. Yes, it is true one would elicit the few that are truly gifted, which emphasis of few. Because these few are very sparse, you will now have to discern from equally large group, just with lower scores. What has this accomplished?

    ReplyDelete
  5. I don’t think I read into your entry and equated grading fair to grading harder. If one as an individual starts “grading fairer” this would perceived to others “as grading harder” and in the long run doing your students an injustice. Their passes compared to others who have honored (graded by someone else) might cause them to be overlooked for a slot. Would that be fair?
    anon#2

    ReplyDelete
  6. I see your point. I also want to apologize. It’s been an arduous week, and my comments above might have seemed a bit harsh. I look forward to reading your blog entry’s when time permits. Keep up the good work.
    anon #2

    ReplyDelete
  7. The problem with grading harder is that the separation between the students you speak of is so small that the grade distribution would be large even though the ability and achievment distribution is so small. If you have an exam with an average of 85 but a std dev of 4 then it is obvious that the students are all lumped together. The difference between an A (one std above) and a C (one std below) is little more than a few questions. In a situation with such a tight distribution how can you make the difference between a C and an A so small? Additionally, should a med student attending a tougher med school with more intelligent students lose all his honors when he could have gone to a less competitive med school and had a greater chance at achieving higher scores? If inflation were removed, the quality of med school attended would have to be next to Step 1 score for determining factors. Thus, the residency program would have to accept students with lesser marks at greater schools. When a residency program lists the statistics for their acceptances this clarification will not be present and their stats will appear worse. Additionally, a medical school that grades harshly while other inflate will not be a popular choice for undergrads. A medical school that eliminates inflation would also appear worse statistically if the rest of the medical schools did not stop inflating grades. I believe the solution lies in the best medical schools initiating the elimination of grade inflation. The trend can only start in a top down fashion because the prestige and name recognition of these medical school will allow them to overcome the period when they are the only schools without grade inflation.

    ReplyDelete
  8. anon: i don't think i said we should grade harder, we should grade fairer. meaning average should be average. the mean grade should not be a B. If things are to be graded on a curve, it should be a normal distribution.

    as far as honors go, for most of th so called less competitive schools is is harder to get honors than the so called more competitive schools. there are top 10 schools who give out greater that 40% honors on some rotations. is that fair to the less competitive school that gives only 20% honors?

    ReplyDelete
  9. I propose actually reporting usmle scores as strictly pass/fail. Then, much like the corporate (real) world candidates would be interviewed based on demonstrated commitment to a field as well as skill set. We are to awash with applications to not do this. Screening candidates based on numbers and scores does a disservice to the system as students are more focused on numerical achievement rather than personal and intellectual growth, growth that would serve everyone once in a residency and beyond.

    ReplyDelete