Academic biomedical research, by many measures, is not working very well right now. To be blunt, there just isn’t enough money to go around. 

We were talking about this a few weeks ago over lunch, and the solution, one of my colleagues said, is to just fund the best scientists. A widespread sentiment, for sure, and one that seems so obvious as to almost be banal.

But, in my mind, this statement exemplifies something pervasively wrong in our approach to fixing the biomedical system: for the most part, our thinking doesn’t seem very scientific.

Like most large enterprises, the biomedical system is a tough one to unpack, and so I’m going to split my thoughts into three separate posts. In the first (i.e. this) post, I want to talk about the trouble intrinsic to the phrase ‘the best’. In the second, I’ll consider the many different meanings we could use for ‘the best scientist’. In the third and final post, I’ll talk about the problems with trying to measure bestness, even once we have established just what precisely we mean by ‘the best’, and how this can lead to a cargo cult mentality, which ultimately undermines our goal of promoting ‘the best’ science.

Although I’m obviously focused on biomedical research, most of the main points apply elsewhere—college admissions, promotion criteria, hiring decisions, etc. Just pick your favorite.

But, of all of these, I know research the most thoroughly, and so that’s where I’m going to ground my discussion.

With that, let’s begin by thinking about ‘the best’.

I. ‘The best’? The best at what?!

Last fall, this image was going around—a series of photos taken by Howard Schatz showing all these different Olympic athletes next to one another. It is a fantastic image. I mean, the diversity of body types is just incredible.

What does that have to do with our current topic?

Well, let’s first consider, for instance, Tara Lipinski—gold-medal figure skater—and Lisa Leslie—4-times gold-medal basketball player. Lipinski is a diminutive 5’ 1”, 95 lbs, while Leslie is 14 inches taller, and 75 pounds heavier! Both are fantastic athletes, both went to the Olympics, both won gold medals.

And now let’s consider the question, Who is the best athlete?

I don’t even know how to answer that question. Should we talk about number of gold medals (Leslie)? Skating ability (Lipinski)? Hand-eye coordination (I’m going with Leslie for this one)? The answer we get depends completely on how we define ‘the best,’ on how we judge them, on how we weight their talents and accomplishments.

Not just that, but what the best looks like, and who is even in contention, is entirely dependent on how we decide to measure the best. I know I’m not the first person to observe that basketball players, on average, look a whole lot different than figure skaters: the host of skills you need to succeed in the two disciplines is entirely different. But part of the power of Schatz’s images is showing just how different athletes really are.

Yeah, sure, I can hear you saying, but this is a straw man argument,[1] because it’s not like there’s some Platonic ideal of the Best Athlete.[2] 

Fine. Point taken.

Let’s consider a far narrower question then: who is the best male runner? You have to grant me that this is a pretty easy category to define—we’re just looking for the guy who’s the fastest at getting from point A to point B.

But, already, we’re going to hit some difficulty because the answer depends entirely upon how far apart point A is from point B.

100 meters? Usain Blot (9.572 seconds, 2009).
1000 meters? Noah Ngeny (2:11.96, 1999).
10,000 meters? Kenenisa Bekele (26:17.53, 2005).
100,000 meters? Donald Ritchie  (6:10:20, 1978 (!!)).

All of them, world-class runners. All of them, insanely fast. (I mean, really, they are so fast that it is hard to comprehend: Ritchie averaged 5:57 a mile for just over 62 miles!). But who is ‘the best’?

Using the criterion of ‘the best’, even in the case of running, seemingly one of the simplest of disciplines, gives different answers depending on we define it. Even the type of runner in the running [3] for The Best Runner varies depending on the distance.

So what’s the take-home? When we’re looking for ‘the best’, our definition of ‘best’ is as much of a factor in the answer as anything else.

I understand why we can be hesitant to admit this fact. It feels uncomfortably close to saying that there’s an intrinsic bias in our judgement. Instead, it’s a lot easier to act as though changing the criteria won’t change the outcome.

But the truth is that it does. Our definition doesn’t affect Bolt’s records, but it does change how we weigh them.

To pretend otherwise, to pretend that there is some natural-law answer to this type of question ignores its own fundamental nature, and does a grave disservice to all involved, the evaluators and the contenders.

So who is the best scientist?

I don’t know, but I do know the first step. And that’s to decide what we mean by ‘the best.’

---

[1] I would point out, though, that ‘the best athlete’ is at least as problematic as ‘the best scientist.’ I’ll get into why I think that in Part 2.

[2] Not that this has stopped people from trying.

[3] Sorry, I just couldn’t help myself with that one.