Fitter, Happier

“See if you can fit it on the paper
See if you can get it on the paper”

 

There were several discussions of scientific “productivity” on Twitter yesterday. It’s long been clear to me that people have wildly different ideas about what this means and how to measure it. Many times you find people talking about how many papers a scientist has published, but does anyone seriously think that that is a useful number? One major factor is that individual researchers and communities have dramatically different ideas about what constitutes a publication unit. I remember being very annoyed when my first grant, which was directly based on my postdoctoral work, was reviewed with a ding that it was based on “a single publication.” Setting aside the fact that I didn’t invent a whole field and there was long literature preceding me, why is that in and of itself a neg? That was four years of work done entirely by me. I probably could have portioned out some number of smaller nuggets and published them separately, but why is that a good thing?

So I was interested in this exchange that came in a larger discussion of standards for review of NIH grants:

Screen Shot 2016-05-12 at 9.41.23 AM

In a strict sense, Drug Monkey is right because science is never complete, but his argument is really a straw man. We can’t pretend that all papers are anything close to equal in terms of scientific productivity. And to head off an inevitable response, I am not talking about Glam. I am also not talking about middle vs. first author papers. It is absolutely the case that first author papers can reflect a wide range of what we deem to be productivity. In my opinion, at the extreme that range may even plausibly span an order of magnitude.

My attitude is that it is more efficient and better for science to publish your data in larger chunks, but I understand that many people feel differently. I’m interested in hearing from people in the comments. Given the same data, what is the argument for splitting it up? How do you know when to stop and publish something?

 

Advertisements

20 thoughts on “Fitter, Happier

  1. The argument is that science is incremental. Done properly, we build upon others’ work. To do this we need to know about that work.

    • I see your point. One has to tread a line between hoarding data and putting out an incomplete or preliminary scientific thought. The problem you raise is a major reason for #pr33ps. A paper may take almost as long to publish at it took to perform the research.

  2. Every paper is an incomplete scientific thought. If you think otherwise you are just fooling yourself.

    There is no argument in favor of preeps. In fact the opposite. If you find value in preeps, you are making my argument for me. All else is selfish gatekeeping.

  3. I have yet to publish a paper as an independent PI, but for me it comes down to my personal preference for what I find interesting. We have some behavior data from some mutant mice now that could be a paper, but I think it would be a boring paper! Sounds like DM is saying we should have one paper on the behavior, one paper on the circuit wiring and one paper on the RNA-seq results, whereas to me these results are all part of one “story” about what happens when gene X is knocked out. It’s my bias, but an “embarrassing fiction”? Really?

    • Your personal preference is one thing. I don’t care how you choose to parse your own stories all that much (beyond the broader issues of not publishing as much data as possible in timely manner). When someone starts in on how their personal preferences amount to a “complete story” and someone else’s preferences do not, well, this is when I start calling you on your bullshit.

    • With that said, JT, I have a question. Is your main interest in knowing the “story” you outline? or is your real interest in you, personally, being the one to discover and/or tell this “story”?

  4. @drugmonkey: Just because there is such a thing as a complete sentence, doesn’t mean you are done talking. There is a reason people who think we should just put out undigested Tumblr feeds of data are misguided. I think you must agree that scientific communication is quantal, so we’re really debating about the size of a quantum. Pointing out that there is still more science to be done is a distraction.

    WRT preeps, I am honestly not following you. If I understand, you see this as a way for people to have their cake and eat it too? People can use them to mitigate the cost of humping glam? I think this should be considered a legitimate channel for scientific publication precisely because that is a waste of time. What do you mean by selfish gatekeeping?

    • A “complete sentence” in this scenario is defined by the fact of publication in a journal, full stop. In an acronym, the LPU. Anything you choose to insert beyond this is your attempt to claim your personal definition of the King’s English rules. It doesn’t. Not in logic and not in practice. This is by no means a “distraction”. The conceit of a “complete story” has produced real negative effects on the conduct of science, mostly having to do with the percentage of work conducted that ends up published and the speed of it being made available to peers. This makes your “complete story” the true distraction from the enterprise of advancing scientific understanding as rapidly as possible.

      People can use them to mitigate the cost of humping glam?
      Exactly. Thereby propping up the Glam Humping business. And continuing the gatekeeping whereby the well-resourced beat the poorly-resourced.

      Look at it this way- suppose a limited finding can be completed equally well by SmallPotato and BigSchwanger labs. It is in the interest of BigSchwanger to define the publication racket in such a way that SmallPotato cannot compete with them on the highly expensive and effortful additional experiments to “complete the story” or “show mechanism” or whatever embarrassing fiction you happen to subscribe to. BigSchwanger wins by default, not by any particular merit, if they can define the game in their own favor. It keeps SmallPotato out of their Glamourous garden. It intimidates other labs from even asking certain research questions. It monopolizes further resources. ….and on the cycle goes.

  5. In my scientific upbringing, one of my mentors strongly encouraged publishing as frequently as possible, whereas the other was much more inclined to hold onto data until the story was “complete” (by which I mean deemed worthy of a high-impact publication by professional editors + peers). The papers I published “as I went” have thus far been cited just as well as those we held onto until we got into higher IF journals (though it’s still too early to know on the more recent pubs whether that holds).

    My suspicion is that in the “hold on” cases, the sheer amount of data that we had to wrap into those papers made them a bit less approachable. The “as I went” papers had the benefit of allowing a highly technical approach (required for the generation of new types of data) to be more methodically communicated. That approach probably reduced impact as judged by editors, but probably increased impact on peers.

    In my own lab I am trying to walk the line between these two approaches. In order to be considered worthy of promotion I need both quantity and quality so that my C.V., on paper, meets my institute’s standards. Aside from just the pressing issue of promotion, I think a balanced approach will be the best way for my lab to be seen as a valuable contributor to the field.

    • allowing a highly technical approach (required for the generation of new types of data) to be more methodically communicated.

      This is another important point. The depth with which a given individual aspect of the study can be communicated, even given essentially unending space for Supplemental Materials, matters. In real science, you just never know what aspect of your study you find uninteresting but will be absolutely critical later for some other lab to interpret or troubleshoot their own work. From methods to results. If other labs have to retrace your footsteps because it wasn’t worthy of your “complete story” or triage of what “demonstrates mechanism” vs boring detail, well, this is highly inefficient and expensive for the overall enterprise.

      • Indeed. This is, for me, the most dissatisfying component of publishing “hold on” papers. The standards to which manuscripts are held at many higher IF journals requires leaving out details that give depth to the scientific process. It’s not all about experiments that are either extremely revealing or that “answer” a question. Sometimes it’s about hammering through a bevy of alternative alternatives until you find traction. Anymore, these kinds of additional experiments get relegated to the Supplemental Material (at best), a line in the text (most likely) or not mentioned altogether (at worst). In the end, some manuscripts are much better (for the field, and for the scientific process) when they contain both the revealing and the mundane, because in that extended story lies the heart of the endeavor… where did your time, money, and energy go when producing the knowledge in your manuscript?

        But! I also feel that publishing whatever the heck data you have in a folder somewhere because it can be shoehorned into a LPU is a real risk of the “as you go” approach. It’s hard to teach students lessons when the goal is to generate just enough data to hint that maybe your hypothesis might be correct. It can lead to blind alleys, disjointed conclusions, and overall drudgery. It’s a real risk and I think sometimes the rewards of adding to the CV with “as you go” pubs (though they remain smaller than high IF pubs) can keep labs (and fields) spinning wheels without gaining traction.

  6. I personally think that getting the data out faster is better. Papers with years of data, many authors, and a million figures within figures are unwieldy and, in my experience, less likely to be read by anyone not immediately in that subject area.

  7. Pingback: Complete stories, demonstrations of mechanism and other embarrassing fictions | Drugmonkey

  8. The problem here is the journals. From my recent experience trying to publish an incremental piece of work, we have constantly been hammered by reviewers and editors who see the work as ‘too preliminary’ and/or ‘lacking mechanistic insight’. Come back when you have animal data? Come back when you can tell me the mechanisms? I have another paper ready to go on a different topic that I know will face the same issues. Even average journals (I’m looking at you JBC!!!) now are seeking glam in such a way that more is required mechanistically to get it in. At EMBO, I would perhaps expect that; at Biochem J, I certainly wouldn’t. There is delusions of grandeur at some journals I think.

  9. Not much to add to this discussion except to note that a person in my department just went up for promotion with zero publications from her pre-tenure period, because she was hoarding everything for the Great Nature Paper that never materialized.

    Protip: don’t do that.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s