Max Mednik
  • Home
  • About
  • Interests
    • Angel investing
    • Magic
    • Scuba Diving
  • Blog
  • Contact

Readings and musings

A for Effort in Research

5/30/2012

7 Comments

 
Picture
Some recent interactions with medical researchers and conferences I've attended have caused me to think about incentives in the field of research, and I'm quite worried.

First, I've realized how extremely finicky and sensitive the scientific process is. Final results can be significantly skewed in the "wrong" direction by variations in equipment, ingredient formulations, specific techniques, and parameters used. Many intermediate ingredients (cells, RNA, etc.) are available off the shelf, which seems convenient, but often has the risk of quite variable quality (I personally saw researchers re-ordering some RNA compound because twice they received something that failed to work as advertised).

The scientific process is complex, difficult, and still so labor-intensive. You would think that in the 21st century the "rote" work would all be outsourced to cheaper labor destinations and/or fully automated with machines, but that's still far from prevalent. Science is still being done in many of the most prominent research universities in a form that's closer to high school chemistry than to 2001: A Space Odyssey.

Secondly, I've frequently heard about "publish or perish" and the extreme focus on publishing positive, statistically significant results. This makes people care more about quantity than quality and on "proving" hypotheses right rather than disproving them or trying new techniques, even if they don't work. There are no rewards for failure, and you can't get a patent for trying. In professional science, as opposed to school, there is unfortunately no "A for effort." And I think we lose a lot of valuable information and create a lot of wasted time by duplicating techniques instead of sharing with each other by publishing things like, "I tried these 5,000 combinations. They didn't work. So if you read this, you might want to try something else."

This results in several complications to the pure pursuit of knowledge and improvement of the human condition. The best publications are peer-reviewed, and the "peers" are the ones competing with the authors for the same publication slots. The one-way anonymity (not double blind) means that people scratch the backs of their friends and form "societies" (which like to meet at conferences) that are really like old boys' clubs for cheering each other on and publishing each other's work.

Also, the focus on publishing creates so much published research that no one can follow it and keep track of it. I'm always shocked when I see scientific citations listing that an article was on pages 1,056-1,064. Who out there is reading thousand-page long journals? I see the same problem with patents: sure, publishing research and filing a patent make the knowledge accessible (when searching for it) but they don't make it prevalent and don't cue anyone to read the findings by themselves.

In addition, because of the drive to publish quantity and show "results" even when they're suspect, it drives the quality of research down, yielding false results. John Ioannidis at Stanford wrote about how too much medicine relies on flawed assumptions, explaining how most published research findings are false. The WSJ wrote several articles explaining how pharmaceutical companies are unable to reproduce most research findings (see above about scientific complexity and sensitivity to specific conditions and compounds).

It's like we're giving people prizes for trying something a thousand times until finally they get lucky enough (or are careful enough) to produce something scientifically significant instead of rewarding them for working hard and producing truthful results (and sharing their experiences either way).

I'm not trying to diss researchers or publications or universities. I know almost all the individuals are honest and extremely hard-working and do believe in the deeper goals of science. I just think the current system is sub-optimal, and I don't know how to fix it. I'm curious to hear what others think.
7 Comments
Francis Zver link
6/5/2012 10:21:09 am

Scientific method wasnt designed for finding breakthroughs, i thought it was designed for being able to replicate results as best as possible when breakthroughs are finally found.
But i do agree with you that it's somehow getting too complicated even to reproduce results reliably.

I dont particularly agree with "publish or perish" model, so I agree with your point. We should be able to publish when we're WRONG.
The best example is the loose cable with CERN. I think they handled it well, definitely telling people to double/triple check their "finding"
The talk by Brian Goldman on Doctors make mistakes, definitely applies to being open to sharing failures as much as successes.


I kindly disagree with the assessment that no one can follow or keep track of publications. If someone wants to spot a trend or track trends, the information can be organized.
It should be possible organize the information in a managable chunk.

Shouldnt showing results even if they're "suspect" be something we're ok with. I goes back to your point of allowing even "failures" to be public so that others learn from it, instead of wasting people's time?

So I say publish EVERYTHING, let everyone know what everyone else is doing... and from there we can all learn from each other.
AND also develop a way to track trends.
Trends in methedologies
trends in equipment used
trends in areas researched

so with these analytics, we should be able to spot what we're not seeing, by analyzing what we spend too much time staring at.
From there... we can progress in a more, direct route. Hopefully, getting to some actual, usable, repeatable, goal.


But... i maybe wrong :-)

Reply
Max
6/5/2012 10:51:19 am

Thanks for the comments and good points. I'm intrigued by your idea of analytics and metrics to help people really absorb new research more efficiently and effectively.

Reply
Francis Zver link
6/6/2012 02:54:39 am

I don't particularly use analytics and metrics in my daily life, but I've watched how it helps people and companies identify interesting trends that have helped them break out of any falsely believed facts.

One example was a presentation I saw at last month's OWASP meeting. A person on Vericode presented on how they manage the ridiculous amount of data they track. And from that data and tracking trends, they realized that even though more devs and people believe SQL vulnerabilities were the most dangerous, the trends from their data show that cross-site scripting actually was the greater danger.

That's just one example, there are others, but I cant really recall them off the top of my head right now. But as I see how my own company I work for (specifically my managers) handle giant amounts of data, and how other teams do it, a simple truth arose... having too much data is not a hindrance or bad thing in itself... the possible hindrance is the lack of understanding on how to handle all that data.

Interesting blog btw. I'm trying to start my own. Any suggestions on how to integrate blogging into one's life?

Francis Zver link
6/6/2012 02:54:47 am

I don't particularly use analytics and metrics in my daily life, but I've watched how it helps people and companies identify interesting trends that have helped them break out of any falsely believed facts.

One example was a presentation I saw at last month's OWASP meeting. A person on Vericode presented on how they manage the ridiculous amount of data they track. And from that data and tracking trends, they realized that even though more devs and people believe SQL vulnerabilities were the most dangerous, the trends from their data show that cross-site scripting actually was the greater danger.

That's just one example, there are others, but I cant really recall them off the top of my head right now. But as I see how my own company I work for (specifically my managers) handle giant amounts of data, and how other teams do it, a simple truth arose... having too much data is not a hindrance or bad thing in itself... the possible hindrance is the lack of understanding on how to handle all that data.

Interesting blog btw. I'm trying to start my own. Any suggestions on how to integrate blogging into one's life?

Francis Zver link
6/6/2012 02:55:00 am

I don't particularly use analytics and metrics in my daily life, but I've watched how it helps people and companies identify interesting trends that have helped them break out of any falsely believed facts.

One example was a presentation I saw at last month's OWASP meeting. A person on Vericode presented on how they manage the ridiculous amount of data they track. And from that data and tracking trends, they realized that even though more devs and people believe SQL vulnerabilities were the most dangerous, the trends from their data show that cross-site scripting actually was the greater danger.

That's just one example, there are others, but I cant really recall them off the top of my head right now. But as I see how my own company I work for (specifically my managers) handle giant amounts of data, and how other teams do it, a simple truth arose... having too much data is not a hindrance or bad thing in itself... the possible hindrance is the lack of understanding on how to handle all that data.

Interesting blog btw. I'm trying to start my own. Any suggestions on how to integrate blogging into one's life?

Max
6/6/2012 03:48:01 am

Good point on the handling of data vs. simple quantity of data.

Yeah, blogging takes work and time. I just set up a reminder to try to do it every 2 days, and it helps to create a backlog of topics/posts that you write when you feel creative or have content to share and which you can distribute over time. Tools like Weebly/tumblr make this easy. Good luck!

Reply
Helen
6/7/2012 04:42:34 pm

I can't agree with you more on the current publication system. What I heard from my life science colleagues is that it must be solid stories that can be published on high-profile journals. But "to tell a good story", how many negative data are omitted either intentionally or not? If all the data at the negative side of the story can be disclosed, I would expect to see they bring about a completely new "story" and improve the efficiency of scientific discovery. Another underlying question is that can the complex biological system really be interpreted as stories? I don't think it is the only way or best way to present outputs of scientific research.
One idea to fix this system which my postdoc friend have is to establish a website sharing every experiment online with everyone in the life science community or even the public. ( I should ask for her permission before saying more. ;) she may plan to really work on building this web.)
Now I don't have constructive ideas as solutions myself, but as an individual researcher, do aim to have my publication not just to avoid perishing but try to make some contribution no matter how small it is.

Reply

Your comment will be posted after it is approved.


Leave a Reply.

    Archives

    July 2024
    June 2024
    May 2024
    March 2024
    February 2024
    January 2024
    November 2023
    August 2023
    June 2023
    May 2023
    April 2023
    February 2023
    January 2023
    December 2022
    November 2022
    October 2022
    September 2022
    August 2022
    January 2022
    October 2021
    August 2021
    July 2021
    June 2021
    May 2021
    March 2021
    February 2021
    January 2021
    December 2020
    November 2020
    October 2020
    September 2020
    August 2020
    July 2020
    April 2020
    January 2020
    December 2019
    November 2019
    October 2019
    September 2019
    August 2019
    July 2019
    May 2019
    March 2019
    January 2019
    December 2018
    November 2018
    October 2018
    September 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    February 2018
    January 2018
    November 2017
    October 2017
    September 2017
    May 2017
    April 2017
    November 2016
    October 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    December 2015
    November 2015
    October 2015
    September 2015
    August 2015
    July 2015
    June 2015
    May 2015
    April 2015
    March 2015
    February 2015
    January 2015
    December 2014
    November 2014
    October 2014
    September 2014
    August 2014
    July 2014
    June 2014
    May 2014
    April 2014
    March 2014
    February 2014
    January 2014
    December 2013
    November 2013
    October 2013
    September 2013
    August 2013
    July 2013
    June 2013
    May 2013
    April 2013
    March 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    September 2012
    August 2012
    July 2012
    June 2012
    May 2012
    April 2012
    March 2012
    February 2012
    January 2012
    December 2011
    November 2011
    October 2011
    September 2011
    August 2011
    July 2011
    June 2011
    May 2011
    April 2011
    March 2011
    February 2011
    January 2011
    December 2010
    November 2010
    October 2010
    September 2010
    August 2010
    July 2010
    June 2010
    May 2010
    April 2010
    March 2010
    February 2010

    Categories

    All
    Angel Investing
    Cacti
    Cars
    China
    Community Service
    Culture
    Design
    Djing
    Dogs
    Education
    Entertainment
    Entrepreneurship
    Family
    Finance
    Food
    Google
    Happiness
    Incentives
    Investment Banking
    Judaism
    Law
    Lighting
    Magic
    Marketing
    Medicine
    Networking
    Nolabound
    Philosophy
    Professionalism
    Psychology
    Reading
    Real Estate
    Religion
    Romance
    Sales
    Science
    Shangri-La
    Social Entrepreneurship
    Social Media
    Sports
    Teams
    Technology
    Travel
    Turtles
    Ucla
    Venture Capital
    Web Services
    Weddings
    Zen

    Subscribe

    RSS Feed

Picture
Picture
  • Home
  • About
  • Interests
    • Angel investing
    • Magic
    • Scuba Diving
  • Blog
  • Contact