The Path from Telescope to Press Hit & More
What is the path from telescope to lauded discovery for an astronomical result, and what happens afterwards?
Let's assume you're an observational astronomer. Your observations are made and the data are unprecedented in quality. You reduce the data and find an exciting and important result. The signal is so strong that you have no doubts about its reality, so extra statistical tests aren't required. You can write a paper immediately and submit it to a leading journal. The referees of the paper suppress their jealousy about you - their competition - making this discovery & quickly accept the paper. Your colleagues and head of department are thrilled and a press conference is organized by your appreciative public information officer. The press embargo remains intact, and the commentary written by an independent expert provides glowing praise. The story is a big hit with the press and public and everybody is happy, apart from people quarreling in comments sections about whether or not this result proves the existence of God. You are especially edified by the strong and unanimous support of your scientific colleagues. The result is now established as SCIENTIFIC FACT and you and everybody else goes on to study other objects.
If you think this description is far-fetched, you're right. This is not how discoveries in astronomy proceed, and that's a good thing. As an example, there is the evolving story of the possible planet around the nearby star Alpha Centauri B.
The Nearest Exoplanet to Us?
Many will recall the announcement, back in October of last year, of the discovery of Alpha Centauri Bb by Xavier Dumusque and colleagues. As the nearest exoplanet to us, this result generated a lot of media attention, as Paul Raeburn explained in a blog post at the Knight Science Journalism Tracker. I also wrote a detailed blog post about the media coverage.
How well does this discovery conform to the clean scenario given above? I'm not sure whether their data was unprecedented in quality - although it probably was - but the Nature paper by Dumusque et al. (not yet freely available on the arXiv) reports that their signal was not so strong that extra statistical tests weren't required. This is reality check #1 for my imaginary scenario given above. The signal they claim to see is a periodic variation in the radial velocity (RV) of the star Alpha Centauri B, caused by the pull of an orbiting planet, but there are other, much larger variations present. Using a simulation, they performed one statistical test of the procedure used to search for the periodic signal, and their method passed the test.
I have no insider information about the refereeing process for this paper, but I would be happy to hear from any referees or authors who are willing to share! I do know that when the publication of the paper approached the embargo was broken. Reality check #2. Also, a skeptical News and Views article by Artie Hatzes - an astronomer not associated with the work - was published in Nature alongside the science paper, leading to an appropriate level of skepticism in much of the press coverage, as I discussed in detail. Reality check #3.
Doubts in New Analysis
The reaction from astronomers to the paper has been mixed, for reality check #4. Hatzes has now written a paper, accepted for publication in The Astrophysical Journal, based on analysis of the same velocity data used by Dumusque et al. Using one test Hatzes saw the 3.24 day signal claimed in the Nature paper, but at a lower statistical significance than reported by Dumusque et al. Using a second test that should be just as sensitive Hatzes was unable to see the 3.24 day signal. In the abstract Hatzes says "It may be premature to attribute the 3.24 day RV variations to an Earth-mass planet."
For more details about the Hatzes paper and reactions to it from astronomers, please read this article by Mark Zastrow in Sky and Telescope and this article by Dennis Overbye in the New York Times. Both are excellent.
On a more positive note, Katherine Rosenfeld, an astronomy grad student at the Harvard-Smithsonian Center for Astrophysics (CfA, my institution), repeated the analysis of Dumusque et al and found a similar signal, although she did not perform a rigorous analysis of the signal's significance. I expect there are other astronomers who have also analyzed the data of Dumusque et al.
This quick reaction from astronomers shows the intense interest in exoplanet research, a sign of a healthy field. Astronomers aren't just pulling out the "extraordinary claims require extraordinary evidence" cliche, they're checking for themselves how strong the evidence is.
This is science in action and kudos to Zastrow and Overbye for writing about it, given that editors are more likely to prefer discovery stories, even at speciality publications like Sky and Telescope. I expect that many readers will appreciate seeing this look at the less sensational aspects of research, marked by uncertainty, debate and patient cross-checks. A crucial factor was that the velocity data were made public when the paper was published, and credit goes to Dumusque et al. and/or Nature for this display of open science.
It's not rare for exoplanet claims to be disputed. Overbye points out the example of Gliese 581g* and doubts were raised about the famous planet 51 Peg until the challenge was retracted (in this case Hatzes helped make the case that the planet was real).
[* Note added June 28th, 2013: Xavier Dumusque kindly read this post and provided a couple of comments where he disagreed with me. Dumusque writes:
Dumusque moved to CfA recently, and I talked to him about the Hatzes paper and his plans. It's clear from Zastrow's article that Dumusque is reacting to the Hatzes paper in a professional way and I confirmed that in person. Having seen the way some people react to criticism, I'm just as impressed with Dumusque's gracious acknowledgment of the Hatzes paper as with his original achievement of digging the possible planet signal out of considerable noise.
More Data Are Needed
Regarding future observations, and reality check #5 (the most obvious one), Dumusque has 18 orbits of Hubble Space Telescope time to look for transits of Alpha Cen B by the possible planet. A detection of a transit would clinch the case for a planet, but the authors estimate there's only about a 30% chance of seeing one, assuming a planet is present, because of the need for orbital alignment.
Dumusque has already obtained more velocity data, a crucial step because longer observations will inevitably lead to a firm detection of the 3.24 day signal, if it's real. He is also interested in trying different analysis techniques to strengthen the case for a periodic signal. There can be tension between the desire to publish an exciting result in a timely way and the urge to be ultra skeptical by carrying out multiple tests. Sometimes it's not clear when you should stop.
I have some experience at seeking weak signals amongst a lot of noise. More than once colleagues suggested that I limit the number of tests I carry out after seeing a tantalizing signal, leading me to think "really?" and "don't tell me what to do!". When you do decide to pull the trigger and write the paper, it's important to be clear about what you've done, as I think they have been.
Describing Marginal Results
One question that arises is whether the Dumusque paper fully communicates the challenges involved in their planet search. Overbye's article says:
For the moment Alpha Centauri Bb should be viewed as a planet candidate. This isn't bad news, since NASA's Kepler satellite has, at the time of writing, 3,216 planet candidates and only 132 confirmed planets.**
[** Note added June 28th, 2013: Xavier Dumusque's 2nd comment is:
The Real Story is Unfinished
The real story deviates in several key ways from the clean scenario I imagined at the beginning of this post, but I think it's more interesting. Crucially, Dumusque et al. and their critics have behaved well and are moving on. Maybe the data that will clinch the case for Alpha Centauri Bb's existence is already sitting on a computer disk, waiting to be incorporated into one long time-series. Or maybe not. I have no doubts that Alpha Centauri B and its bright companion will remain popular targets for exoplanet searches no matter what is found in the near future.
Much of the excitement about the initial announcement concerned the potential for us to travel to this star system. Remember that this planet - if real - is a roaster, since it's much closer to its star than Mercury is to the Sun. Even if Alpha Centauri Bb is real there is currently no direct evidence that Alpha Centauri B has rocky planets in the habitable zone, only statistical arguments. Maybe by the time we find evidence for such planets, if they exist, we'll be ready to start planning that long trip. I won't try to predict what developments will occur in spaceflight technology, but I will predict that the story of Alpha Centauri's exoplanets is far from done.
Let's assume you're an observational astronomer. Your observations are made and the data are unprecedented in quality. You reduce the data and find an exciting and important result. The signal is so strong that you have no doubts about its reality, so extra statistical tests aren't required. You can write a paper immediately and submit it to a leading journal. The referees of the paper suppress their jealousy about you - their competition - making this discovery & quickly accept the paper. Your colleagues and head of department are thrilled and a press conference is organized by your appreciative public information officer. The press embargo remains intact, and the commentary written by an independent expert provides glowing praise. The story is a big hit with the press and public and everybody is happy, apart from people quarreling in comments sections about whether or not this result proves the existence of God. You are especially edified by the strong and unanimous support of your scientific colleagues. The result is now established as SCIENTIFIC FACT and you and everybody else goes on to study other objects.
If you think this description is far-fetched, you're right. This is not how discoveries in astronomy proceed, and that's a good thing. As an example, there is the evolving story of the possible planet around the nearby star Alpha Centauri B.
 |
| A beautiful image of the Southern Cross and the Pointers by talented astrophotographer Phil Hart. Alpha Centauri (containing Alpha Centauri B and two other stars) appears as the bright, yellow-tinted star on the left of the two pointers. The Southern Cross is very familiar to Australians - like Phil and me - because it is not only prominent in the Southern sky, but it's also on the Australian flag. Credit: Phil Hart,www.philhart.com |
The Nearest Exoplanet to Us?
Many will recall the announcement, back in October of last year, of the discovery of Alpha Centauri Bb by Xavier Dumusque and colleagues. As the nearest exoplanet to us, this result generated a lot of media attention, as Paul Raeburn explained in a blog post at the Knight Science Journalism Tracker. I also wrote a detailed blog post about the media coverage.
How well does this discovery conform to the clean scenario given above? I'm not sure whether their data was unprecedented in quality - although it probably was - but the Nature paper by Dumusque et al. (not yet freely available on the arXiv) reports that their signal was not so strong that extra statistical tests weren't required. This is reality check #1 for my imaginary scenario given above. The signal they claim to see is a periodic variation in the radial velocity (RV) of the star Alpha Centauri B, caused by the pull of an orbiting planet, but there are other, much larger variations present. Using a simulation, they performed one statistical test of the procedure used to search for the periodic signal, and their method passed the test.
 |
| The radial velocity technique for detecting an exoplanet. Credit: http://astronomyonline.org/Exoplanets/ProfessionalDetection.asp |
I have no insider information about the refereeing process for this paper, but I would be happy to hear from any referees or authors who are willing to share! I do know that when the publication of the paper approached the embargo was broken. Reality check #2. Also, a skeptical News and Views article by Artie Hatzes - an astronomer not associated with the work - was published in Nature alongside the science paper, leading to an appropriate level of skepticism in much of the press coverage, as I discussed in detail. Reality check #3.
Doubts in New Analysis
The reaction from astronomers to the paper has been mixed, for reality check #4. Hatzes has now written a paper, accepted for publication in The Astrophysical Journal, based on analysis of the same velocity data used by Dumusque et al. Using one test Hatzes saw the 3.24 day signal claimed in the Nature paper, but at a lower statistical significance than reported by Dumusque et al. Using a second test that should be just as sensitive Hatzes was unable to see the 3.24 day signal. In the abstract Hatzes says "It may be premature to attribute the 3.24 day RV variations to an Earth-mass planet."
For more details about the Hatzes paper and reactions to it from astronomers, please read this article by Mark Zastrow in Sky and Telescope and this article by Dennis Overbye in the New York Times. Both are excellent.
On a more positive note, Katherine Rosenfeld, an astronomy grad student at the Harvard-Smithsonian Center for Astrophysics (CfA, my institution), repeated the analysis of Dumusque et al and found a similar signal, although she did not perform a rigorous analysis of the signal's significance. I expect there are other astronomers who have also analyzed the data of Dumusque et al.
This quick reaction from astronomers shows the intense interest in exoplanet research, a sign of a healthy field. Astronomers aren't just pulling out the "extraordinary claims require extraordinary evidence" cliche, they're checking for themselves how strong the evidence is.
This is science in action and kudos to Zastrow and Overbye for writing about it, given that editors are more likely to prefer discovery stories, even at speciality publications like Sky and Telescope. I expect that many readers will appreciate seeing this look at the less sensational aspects of research, marked by uncertainty, debate and patient cross-checks. A crucial factor was that the velocity data were made public when the paper was published, and credit goes to Dumusque et al. and/or Nature for this display of open science.
It's not rare for exoplanet claims to be disputed. Overbye points out the example of Gliese 581g* and doubts were raised about the famous planet 51 Peg until the challenge was retracted (in this case Hatzes helped make the case that the planet was real).
[* Note added June 28th, 2013: Xavier Dumusque kindly read this post and provided a couple of comments where he disagreed with me. Dumusque writes:
"The first is the comparison between Alpha Cen Bb and Gl581g that Overbye has done. In the case of Gl581g, the signal false alarm probability was ~1% when they published the article. In addition, this signal was not found on HARPS data, but only when combining HARPS and KECK data (HARPS being twice as precise). Therefore I do not find it fair to compare the two claims of a planetary signal."My response: There are differences in the strength of the evidence for the three exoplanet candidates or exoplanets that I mention, Alpha Centauri B, Gliese 581g and 51 Peg. My goal was only to list a couple of extra examples where exoplanet claims have been disputed, not to list examples where the evidence was as strong as for Alpha Centauri Bb. I agree that the planet claim for Gliese 581g may turn out to be incorrect. For example, analysis by Philip Gregory - not mentioned by Overbye - does not support the detection of Gliese 581g. At the other extreme, the planet claim for 51 Peg turned out to be correct after it had been disputed.]
 |
| Radial velocity as a function of orbital phase for the famous exoplanet 51 Peg. Note the obvious sinusoidal signal and the large amplitude of over 50 meters/second. Credit: Geoff Marcy and Paul Butler, San Francisco State University. |
More Data Are Needed
Regarding future observations, and reality check #5 (the most obvious one), Dumusque has 18 orbits of Hubble Space Telescope time to look for transits of Alpha Cen B by the possible planet. A detection of a transit would clinch the case for a planet, but the authors estimate there's only about a 30% chance of seeing one, assuming a planet is present, because of the need for orbital alignment.
Dumusque has already obtained more velocity data, a crucial step because longer observations will inevitably lead to a firm detection of the 3.24 day signal, if it's real. He is also interested in trying different analysis techniques to strengthen the case for a periodic signal. There can be tension between the desire to publish an exciting result in a timely way and the urge to be ultra skeptical by carrying out multiple tests. Sometimes it's not clear when you should stop.
I have some experience at seeking weak signals amongst a lot of noise. More than once colleagues suggested that I limit the number of tests I carry out after seeing a tantalizing signal, leading me to think "really?" and "don't tell me what to do!". When you do decide to pull the trigger and write the paper, it's important to be clear about what you've done, as I think they have been.
Describing Marginal Results
One question that arises is whether the Dumusque paper fully communicates the challenges involved in their planet search. Overbye's article says:
"“Calling to question a detection is always something fruitful,” Dr. Dumusque wrote in an e-mail. But he added that it was clear in his team’s paper that “the signal we are searching for is at the limit of the data precision.”"I think that's true for an expert reader, but Nature papers are intended for a wider audience than just specialists. For example Leslie Sage, the Nature astronomy editor said in a presentation about Nature papers:
"If the paper is not comprehensible to people outside a narrow specialty, why bother publishing in Nature?"The impression given by the paper to a non-expert is that there are no doubts about the detection. For example, the paper says: "We report here the discovery of a planetary companion around Alpha Centauri B" and no qualifiers or caveats such as "likely", "possible" or "candidate" are used. Also, when writing about the planet the word "confirm" is used only to "confirm the expected rocky nature of the detected planet around Alpha Centauri B". A more obvious sign of the marginal quality of the detection is their figure showing the radial velocity data, reproduced below.
 |
| The radial velocity data for Alpha Centauri B from Dumusque et al.'s Nature paper. Note the much smaller amplitude of the variations and the much lower signal to noise compared to the 51 Peg figure shown above. Credit: The figure and caption appearing above is from a Sky and Telescope article by Camille Carlisle, dated October 18, 2012. Credit in that article was given to X.Dumusque et al. Nature, 2012. |
For the moment Alpha Centauri Bb should be viewed as a planet candidate. This isn't bad news, since NASA's Kepler satellite has, at the time of writing, 3,216 planet candidates and only 132 confirmed planets.**
 |
| Kepler's list of planet candidates before the latest data release. Credit: NASA Ames/Kepler Mission/Jason Rowe/Chris Burke/Wendy Stenzel |
[** Note added June 28th, 2013: Xavier Dumusque's 2nd comment is:
"Regarding the fact that this is just one more candidate as for Kepler ~3000 detections, the two techniques are not affected by the same false positive. Therefore a comparison between transit candidate and RV candidate is not straightforward. I suppose this was not the point here, but it can be interpreted in this way."
My response: I agree that the two techniques are not affected by the same false positives and a comparison between transit candidates and radial velocity candidates is not straightforward. I also agree that this wasn't the point I was trying to make. I decided to limit the length of my description, but my original intention was to point out that (1) the majority of Kepler's planets have not been confirmed and (2) the Kepler team - to their credit - have consistently and accurately described these objects as planet candidates. One reason the Kepler results have been celebrated is because the false positive rates are likely to be reasonably low, although different groups estimate different amounts. For example, Alexandre Santerne et al. estimate a false positive rate of about 35% for close-in giant planet candidates, while Francois Fressin et al. estimate a global false positive rate of 9.4% and 17.7% for giant planets. Many of these false positives are caused by eclipsing binaries - double stars that transit each other - located in the background, so they are real, periodic signals but not caused by exoplanets.
To give some numbers for the radial velocity data for Alpha Centauri Bb, Dumusque quote a false-alarm probability of 0.02% for their 3.236 day signal. By contrast, in table 7 of his paper, Hatzes quotes false-alarm probabilities ranging from 0.4% to 40% for analysis of the same data. Because large false-alarm probabilities are found for some of his tests, there is concern that the signal is not real.]
The Real Story is Unfinished
The real story deviates in several key ways from the clean scenario I imagined at the beginning of this post, but I think it's more interesting. Crucially, Dumusque et al. and their critics have behaved well and are moving on. Maybe the data that will clinch the case for Alpha Centauri Bb's existence is already sitting on a computer disk, waiting to be incorporated into one long time-series. Or maybe not. I have no doubts that Alpha Centauri B and its bright companion will remain popular targets for exoplanet searches no matter what is found in the near future.
Much of the excitement about the initial announcement concerned the potential for us to travel to this star system. Remember that this planet - if real - is a roaster, since it's much closer to its star than Mercury is to the Sun. Even if Alpha Centauri Bb is real there is currently no direct evidence that Alpha Centauri B has rocky planets in the habitable zone, only statistical arguments. Maybe by the time we find evidence for such planets, if they exist, we'll be ready to start planning that long trip. I won't try to predict what developments will occur in spaceflight technology, but I will predict that the story of Alpha Centauri's exoplanets is far from done.
Comments
Post a Comment