Neptune and Vulcan (Part 2): A not-so-random sample

Historical anecdotes and general thoughts on Neptune/Vulcan dilemmas

[Follow-up to: Neptune and Vulcan (Part 1)]

I

In the first part of this series I introduced the Neptune/Vulcan class of dilemmas thusly:

All you need to have a Neptune/Vulcan dilemma is:

1. A fundamental theory or principle which entails a prediction.
2. A failure to confirm said prediction which doesn’t also disprove it (e.g., because the prediction has enough free parameters to be effectively unfalsifiable, at least with available technology, or because the exact parameters of the prediction can only be computed through a series of approximations and/or complicated calculations which introduce large uncertainties).

This problem has two possible types of solutions:

• Neptune: the prediction is true for some allowed set of parameters
• Vulcan: the prediction is wrong and the anomaly can only be resolved when the fundamental theory in question is replaced by a better paradigm.

I also briefly discussed the following examples of Neptune/Vulcan dilemmas in modern physics/astronomy¹:

• Dark Matter (is it real?)
• Fermi’s paradox (are we alone in the Universe?)
• The proton radius (should our different measurements of it agree?)
• The lithium problem (why is there less of it than predicted by the Big Bang?)

That post ended with the natural epistemological interrogation:

So how should we generally approach Neptune/Vulcan dilemmas? That’s the epistemic question for which I really would like to have a general answer. Unfortunately, I don’t have one. But can we learn any general principles from historical examples of Neptunes and Vulcans? That’s what Part 2 will be all about…

As promised, let’s delve into the clearest historical examples that came to my mind.

II

The first few Neptunes I could think of were:

• Neutrinos
  During the 1910s, studies of beta decay seemed to indicate a violation of the law of conservation of energy: when you measured the energies of all particles before and after the decay, it seemed as though the total energy decreased. As this result was confirmed time after time by experiments, theoretical physicists started to panic — conservation of energy is, after all, one of the cornerstones of physics which had survived since the times of Newton into those early days of relativity and quantum mechanics. Niels Bohr, perhaps the most notable pioneer of quantum mechanics, was ready to give up on conservation of energy as a fundamental law of Nature.
  
  If you’ve been paying attention, it shouldn’t surprise you to learn that the solution to this mystery came when, in 1930, Wolfgang Pauli proposed there existed a new particle (which he called the neutron and we nowadays call the neutrino²) very much like dark matter in cosmology: almost impossible to detect due to interacting with regular matter only via the weak force (which governs beta decay) and gravity (which is even weaker than the weak force). Although the following years saw a gradual accumulation of evidence in favour of Pauli’s neutrino and against Bohr’s rejection of energy conservation (by showing an apparent limit to the amount of missing energy), for a long time neutrinos seemed impossible to even look for thanks to the weakness of its interactions with all other known matter. Only in 1942 was a feasible method for directly detecting neutrinos proposed, and it took until 1956 for the first detection to be announced.

Wolfgang Pauli despairs at the delay in detecting neutrinos, by S. Harris.

• Positrons
  The first time that Erwin Schrödinger wrote down his famous equation in 1925, he did so by starting from the classical equation linking the energy and momentum of a free particle

(where E stands for energy, p for momentum, and m for mass) and then substituting E and p with equivalent quantum mechanical objects.

As it happens, just before writing this equation Schrödinger had done the same thing using the (more accurate) relativistic version of this equation

(where c is the speed of light in vacuum) and rejected the resulting equation due to having noticed a few theoretical problems with it — chief among which is the fact that it allowed for clearly unphysical negative energy solutions (due to the squaring of E in the above equation). In 1926, Klein and Gordon reached the same equation and were less troubled by its theoretical shortcomings, and therefore that equation is nowadays known as the Klein-Gordon equation. Not that they figured out any way to solve those problems, mind you³.

In 1928, Paul Dirac tried a clever mathematical trick: to take the square root of the above equation

and then “quantise” that (hoping that the lack of the power of two would get rid of the negative energy solutions). Interestingly, the resulting Dirac equation succeeded at capturing the dynamics of electrons (and indeed any particle with spin 1/2⁴) yet it turned out to still admit negative energy solutions. Originally Dirac attempted to explain these solutions as “holes” in a “sea” of regular positive-energy electrons, but a back-and-forth exchange with Robert Oppenheimer and Hermann Weyl led him to in 1931 reinterpret these mathematical solutions as an actual physical particle of which hints had never been observed: in all ways like an electron but with positive charge. The following year Carl Anderson discovered the positron, confirming a prediction purely based on the “unwanted” solutions to Dirac’s equation.

• The Higgs boson
  One day I hope I will get to write something more in-depth about the standard model of particle physics and the history of its development. For now what you need to know is that the big ambition of (particle) physicists is to be able to describe all interactions (between fundamental particles) using the same fundamental system of equations. These days the “finished”⁵ version of the standard model of particle physics does just that for three out of four known fundamental interactions (with gravity being the odd one out). And you also need to know that back in the 1960s particle physicists had most of the pieces of the puzzle that would become the standard model, but were having some trouble putting them together.
  
  The big problem is that one of the pieces of this puzzle was a description of weak nuclear interactions involving three particles — known as the W⁺, W⁻, and Z bosons — which experiments inconveniently showed needed to have masses greater than zero. Inconveniently because the standard model is built on certain symmetry principles that it is impossible not to break unless those particles are literally massless.
  
  In 1964, three research groups independently proposed a mechanism which allowed this model of weak interactions to be incorporated into the rest of the standard model. In a nutshell, they postulated that there is a certain something, now known as the Higgs field, which interacts with Z and W bosons and makes them behave as though they have masses (or, if you prefer, “gives” them masses; the distinction is mostly philosophical). This Higgs field is at once the simplest it could be and extremely exotic: it is a scalar field, which means it is defined by a single number at each point in space⁶ (temperature and pressure are examples of scalar fields in the macroscopic world). Simple as it is, no fundamental scalar field had ever been confirmed to exist — and no other has since.
  
  One of these co-discoverers, Peter Higgs, went further and showed that an unavoidable consequence of this mechanism was the production of a then-unknown particle, which became known as the Higgs boson⁷. Then decades went by during which the standard model passed experimental test after experimental test, but during which no experiment conclusively proved the existence of the Higgs boson, instead only constraining the possible masses it could have. Until in 2012 it was finally discovered in the LHC.
  
  Technically until the last moment there were viable alternative mechanisms for generating the mass of W and Z bosons. But this mechanism was the simplest (hence its almost simultaneous proposal by three independent groups) and widely seen as the theory to be tested⁸.
• Black holes
  It is said that, when Einstein formulated his theory of general relativity, he did not expect any exact solutions to its equations to be found (apart from the trivial solution for a completely empty spacetime). So when Karl Schwarzchild published one just one month after Einstein published his complete theory of relativity, that was a big deal.
  
  Einstein himself was apparently pleasantly surprised to discover that such exact solutions were possible, but a bit less so at the implications of this particular solution. Schwarzchild’s solution describes the spacetime around a spherical concentration of mass and, for sufficiently dense concentrations, includes a singularity in spacetime (basically a point where you divide by zero and get infinity and poor old Einstein’s equations basically cease to function) and a horizon which, once crossed from the outside, cannot be crossed again. You have guessed it: for sufficiently high densities it describes a black hole⁹.
  
  Over time, it became more and more obvious that black holes were the natural outcome predicted by general relativity for the end of the life of sufficiently massive stars. And even though light can’t escape them, over time we became able to find them: first by their gravitational effect on nearby matter, then by their newly-discovered Hawking radiation, and finally by their gravitational wave signatures.
  
  By the way, have I mentioned that primordial black holes are a relatively popular dark matter candidate?
• Gravitational waves
  In classical physics, we make sense of electromagnetism by saying:
  • There exist electric and magnetic fields, quantities which take values at every point in space.The values of these fields at any given point determine how objects with electric charge move at that point.The values of these fields at any given time are influenced by how objects with electric charge are distributed and moving.
  A non-obvious consequence of this synergy between electric charges and electric/magnetic fields is that when electric charges accelerate they cause electric/magnetic fields to oscillate, producing electromagnetic waves (a.k.a. light).In general relativity, we make sense of gravity by saying:
  • There exists a spacetime metric, a quantity which takes values at every point in space (and time too, but we don’t need to think about that now).The values of this metric in any given region ( or, actually, its derivatives at any given point) determines how objects move in that region.The value of this metric at any given time (or, actually, the values of its derivatives) is influenced by how objects are distributed and moving.
  The analogy where gravity is like electromagnetism with a mass instead of a charge and a spacetime metric (or, actually, an Einstein tensor) instead of electric and magnetic fields practically makes itself. There are some mathematical differences, but at the end of the day the result looks very similar: when masses accelerate (unless they do so whilst preserving spherical symmetry) they cause the spacetime metric to oscillate, producing waves in spacetime (which we call gravitational waves).
  
  That gravitational waves were an unavoidable consequence of general relativity only became well established in the 1950s¹⁰. However, the predicted magnitude of gravitational waves produced by ordinary phenomena was vanishingly small — a consequence of the weakness of gravity compared to other fundamental forces. In fact, even the most extreme phenomena at a cosmic scale produces only such weak radiation that it took a major scientific collaboration several decades to build a detector capable of distinguishing such waves from the ubiquitous noise of seismic activity and thermal excitation of particles on Earth. But they were there, and since 2015 we’ve managed to detect them a few times. Along the way there were a few failed attempts, including even an infamously wrong claim of detection in 2014.

III

The first few Vulcans I could think of were:

• The aether
  Since Newton dedicated some time to determining whether light is a particle or a wave and decided it is a wave, the consensus became that it must be a wave. However, at the time the concept of waves propagating through a vacuum was inconceivable and thus most scientists assumed this must mean there existed some substance through which light propagated — the aether (no actual relation to the classical element with the same name). Only when Maxwell formulated his famous equations for electric and magnetic fields did people first attempt to think of light as a wave propagating not through some sort of material medium but through oscillations in vector fields. But there was a problem.
  
  In classical physics, if I am moving with velocity +v relative to you and you observe an object moving with velocity +s, then I should observe that same object moving with velocity s-v. But if you solve Maxwell’s equations for the same light wave travelling past two different observers who are moving relative to each other, you find that they should both observe the light to be moving at the same velocity — in clear violation of a fundamental principle of classical physics.
  
  The way physicists used to make sense of this tension at the heart of such fundamental equations was by assuming that these equations should only be valid for one very specific type of observer: one not moving relative to the aether. Oh well, disappointing but a logical conclusion: after all, why should these mathematical constructs derived solely for the purpose of making calculations possible for humans be any sort of fundamental building block of reality?¹¹
  
  There were only two problems with this solution: not only had the aether never been detected by any experiment, but experiments meant to measure variations in the speed of light as measured by observers moving relative to each other kept stubbornly failing to falsify the preposterous proposition that the speed of light be the same for all observers.
  
  Then in 1905 Albert Einstein apparently made a New Year’s resolution to revolutionise all of physics, and part of that involved proposing that light is actually a particle and showing that physics actually makes sense if the speed of light is the same for all observers¹². Thus the concept of aether ended up becoming superfluous and made its exit from physics textbooks — even if not from physics-oriented blog names.
• Neutrino masses
  The standard model of particle physics has produced some of the most accurate predictions in the history of science. However, even though most of these have been confirmed time and time again with ever increasing experimental precision, one prediction is embarrassingly off the mark: the value of neutrino masses.
  
  Similarly to W and Z bosons¹³, the standard model requires neutrinos to have exactly zero mass — yet we measure them to have very-small-but-definitely-not-zero masses. Unlike with W and Z bosons, we have no idea how to fix the standard model to account for neutrino masses and the whole thing is seen as a (if not the) major piece of evidence of unknown physics beyond the standard model.

IV

So what intuitions can I distil from these examples?

I genuinely struggled to come up with examples of Vulcans of the same caliber as the aether and the above Neptunes, which came to my mind relatively easily. I don’t think this is just down to “optimism bias” on my part but it doesn’t in itself mean that Vulcans are less likely than Neptunes — although it does suggest that these dilemmas are less likely to be resolved as Vulcans, if nothing else due to the added difficulty of proving non-existence compared to existence. After all, I can’t be certain that all current examples of Neptune/Vulcan dilemmas I listed above aren’t instances of Vulcans. But even if they were then Vulcans would still only just beat the Neptunes 6-5 in the examples I thought of¹⁴.

That Neptunes should be more likely than Vulcans should not be an unexpected conclusion of our little sampling exercise — after all, anything else would suggest our standards for considering a theory “fundamental” to be too lax. Our fundamental theories should already have passed some number of tests which give us some confidence that their predictions are correct as long as they are not beyond the domain of applicability of the theories in question¹⁵.

After all of this, my general thoughts for when first facing one of these dilemmas are:

• The higher your a priori confidence in the fundamental theory in question, the more likely you should find a Neptune scenario.
• The more immediate a consequence of the fundamental theory the prediction is, the more likely you should find a Vulcan scenario.
• If finding a Neptune would increase your confidence in your fundamental theory, failing to find one should decrease it¹⁶.
• If a Neptune/Vulcan dilemma will be resolved, it’s more likely to be resolved as a Neptune. This is merely a consequence of it being easier to find something than to prove something doesn’t exist (plus Neptunes being at least a little more likely a priori if there is adequate confidence in the fundamental theory in question).

I would definitely bet on Neptunes in each the dilemmas listed above: dark matter being real, the details of how life comes about just making alien life less likely than the naïve estimates, disagreements about the proton radius being down to failing to appropriately account for all the mathematical and experimental uncertainties, and lithium being used up more by some unknown process much later than the Big Bang. But I would be much less confident about betting that all of those are Neptunes¹⁷.

At the end of the day I don’t have a satisfying answer to the question of how to approach these dilemmas, but I think there is value in being aware of this category and of the thoughts above.

I suspect the full answer is a special case of a more general type of epistemic dilemma — but that is a story for another day.

Footnotes

1 In this post I will stick to physics/astronomy examples, as those are the ones I’m most familiar with. Nevertheless, I can’t resist mentioning a spot-on (as far as I can tell) non-science example brought up by Ophis_UK in a comment to a Reddit post featuring Part 1: the Q document. This is a document which is hypothesized to have been drawn upon (alongside the Gospel of Mark) in the writing of the Gospels of Matthew and Luke. Evidence for it appears to be based solely on comparisons between these three Gospels, no such document (or contemporary references to it) has ever been found, yet its existence is part of the foundation of a lot of modern Gospel studies.

2 The name “neutron” was scooped by James Chadwick in 1932, after which the diminutive was introduced to the name by the Italian giant Enrico Fermi.

3 Clearly the lesson here is: if your findings are potentially useful, publish them even if you can’t figure out how to solve the problems they raise.

4 Due to complicated mathematical reasons stemming from Dirac’s realisation that in order to make his equation self-consistent some of the terms would need to be matrices rather than numbers.

5 Even though there are known problems with the standard model of particle physics, I think it’s been around long enough that if/when a significant improvement on it is made the resulting model will most likely be given a different name. So I think it is fair to call what we currently have the “finished” version of it, even if it is not the complete description of the phenomena it tries to describe.

6 Compare with, for example, the mundane electric field: a 3D vector with three values at each point in space.

7 As a consequence of the Higgs boson being an excitation of a rare scalar field, it is also unusual by being the only confirmed example of a fundamental particle with spin equal to zero.

8 As part of my MSc, I actually did a module on experimental particle physics which basically covered the experimental evidence for all of the standard model just before the discovery of the Higgs boson was announced. The teacher was very clear that he was very sure the Higgs boson would be found by the LHC on the basis of all the indirect evidence provided by these previous experiments, and that very few people thought otherwise. I think it is fair to say that in this search the Higgs boson was to Neptune as alternative mechanisms would have been to modified theories of (Newtonian) gravity in Le Verrier’s time.

9 Note that black holes defined as dense enough objects that even light can’t escape them had already been studied in the context of Newtonian physics.

10 It took a few decades because general relativity mathematics was very new and confusing and the first time Einstein proposed (with very little confidence) the existence of gravitational waves he proposed the existence of three types of gravitational waves. One of these was the type of gravitational waves we now know exists. The other two were relatively quickly shown to be unphysical features of the coordinate system Einstein had used. So the suspicion was justified.

11 Cue for quantum physicists to laugh their hearts out.

12 See my post about special relativity for more details.

13 See discussion of these in the Higgs boson section above.

14 If you were to estimate probabilities from these examples (which if you did you absolutely shouldn’t take seriously) then the probability of a Neptune/Vulcan dilemma resolving in favour of Neptunes conditional on it resolving in the first place would be 67% and the probability of it being a Neptune regardless of whether the dilemma is resolved would be between 46% and 77%.

15 And, actually, even if they are we should still take these predictions as somewhat likely to qualitatively point us in the direction of truth. For example, in general relativity we know that singularities are a tell-tale sign that we are making calculations well beyond the domain of applicability of the assumptions on which our theory is built. Yet even if we don’t expect energy densities at these points (such as in the centre of black holes or at the “Big Bang”) to be literally infinite, we definitely expect them to either be astronomically high or an invalid concept due to some sort of breakdown of spacetime.

16 This may sound obvious but it is important to avoid falling prey to confirmation bias by seeing positive evidence as strengthening a belief whilst not seeing negative evidence as doing the opposite. Even if one failed detection is not sufficient to make you bet on a Vulcan, a sufficient number of independent dilemmas based on the same fundamental theory should.

17 If we took the estimated probabilities in footnote 22 seriously (which, again, you shouldn’t) then the probability that all four of those are Neptunes should be less than 35%. (Yes, I know I’m using a number where outcomes for those four is assumed to calculate a probability for those same outcomes. I did tell you not to take this seriously.) That figure does not offend my intuition, off as it may be from a rigorous estimate.