This post follows one I wrote earlier in the summer, after ASSC. Since then I have been trying (in a first-person therapy sort of way) to figure out what made me so worried after the IIT vs. GNWT showdown, and the media coverage that followed it. In this post (which is going to be more technical, and more focused on theories of consciousness) I aim to articulate why many of us were concerned about how the results of the first Accelerating Research on Consciousness initiative were portrayed, and the lessons that this holds for future ARCs, particularly those attempting to test theories that are metaphysically unusual such as IIT.
I am personally invested in trying to understand what happened here, as together with Axel Cleeremans I am co-leading a similar project, also funded by TWCF, in which we are comparing different higher-order theories (HOTs). Our project hasn’t started yet, so now feels like a good time to think about how best to organise ourselves. I also want to start with a disclaimer: I thoroughly admire the efforts to change the field that the Cogitate project has engaged in. Running adversarial collaborations is hard – and running the first one is no doubt an order of magnitude harder. Simply put, we would not be having these discussions (and I would not be writing this blog post) If it wasn’t for their project. We would in all likelihood be pursuing “regular” projects which, as Yaron and colleagues have strikingly highlighted, have repeatedly suffered from confirmation bias and “looking under the lamppost”.
In what follows I focus on three distinct issues. The first is the origin of predictions, and the idiosyncratic nature of these. The second is the role of background assumptions, and how consciousness science interfaces with mainstream neuroscience. The third is the bigger-picture (and perhaps harder to resolve) issue of how to test between two theories that have radically different metaphysical starting points and implications.
1) The hypothesis machine
The ARCs adopt a novel approach to hypothesis generation. Our project comparing different HOTs started with a series of Zoom workshops held at the invitation of TWCF. In these meetings, different proponents of different theoretical perspectives on higher-order theory gave presentations, held discussions, and were encouraged to identify distinguishing features of their views. This then led various members of the group to suggest experiments that could obtain results or empirical signatures that should be obtained under each theory, and which would put pressure on the theory if they were not obtained.
My understanding is that a similar process was followed to generate the predictions of other ARCs, including Cogitate. More broadly, the ARCs aspire to falsification of predictions and ultimately the elimination of theories. What this has meant in practice is that theorists have been encouraged to “sign on” to predictions that, if falsified, would put pressure on their theory. This is a welcome exercise in robust theorising. Usually theoretical papers are aimed at showcasing what your model can do, and the conceptual ground it covers, rather than identifying fault lines along which it might be broken. But from this noble starting point, I worry that the current adversarial collaboration approach has inadvertently led to some problems.
These projects get off the ground if a theorist is willing to sign their name to a prediction. Such a prediction does not necessarily have to have been previously published or subject to scrutiny by the field – indeed, it is more likely that it would not have been, due to the aforementioned issues with how theorists tend to describe their theories in print. Instead, it is likely to be a personal interpretation of a theory. And this is where, for me and many others, things start to get strange.
For GNWT and IIT, the predictions were carefully laid out in a public pre-registration document and peer-reviewed paper by the Cogitate team. Let’s start with the GNWT predictions. The core predictions are straightforward – that prefrontal cortex, as a key region for supporting the global workspace, should show multivariate decoding of conscious contents (in the case of Cogitate’s Experiment 1, stimulus category and orientation) during suprathreshold visual perception. The neuronal version of GWT has long held that a frontoparietal network should play this role (putting aside details of what exactly broadcast entails). The predictions on timing and onset/offset responses are more subtle, and it’s debatable how central they are to GNWT (eg it seems reasonable that there could be continuous broadcast, and the response by Dehaene in the Discussion section seems to suggest that one reason that there wasn’t an offset response is that subjects might not have been conscious of the image throughout the presentation interval… which in turn suggests that the punctate responses in PFC could be consistent with a continuous broadcast of limited conscious content! But let’s put these details to one side for now). Note that (as in the classic Dehaene et al. 2001 paper) GNWT should also predict robust decoding of stimulus features in visual areas, but would associate this with unconscious aspects of visual processing.
The IIT predictions are the ones that have got many of us exercised and alarmed (and I am saying this as someone who has been and continues to be intrigued by IIT). In short, IIT theorists proposed that maximal decoding should be found in a posterior “hot zone” (Prediction 1), that decoding in this region should be sustained throughout stimulus presentation (Prediction 2), and that there should be sustained connectivity between high and low-level visual areas (Prediction 3). The hot zone refers to primarily visual (parieto-occipital, “back of the brain”) areas that have been reported in (empirical) studies of perceptual consciousness, most notably in Siclari et al. (2017) as tracking dream experiences1. The clear and detailed “physical substrate” paper by Tononi et al. (2016) in turn suggests that brain areas that have topographic maps of feature space might have the right cause-effect structure to support high phi: “Specifically, the grid-like horizontal connectivity among neurons in topographically organized areas in the posterior cortex, augmented by converging–diverging vertical connectivity linking neurons along sensory hierarchies, should yield high values of Φmax.”
So while I am no doubt glossing over many details, I take it that the general argument is that a set of topographically organised visual areas are candidate physical substrates for consciousness of various aspects of visual phenomenology, and together yield high phi.
A first issue is that, as I’ve already noted, a GNW theorist should also predict that visual areas should contain visual representations – whether or not they are globally broadcast. So should a higher-order theorist. So the key difference here seems to turn on whether these representations are sufficient for conscious experience, or not. This was not tested in Experiment 1 of Cogitate (this is partly why I am excited about the results of Experiment 2). And even if you turn out not to need PFC, then the IIT predictions would also be consistent with a number of first-order perspectives such as those put forward by Victor Lamme and Ned Block2.
2) Bigger fish to fry
One response to this is to say, “this is all fine, because remember we are being strict Popperians here. The IIT prediction might not be unique, but if we don’t see it, then the theory is in trouble!”
Well, yes, but if we didn’t see it, then we would have bigger fish to fry.
To explain why, I need to take a little autobiographical detour. I often feel I lead a split life in science. My heart belongs to my wonderful friends and colleagues in the consciousness community. ASSC is my “home” conference, and I have been lucky in recent years to join both CIFAR and TWCF initiatives to advance consciousness science around the world. But for much of the rest of the year, I run a lab focused on understanding the neural and computational basis of metacognition, including how metacognition goes awry in neurological and psychiatric disorders. This means I am often also attending cognitive / computational neuroscience conferences, or going to clinically-oriented meetings. Consciousness science doesn’t have much traction in this community – indeed, it’s still considered a bit oddball and “out there”, even after the pioneering efforts of Crick and Koch and everyone who has followed in their footsteps. We are on surer ground when our focus is on psychophysics and minimal tests of conscious vs. unconscious processing. But when a central prediction of a much-anticipated and reported adversarial collaboration is “there will be high/sustained decoding of visual features in visual cortex”, then we are going to run into trouble with our mainstream neuroscience colleagues, no matter how noble the falsification goal is.
Why? Because the evidential value of this prediction against the vast literature of mainstream neuroscience is null. Let’s be Bayesian about this to hopefully make it a bit more concrete. We are interested in how to update our belief in a theory, after seeing the data. In this case, we can compute the posterior in IIT, after either seeing hot-zone decoding (HZ+) or not (HZ-):
p(IIT|HZ+) ∝ p(HZ+|IIT)p(IIT)
p(IIT|HZ-) ∝ p(HZ-|IIT)p(IIT)
IIT says it’s more likely (given IIT is true) to observe HZ+ than HZ- (the first term on the right-hand side), so we should increase our belief in IIT if we see HZ+, and decrease it if we see HZ-.
So far, so good. But we should not only condition our likelihoods on IIT. Science does not operate in a vacuum. Instead, we should really compute our likelihoods against background conditions C, which here can stand in for mainstream neuroscience. Now our posteriors become:
p(IIT|HZ+, C) ∝ p(HZ+|IIT, C)p(IIT|C)
p(IIT|HZ-, C) ∝ p(HZ-|IIT, C)p(IIT|C)
Let’s put aside the last term for now (this is our prior belief in IIT given mainstream neuroscience, which will become relevant in the next section). The key difference is that our first likelihood should now be very high, almost 1 (let’s say 0.99), because observing HZ+ is very likely given what we know from neuroscience (ie the standard model of sensitivity to edges in early visual areas, shape and colour in intermediate areas, objects in IT cortex, etc). The consequence is that our credence in IIT should hardly change when this result is obtained, as we are multiplying our prior by a value very close to 1.
But what about observing HZ-? This is indeed very informative, as if obtained it should radically decrease one’s belief in IIT. But it should also make you question your belief in mainstream neuroscience, because the likelihood of seeing HZ-, irrespective of IIT, is so small! Indeed, if there had not been decoding of visual features in posterior cortex for this experiment I imagine the research team would be justified in thinking there might be a bug in the analysis pipeline, etc – revealing the powerful influence of background assumptions inherited from mainstream neuroscience.
We are then left with what we can say about support for a theory. Unfortunately things are no better here, because the likelihood of observing hot zone decoding of suprathreshold visual content under pretty much any other theory (GNWT, HOT, IIT, RPT, FO, etc) should be similar. The evidence in question here is not diagnostic of IIT, and therefore cannot be used in support of the theory, formally or informally (as I said, things change considerably for Experiment 2, where there is a direct manipulation of conscious vs. unconscious processing – and the anatomical prediction for GNWT is also on safer ground, given that our understanding of PFC function especially in visual tasks is less secure).
This is I think at the root of why so many of us in the field are frustrated with the media coverage of IIT since the Cogitate results were made public. Despite the admirable focus on falsification by TWCF and the Cogitate PIs, the language used in the preprint and in the media coverage has often claimed support for the theory. For instance, in the preprint Discussion section written by IIT theorists:
“The results corroborate IIT’s overall claim that posterior cortical areas are sufficient for consciousness, and neither the involvement of PFC nor global broadcasting are necessary. They support preregistered prediction #1, that decoding conscious contents is maximal from posterior regions but often unsuccessful from PFC, and prediction #2, that these regions are sustainedly activated while seeing a stimulus that persists in time.”
3) The spectre of metaphysics
A nagging worry about all of this is that much of what makes it so difficult to get big theoretical beasts such as GNWT and IIT into the same ring comes down to metaphysics. GNWT operates within the mainstream cognitive neuroscience tradition, in that it endorses computational functionalism, where global broadcast is one kind of computation over mental representations. IIT is very different. It says that what matters is not representation or computation but the cause-effect structure of a physical substrate. This substrate may be passive and not doing anything, which leads to the striking prediction that (if we assume neurons are units in the substrate) silencing already-silent neurons may change conscious experience. It also leads to the prediction that phi can divorce from observable function (as in the unfolding argument) and that non-brain things are conscious in varying degrees and in varying forms. If you are an IIT theorist and this is your worldview, then perhaps you are (very reasonably) less likely to endorse the background assumptions of mainstream neuroscience. Simply put, you think consciousness is not going to succumb to explanations from mainstream neuroscience – so why should you care about the likelihood of such-and-such a finding about consciousness given mainstream neuroscience is true? You shouldn’t.
But the rest of the world, and the rest of mainstream neuroscience, will.
So perhaps there is an impasse here that is not at the level of how we do science, but at the level of how we pursue theory. I am all for exploring out-there ideas. But today’s neuroscience is not like today’s theoretical physics. We don’t yet have a standard model that is getting old and dusty and perhaps in need of radical overhaul. We are only just getting started with probing the brain with some ever-improving and amazing imaging technologies. We will have new findings in spades in the next few decades, including conceptual advances that we cannot begin to anticipate. Given our field is still in these nascent stages, calls for a revolutionary new approach that is at odds with mainstream neuroscience are at best premature. Theories and models are important in this endeavour, but we would be wise to stay humble and patient in the face of incoming data, and give a mature science of consciousness a chance to get out of first gear.
Notes
1Hakwan Lau has written an extensive open review of the Cogitate preprint that also explores the finer details here of ROI selection underpinning the hot zone. I do not go into this here as I want to stick to the bigger picture, but these choices are of course crucial.
2There is an interesting and subtle role for the timing and connectivity predictions here. As I mentioned above, it’s not clear whether GNWT should not also predict continuous broadcast, or at least also predict continuous unbroadcast first-order representation in visual areas. Scalp evoked responses tend not to be sustained – so one could argue that the sustained prediction of IIT was somewhat unique. But remember that here we are dealing with multivariate decoding which should bear stronger resemblance to the underlying neural code, and there is classical electrophysiological work in perceptual decision-making that has found sustained content-specific representations (eg single-unit representations of motion direction in monkey MT or tactile stimulation frequency in S1). So I am not convinced this is a unique prediction of IIT that goes beyond mainstream neuroscience. In fact, the final connectivity prediction might be the most unique, and this is one that IIT failed to pass.
3I am grateful to Matthias Michel for feedback on an earlier version of this post.
The Elusive Self 