Decoding a brain image#
A map of activations, the result of an experiment of yours, says where the brain responds. Decoding asks the reverse question: which mental processes do the studies that activate the same places study? NeuroLang decodes an image towards the terms of a meta-analysis engine, Neurosynth’s or NeuroStore’s, with the probabilistic model those engines already use for their maps (Neurosynth’s Association and Uniformity Tests as Probability Ratios), in a query of plain Datalog over the image as a relation of voxels and values. This page loads an image, explains the query, and decodes two images, a mask of the amygdala and a statistical map of a motor task, and three bundles of white matter by the ends of their streamlines.
Loading an image#
In the REPL in a terminal, .image loads a NIfTI image, gzipped or not:
nl(neurosynth:datalog)> .image doc/_static/decoding/amygdala_mask.nii.gz
as a relation named after its file, amygdala_mask(i, j, k, value) here
(--name gives another): the voxels of the brain where the image is not
0, with their value, on the grid of the engine. The image
is resampled to that grid if it has another, a mask (an image of one value
other than 0) to the nearest voxel so that it stays a mask, and its first
volume is kept if it has several. It loads, too,
amygdala_mask_probability(i, j, k, p), the probability that each voxel is
active, which the decoding weighs the voxels by (see The score: the lift
of a term). Two decodings are offered:
--mask, masked decoding1 for each voxel of a binary mask, which needs no flag, or of a map of values from 3.1 (
image.threshold): its voxels weigh the same.--zscore, from z-scoresfor a map of z-scores, the probability that the z-score of each voxel is that of an active voxel: its voxels weigh their probability.
A map loaded without either has none, and is not decoded until it is
loaded again with one.
.image shows the image in the brain viewer, and prints the query
decoding it. --name NAME names the relations NAME and
NAME_probability instead, to load several images; .image PATH --name
image replaces the image loaded first.
In the browser, and on a server, Upload data (in the toolbar) takes the
.nii or .nii.gz file: the page keeps it with the session, runs
.image on it, and asks whether to decode it (Run the decoding query,
or Not now): the decoding query, a comment explaining each of its lines,
goes in the box and runs. It replaces what is there, which Bring back
the query before puts back until the next run. For a map of values, the
page offers the two decodings, Decode as z-scores and Decode as a mask
(from 3.1), loads it again with --zscore or --mask, then runs the
query. Only the cortex, checked,
decodes the cortex of the image alone (--cortex, below). A
server reads none of its own files for its users, so an image must be
uploaded there. The engine of the page in the browser leaves out the peaks
of the studies’ activations (activated) to fit in a tab: it loads and
shows the image, and decodes it with ?engine=full added to the address.
The largest image taken, 256 MB uncompressed, is image.max_mb in the
viewer.yaml of neurolang_apps.utils.tui.
The score: the lift of a term#
The meta-analysis engines choose a study at random, selected_study(s),
and say that it activates the voxel \(v\), \(A_v\), if one of its
peaks is within 6 mm of it. A term \(T\) holds when the study mentions
it (term_in_study). Decoding chooses, besides, a voxel \(V\) of the
image, independently of the study, the more likely the more probably
active: of the probability \(p_v\) that the voxel \(v\) is active,
Every voxel of a mask is active, \(p_v = 1\), and equally likely. A voxel of a map of z-scores is active, or of the null, its z-score drawn from the one or the other distribution; the probability that it is active is 1 minus its local false discovery rate (Efron 2004),
\(f\) the density of the z-scores of the map, a Gaussian kernel
estimate, \(f_0\) that of the null, the normal of the median and the
median absolute deviation of the map (its empirical null, which the
smoothness of a map makes wider or narrower than \(N(0, 1)\)), and
\(\pi_0 = 1\), the share of null voxels at its most. A z-score below
the median, a deactivation, is never active, nor are the voxels of a
probability below 0.01 kept (image.min_probability in
viewer.yaml). On the motor map below, z = 2 gives p = 0.16, z = 3 gives
0.92 and z = 4 or more 1: no threshold is chosen, the map says where its
activations start.
The event \(A_V\), that the chosen study activates the chosen voxel, has the probability
the activation of the image averaged over its voxels, by all the studies and by those of the term. The lift of the term, the score of the decoding, is their ratio,
the studies of \(T\) activate the image lift times as often as the
average study. A lift above 1 associates the term with the image; the
terms of the largest lifts decode it. It is the uniformity of
Neurosynth’s Association and Uniformity Tests as Probability Ratios with the voxels of the image, weighted,
in place of those of the whole brain.
The query: the voxels as facts, one of them chosen#
The decoding query of the engines (image_decoding_query in
engines.yaml, {image} being the name of the image and {voxel}
the voxels chosen, selected_voxel(i, j, k) of its
image_decoding_voxels) says this as a
probabilistic program: each voxel of the image is a fact, active with its
probability, a voxel of the brain is chosen uniformly, the engine’s
selected_voxel (that of uniformity, Neurosynth’s Association and Uniformity Tests as Probability Ratios),
and a study hits the image when it activates the chosen voxel and that
voxel is active:
% Each voxel of the image is active with its
% probability, independently: a probabilistic fact.
{image}_active(i, j, k) :: p :- {image}_probability(i, j, k, p)
% A study s hits the image when a voxel chosen at
% random in the brain (selected_voxel), or in its
% cortex (cortical_mask), is active and s activates
% it (a peak within the kernel of the voxel).
{image}_hit(s) :- {voxel}, {image}_active(i, j, k), activated(i, j, k, s)
% For each term t, the probability that a study of t,
% chosen at random, hits the image.
p_{image}_given_term(t, PROB) :- ({image}_hit(s) & selected_study(s)) // (term_in_study(s, t) & selected_study(s))
% The probability that any study, chosen at random,
% hits the image.
p_{image}(PROB) :- {image}_hit(s) & selected_study(s)
% The number of studies of each term: the lift of a
% term of few studies is noisy.
term_studies(t, count(s)) :- term_in_study(s, t)
% The lift of each concept of the Cognitive Atlas of
% 30 studies or more: how many times more often its
% studies hit the image than the average study does.
% The larger, the more the concept decodes the image.
ans(term, lift, studies) :- p_{image}_given_term(term, a), p_{image}(b), term_studies(term, studies), (studies >= 30), cogat_term(concept, term), (lift == a / b)
image_active(i, j, k)is a probabilistic fact of each voxel, of its probability \(p_v\);selected_voxel(i, j, k)a choice of one of the \(N\) voxels of the brain mask, each of probability \(1/N\);image_hit(s)holds when the studysactivates the chosen voxel, which is active;p_image_given_term(t, p)conditions it on the studies of each term, for every term at once, andp_image(q)is its probability over all the studies;the answer keeps the terms of 30 studies or more,
term_studies, since one study of a rare term activating the image makes its lift large, and the concepts of the Cognitive Atlas (Querying the Cognitive Atlas Ontology) rather than every word of the abstracts. Dropcogat_termto rank every term of the engine.
The voxel is chosen independently of the study, and the facts of the voxels are independent of it too, so that
the voxels of the image outside the brain mask, which no study activates
(activated is in the mask), adding nothing; averaged over the studies
of \(T\), or over all of them, \(\frac{1}{N} \sum_v p_v P(A_v
\mid T)\) and \(\frac{1}{N} \sum_v p_v P(A_v)\). These are
\(P(A_V \mid T)\) and \(P(A_V)\) above times the same constant
\(\sum_u p_u / N\), so their ratio, lift, is the lift of the term:
the voxel chosen in the image alone, uniformly
(image_voxel(i, j, k) ^ q :- image_probability(i, j, k, p),
image_size(n), (q == 1.0 / n)), gives the same lifts.
Choosing the voxel in the cortex instead decodes the cortical part of the
image only, its subcortical and cerebellar voxels left out (.image
--cortex, the page’s Only the cortex, {voxel} being then
selected_voxel(i, j, k), cortical_mask(i, j, k)):
image_hit(s) :- selected_voxel(i, j, k), cortical_mask(i, j, k), image_active(i, j, k), activated(i, j, k, s)
cortical_mask being the voxels of the Harvard–Oxford cortical atlas.
For the motor map below the lifts move by 0.08 at most (planning enters
the first six); a map with subcortical activations changes more.
The engine solves the program exactly (Lifted probabilistic inference): the
choice of the voxel makes the probability a sum over the voxels, which it
computes for each study, the image joined with activated first, before
the terms of the studies. It is the weighted query one would write by
hand,
image_score(s, sum(p)) :- image_probability(i, j, k, p), activated(i, j, k, s)
image_total(sum(p)) :- image_probability(i, j, k, p)
image_hit(s) :: q :- image_score(s, x), image_total(y), (q == x / y)
of the same lifts, for the motor map below 13 s and 6 GB where this one takes 7 s and 5 GB.
The query reads the image as any relation of voxels and probabilities: a
map of yours loaded with .load, my_map_probability(i, j, k, p), is
decoded by the same query with my_map in place of image. A term
none of whose studies activates the image has a probability of 0, and is
no answer.
The same lift is \(\sum_v p_v P(A_v \mid T) / \sum_v p_v P(A_v)\),
the image joined with the engine’s p_a_given_t(t, i, j, k, p) and
p_a(i, j, k, p); but the engine computes those for every term at every
voxel before the join, more than 40 GB for the amygdala below, where the
query above reads only the studies.
The overlap: the voxels as facts, none chosen#
Without the choice of a voxel, a study hits the image if it activates any active voxel of it,
which NeuroLang writes as it is said:
image_active(i, j, k) :: p :- image_probability(i, j, k, p)
image_hit(s) :- activated(i, j, k, s), image_active(i, j, k)
the rest of the program as above. For a mask or a small region, it is how much more often the studies of a term report a peak in it: fear 2.92, valence 2.50, emotion 2.32 for the amygdala below. On a whole map, nearly every study activates some active voxel of it, 98% of them for the motor map, and the lifts of all the terms are near 1: the choice of a voxel above tells them apart.
A mask: the amygdala#
amygdala_mask.nii.gz (download) is a sphere of 8 mm around the
right amygdala, at (24, -2, -20) mm in MNI space, on the grid of 2 mm:
nl(neurosynth:datalog)> .image doc/_static/decoding/amygdala_mask.nii.gz
loaded amygdala_mask(i, j, k, value): 257 voxels from
doc/_static/decoding/amygdala_mask.nii.gz, a mask
loaded amygdala_mask_probability(i, j, k, p): 257 voxels, the probability
that each is active: 1 for each voxel of the mask
Decode it with this query:
% Each voxel of the image is active with its
...
The 118 concepts of the Cognitive Atlas among the terms of Neurosynth with 30 studies or more get a lift; the largest ten are those of emotion and faces. The studies of fear activate the right amygdala four times as often as the average study:
A tractogram: the ends of its streamlines#
A tractogram (.trk or .tck, in mm in the space of the engine,
MNI152) is decoded where its streamlines end: .image loads it as the
voxels of the brain whose centre is within 3 mm (image.endpoint_radius,
--radius MM for another distance) of the first or the last point of
one of its streamlines, the value of each the number of ends near it, and
each of probability 1, as a mask; the brain viewer shows them, its
streamlines drawn over them (evenly subsampled, as .view draws a
bundle). The decoding query is the same, and
Upload data takes the file and offers it as for an image. The voxels
near the ends are found when the file is loaded, with a k-d tree of the
ends, not by a rule: in Datalog, it would be a join of every voxel of the
brain with every end, compared by their distance.
The left arcuate fasciculus of the HCP842 atlas of Yeh (AF_L.trk, which
.load_tractogram("af", kind="hcp842", bundle="AF_L") of the engines of
tracts downloads):
nl(neurosynth:datalog)> .image AF_L.trk
loaded af_l(i, j, k, value): 2104 voxels from AF_L.trk, the voxels within
3 mm of the ends of its 2010 streamlines, value the number of ends near
each
loaded af_l_probability(i, j, k, p): 2104 voxels, the probability that
each is active: 1 for each voxel within 3 mm of an end of a streamline
Decode it with this query:
...
Three bundles of the atlas, each loaded in half a second and decoded in about 10 s (5 GB), give terms of their function, the five of the largest lift:
bundle |
voxels near its ends |
the concepts of the largest lift |
|---|---|---|
left arcuate fasciculus ( |
2,104 |
sentence comprehension 2.09, rehearsal 2.05, language comprehension 1.98, reading 1.97, meaning 1.95 |
left corticospinal tract ( |
1,895 |
imagery 2.25, movement 2.20, rhythm 2.18, rehearsal 2.10, coordination 1.99 |
left uncinate fasciculus ( |
587 |
semantic memory 2.68, autobiographical memory 2.45, morphology 2.22, passive viewing 2.08, empathy 1.95 |
A whole-brain tractogram ends nearly everywhere: the 300,000 streamlines
of the Disconnectome7T tractogram end near 230,120 voxels, the whole brain,
and its lifts are those of the brain, the largest 1.48; decode a bundle.
.image warns when fewer than half of the ends (image.endpoint_share)
are in the brain: the tractogram is likely in another space.
What the decoding does not say#
The lift ranks the terms by how often their studies activate the image;
it is no test. Whether a lift is larger than chance, or than that of
another term, is the question of Neurosynth’s Association and Uniformity Tests as Probability Ratios: the
bootstrap of the studies (RESAMPLE[boot_studies]) gives an interval
for the lift of one term. With the rules of the query, but its last line,
the lift of fear for the amygdala is
decoding(t, lift) :- p_image_given_term(t, a), p_image(b), (lift == a / b)
lift_replicate(r, x) :- RESAMPLE[boot_studies](decoding('fear', x)) = r
interval(lift, percentile(x, 2.5), percentile(x, 97.5)) :-
decoding('fear', lift), lift_replicate(r, x)
ans(lift, ci_low, ci_high) :- interval(lift, ci_low, ci_high)
4.10 within [3.47, 4.64], in about 10 s: the replicates of the studies
never bring it near 1. Two terms are compared on the same replicate r, as in
Neurosynth’s Association and Uniformity Tests as Probability Ratios.
On the neurosynth engine at 2 mm, each image is decoded in 6 to 11 s, the 257 voxels of the amygdala as the 38,141 of the motor map; on the neurostore engine, of twice as many studies, in 35 to 40 s, with 2 GB of memory beyond that of the engine. The overlap query takes three to four times as long.
The figures of this page are made by doc/tools/make_decoding_figures.py
on the neurosynth engine at 2 mm, which loads both images with .image
and runs the query it prints.