Contributing to NeuroLang#
Thank you for considering a contribution! This page explains how to set up a development environment and submit changes.
Prerequisites#
Python ≥ 3.12
git
uv (fast Python package manager) — install with:
curl -LsSf https://astral.sh/uv/install.sh | sh
Or on Windows (PowerShell):
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
Development Install#
git clone https://github.com/NeuroLang/NeuroLang.git
cd NeuroLang
uv sync --all-packages --all-extras --no-extra embeddings
This installs NeuroLang’s four packages in editable mode, in the
environment .venv of the clone, together with all development
(pytest, pytest-cov) and documentation (Sphinx) dependencies. To
install them in another environment, uv pip install -e ".[apps,dev,doc]"
does the same from the root of the clone, whose pyproject.toml is a
project, never published, depending on the four.
The Packages#
The repository is a uv workspace of four packages, each of which imports only the ones before it:
|
the expression engine: expressions, walkers, types, logic, configuration |
|
relational algebra and its sets, Datalog, probabilistic Datalog |
|
the languages and frontends, brain regions |
|
the command line, its interactive mode, the query assistant, the engines |
.importlinter states these layers, which CI checks with
uv run lint-imports. A module put in the wrong package, or importing
one above it, fails that check. The modules keep their former names under
neurolang: neurolang.datalog is nl_core.datalog, and
neurolang.utils.cli is neurolang_apps.utils.cli. Code inside the
packages imports the new names.
The tests of a package are in it, beside the modules they test, and use only that package and the ones before it. CI runs the tests of the packages a pull request changes and of those built on them.
Running the Tests#
uv run pytest packages/
or those of one package, e.g. uv run pytest packages/nl_core. To run
with coverage:
uv run pytest --cov=nl_expr --cov=nl_core --cov=neurolang \
--cov=neurolang_apps packages/
The test suite should pass fully before you submit a pull request.
Building the Documentation#
make -C doc html
This will build the HTML docs into doc/_build/html/. Open
doc/_build/html/index.html in your browser to preview.
Note
The documentation build executes only the gallery examples whose file
name matches plot_load_destrieux (filename_pattern in
doc/conf.py); the other examples are rendered without being run.
Running them requires the neuroimaging packages (nibabel, nilearn) and
downloads data. To skip running the gallery entirely during a quick doc
build, disable plot_gallery:
make -C doc html SPHINXOPTS="-D plot_gallery=0"
Running the REPL in the Browser Locally#
The REPL in the browser runs NeuroLang on Pyodide, from a static page. To run that of your checkout on your machine:
uv run --extra apps neurolang-repl-serve
The first run builds the page’s bundle (NeuroLang’s wheels and the data of
the neurosynth engine, a few minutes) in
~/.cache/neurolang/repl-site; later runs serve it at once,
--rebuild builds it again from the checkout, and --clear removes the
bundle built before and then builds it from nothing (the text embeddings
below are kept). The page opens at
http://localhost:8766/_static/repl/index.html; the browser loads Pyodide
from its CDN. --host 0.0.0.0 --server-name NAME serves the other
machines of the network, which reach it by NAME (see below),
--port changes the port, --no-browser does not open the page, and
--help lists the options. The command is that of the workspace’s root
project (tools/serve_repl).
CI builds the REPL in the browser for master only, deployed with the
documentation: its bundle takes an hour on a runner’s CPU when the
embeddings of the studies are not kept. A pull request’s CI uploads its
documentation, the artifact repl-pr-NUMBER (kept 30 days), without a
REPL that can start; to try the REPL of a branch, build it locally with
neurolang-repl-serve, as above.
The same page can run NeuroLang in the Python of this machine rather than in the browser (Serving the REPL tells how to serve it to a group of users, with HTTPS, accounts, the AI assistant’s keys and a service):
uv run --extra apps --extra tui --extra llm neurolang-repl-serve --local --data-dir neurolang_data
builds no bundle: the page’s worker (local_worker.js) sends what it is
asked to the server, which builds the engine from --data-dir (by default
neurolang_data, as neurolang-query), with all of its analyses
(lift, ratios…) unless --light, and runs the page’s REPL
(neurolang_apps.utils.tui.local_backend). Queries take the time and the
memory they take in neurolang-query; Stop interrupts the query on the
server, and the AI assistant is the one configured on this machine, unless a
key is given in the page. The server answers this machine only, unless
--host: whoever it answers runs queries, and saves files, with its
Python.
Without a password (below), the server answers only the requests sent to
localhost, 127.0.0.1, ::1, the --host it is given (unless
0.0.0.0 or ::), and the names of --server-name NAME (repeat it
for several): a page of another site whose name it made lead to your
machine (DNS rebinding) would otherwise run queries in its Python. The
other machines of the network need --server-name with the name, or the
address, by which they reach it, e.g. --host 0.0.0.0 --server-name
repl.lab.org; the server’s answer to another name says so. With a
password, any name is answered. With a password or without, a POST is
answered only if it is the page’s: in JSON (Content-Type:
application/json, which a form of another site cannot send), and from
this site if the browser gives its Origin; otherwise any site you visit
could run queries here. A connection that says nothing for a minute (its
TLS handshake, its request) is closed, and the body of a request to
/api/ is a JSON object of 1 MB at most.
To serve it over HTTPS, give a certificate and its private key, PEM files (the key may be in the certificate’s file):
uv run --extra apps --extra tui --extra llm neurolang-repl-serve --local --host 0.0.0.0 --server-name repl.lab.org --cert cert.pem --key key.pem
A key with a passphrase is opened with the first line of
--key-password-file FILE (readable by you only; the line as it is, but
for its line ending), else the passphrase is
asked in the terminal; a service needs the file. If they cannot serve, the
server says why: the certificate’s file has no key, the key is not that of
the certificate, or the passphrase is not that one. A PEM file is text,
its first line -----BEGIN CERTIFICATE----- or -----BEGIN PRIVATE
KEY----- (ENCRYPTED PRIVATE KEY: with a passphrase); openssl x509
-in cert.pem -noout -subject and openssl pkey -in key.pem -noout read
them, and openssl x509 -inform DER -in cert.der -out cert.pem converts a
binary (DER) one.
To have it ask a password, put one on the first line of a file (readable by
you only: chmod 600; the line as it is, spaces included, but for its
line ending) and give --password-file FILE: the browser asks
for it (any user name will do) before the site’s first file, and the
server answers no request without it. Over the network it needs
--cert, as the password goes with each request. With --local,
--env-file ENV gives those with the password the key of the AI
assistant: ENV’s NAME=VALUE lines are put in the server’s environment,
where the provider of ~/.config/neurolang/llm.yaml reads its key (the
variable its api_key_env names, e.g. ALBERT_API_KEY=...). The key
stays on the server; a key given in the page replaces it. With a password,
the visitors share one session of the REPL, its rules and a key one of them
gives, and a restart restarts it for all. The site’s main page, /, leads
to the REPL.
For several users, each with sessions of their own, give a users file
instead of a password, --users users.yaml, readable by you only:
users:
ann:
password: scrypt$16384$8$1$... # neurolang-repl-serve --add-user
groups: [admin, lab]
env: # optional: the key of her assistant
ALBERT_API_KEY: ...
bob:
password: scrypt$16384$8$1$...
groups: [lab] # the key of the group lab
groups:
lab:
llm: # the assistant of the group's users
provider: openrouter # anthropic, openai, openrouter, custom
model: anthropic/claude-sonnet-5.5
key: ...
own_key: false # its users may not give their own key
guest:
env: # or a variable the server's
ALBERT_API_KEY: ... # configuration (llm.yaml) reads
A user’s assistant answers with the key their page gives (unless they may
not give their own), else their own llm (which an admin sets, or they
save on their account page), else that of their groups (the first by name
that has one), else the variables (env) of theirs and of their groups
with the server’s configuration, else the server’s (--env-file), else
none: they give theirs in the page. Whether a user may give their own key
(in the page or on their account page) is their own_key, else no if one
of their groups says no, else yes. A file of the users alone, without
users: and groups:, is read as well.
A user’s groups are labels (groups: [admin, lab]); those of the
group admin have the admin console, at /admin: there for them only
(any other user is refused), it lists the users, their groups and whose key
their assistant answers with (their own, the server’s, or none: they give
theirs in the page; a key itself is never shown), adds a user, changes a
password, groups or key, removes a user (and ends their sessions), and
lists the sessions, which it can end. Its changes are written in the users
file at once (comments in it are lost), and reach the open pages of the
users they change at once; the file is read again when you edit it by hand
or from the command line, which reaches a user’s pages at their next start.
Every user has an account page, /account, linked from the REPL’s
page: their name and groups, whose assistant they have, a form that saves
their own (provider, model and key, kept on the server and never shown
again), if they may, and a form that changes their password (the
current one first; the new one of 8 characters at least, as in the console
and from the command line), after which they stay logged in in that
browser, and log in again in the others. A forgotten password is reset by
an admin.
The users are managed from the command line too, which makes the file for the first one, the first admin; each asks the password or the key, which it does not echo:
neurolang-repl-serve --users users.yaml --add-user ann --groups admin,lab
neurolang-repl-serve --users users.yaml --set-password ann
neurolang-repl-serve --users users.yaml --set-groups bob --groups lab
neurolang-repl-serve --users users.yaml --set-key bob ALBERT_API_KEY # empty: removes it
neurolang-repl-serve --users users.yaml --set-group-key lab ALBERT_API_KEY
neurolang-repl-serve --users users.yaml --set-group-llm lab openrouter anthropic/claude-sonnet-5.5
neurolang-repl-serve --users users.yaml --set-llm ann anthropic # empty key: removes it
neurolang-repl-serve --users users.yaml --own-key group:lab no # or NAME yes|no|inherit
neurolang-repl-serve --users users.yaml --remove-user bob
neurolang-repl-serve --users users.yaml --list-users
(with uv run --extra apps before them from a checkout); a server
running reads the file again. The
server takes a POST only from its own page (in JSON, from its origin), so
that another site cannot have a logged-in browser change anything.
With --feedback-db feedback.sqlite, the ratings of the page are stored
on the server, in a SQLite database (that of neurolang-feedback-server),
with the name and the groups of the user who rates; the console shows the
latest and exports, for each engine, the rated answers of the assistant
as YAML (its examples). It needs --local. The sessions write it at once (WAL mode), which needs it on a
local disk, not a network one.
With --users, a page opened without a login leads to a login page,
/login, which gives a login cookie, until Log out (which ends the
user’s logins in every browser; /logout, at
the top of the REPL with --local, beside the user’s name, linked to
their account page); behind a proxy of HTTPS, --secure-cookie sends
that cookie over HTTPS only. With --guests (and --local), the
login page lets visitors in as guests, who bring their own key; see
Serving the REPL. With --local, each tab of a
user has a session of its own, in a process of its own: a builder process
per engine builds it once and forks a session’s process from it, so that the
sessions share the memory of the engine’s data until they write it, and
each has its own rules, AI assistant and queries, which run at the same
time; a stop or a restart is that of the tab’s session only (a reload
keeps it, a duplicated tab has a new one). A user’s
env is put in the environment of their sessions only: the key of their
assistant, over that of --env-file, and a key their page gives over
both. At most --max-sessions (8, at least 1) run at once: a tab
starting one more ends the idlest session of its user, or another’s unused
for a minute. A session unused for --idle-minutes (120, above 0) ends
(its page says to restart it). The users can
read what the server’s account can, and their sessions share the data
directory, where the files they save by a relative name may collide; give
it to people you trust. The feedback server, where the sessions’ rated
answers may go, takes their writes at once (SQLite in WAL mode).
To keep it serving, as a service of your user started now and at each login,
add --daemon install to the options it serves with; --daemon
reinstall replaces it with the new options (once their bundle is built
and the service’s Python can serve, so that a failure leaves the service
as it was), and --daemon remove, with no other option, stops and
removes it; if systemd or launchd fails to start it, it is removed and the
error told:
uv run --extra apps --extra tui --extra llm neurolang-repl-serve --daemon install --local --host 0.0.0.0 --cert cert.pem --key key.pem --users users.yaml --env-file llm.env --feedback-db feedback.sqlite
On Linux it is a unit of systemd’s user manager,
~/.config/systemd/user/neurolang-repl.service (journalctl --user
-u neurolang-repl shows its log; loginctl enable-linger keeps it
running after you log out); on macOS a launchd agent,
~/Library/LaunchAgents/org.neurolang.repl.plist, logging to
~/Library/Logs/neurolang-repl.log. install builds the bundle
first, where you see it; the service runs the Python of the environment
that installs it (with uv run, the checkout’s .venv/bin/python;
--python PATH chooses another), with that environment’s PATH and the
paths made absolute, but not the other variables of your shell (API keys). It uses that environment as it is: a
later uv sync without --extra apps (and the other extras it was
installed with) removes their packages from it, and the service then fails
to start until they are back (uv run removes nothing). A port below
1024 needs a service of the system, which this does not install.
query_similarity needs the text embeddings in the bundle: the SPECTER2
query model and the embeddings of the studies. They are kept in
~/.cache/neurolang/repl-embeddings (--embeddings) and made
there, the first time, by a build with torch: uv run --extra apps --extra
embeddings neurolang-repl-serve --rebuild, which downloads SPECTER2 and the
abstracts of the studies from PubMed and embeds them (minutes on a GPU,
longer on a CPU). Without them the page works, but query_similarity
says that it was built without them.
Submitting a Pull Request#
Fork the repository on GitHub.
Create a feature branch:
git checkout -b my-featureMake your changes.
Ensure all tests pass:
uv run pytest packages/Add a changelog entry under Unreleased in
doc/changelog.rst.Update docstrings for any changed public API.
Push your branch and open a pull request against
master.
CI runs on every pull request, also one whose base is the branch of another
pull request (a stacked pull request): the tests are those of the packages
it changes with respect to its base. Its documentation is the artifact
repl-pr-NUMBER; the REPL in the browser of a branch is built locally
(see Running the REPL in the Browser Locally).
PR checklist:
[ ] Tests pass (
uv run pytest packages/)[ ] New code has docstrings (NumPy format)
[ ] Changelog entry added to
doc/changelog.rst[ ] No new Sphinx build warnings (
make -C doc html)
Code Style#
NeuroLang follows PEP 8 with
NumPy-style docstrings. Run flake8 before committing:
uvx flake8 packages/
Adding a New Dataset Engine#
The neurolang-query CLI uses a declarative engine registry to manage
dataset backends. To add a new engine:
Write a YAML entry in
packages/neurolang_apps/neurolang_apps/utils/engines/engines.yaml:my_dataset: description: "Short description of the dataset" requires_mni_mask: false python_init: "neurolang_apps.utils.engines.my_dataset.init" datalog_init: | derived_pred(X) :- base_pred(X) relations: base_pred: "my_dataset/base_data.csv" predicates: my_predicate: arity: 2 columns: [name, value] description: "What this predicate represents"
The optional fields
datalog_init(inline Datalog rules) andrelations(CSV/TSV files loaded as predicates) run after the Python init. See Concepts for details. Example queries (examples,assistant_examples) and the query assistant’sassistant_guidancego inengines/my_dataset/examples.yamlandengines/my_dataset/guidance.yaml, merged into the entry (see Engine Configuration (engines.yaml)).If your engine needs custom Python logic (data downloads, non-trivial processing), create the init module at
packages/neurolang_apps/neurolang_apps/utils/engines/my_dataset/init.pythat defines:from pathlib import Path import nibabel as nib from neurolang.frontend import NeurolangPDL def init_engine( nl: NeurolangPDL, mask: nib.Nifti1Image | None, data_dir: Path ) -> None: """Register symbols and load data into *nl*.""" # Register predicates with nl.add_tuple_set, etc.
If your engine needs the common neuroimaging symbols (
agg_count,agg_create_region,agg_create_region_overlay,principal_direction,region_union), setuse_base_symbols: truein the YAML entry instead of callinginit_base_engine()from Python:my_dataset: description: "..." requires_mni_mask: true use_base_symbols: true builtins: [exp, log, startswith] ...
This replaces the need for a Python init script when your engine only needs the standard neuroimaging symbols plus declarative YAML fields (
atlases,relations,datalog_init, etc.). See thedestrieuxengine entry for a complete example of a YAML-only engine.For engines that require custom data processing (like NeuroSynth’s peak coordinate conversion), you still write a Python init module. In that case, import and call
init_base_engineif you need the base symbols:from neurolang.utils.engines.base import init_base_engine def init_engine(nl, mask, data_dir): init_base_engine(nl, mask) # ... custom logic ...
If using
use_base_symbols: truein the YAML, do not also callinit_base_enginefrom Python — the registry handles it for you.Add unit tests in
packages/neurolang_apps/neurolang_apps/utils/tests/test_cli.py, specifically in theTestEngineRegistryclass to verify that your engine is discoverable and its config loads correctly.Run
uv run pytest packages/neurolang_apps/neurolang_apps/utils/tests/test_cli.pyto confirm everything passes.
Reporting Bugs#
Open an issue on the GitHub issue tracker. Include:
NeuroLang version (
python -c "import neurolang; print(neurolang.__version__)"if available)Python version
Operating system
Minimal reproducible example