Using thread-safe source_var/target_var to replace POINTER variables

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
bcaldwell

Using thread-safe source_var/target_var to replace POINTER variables

Post by bcaldwell »

I'm working on making a simplified HNN model threadsafe so that I can turn on cvode.cache_efficient(1) for about a 30% performance improvement. However, some of the mod files that implement dipole recording use POINTER variables for (1) aggregation and (2) connect to voltages as inputs. Instead of POINTER I was inspired to use source_var()/target_var() by this post:
viewtopic.php?p=5286

The first case seems necessary, but the second is done within the context of a single section. So I wonder if source_var()/target_var() is still an appropriate substitute or is there something simpler?

In fact, in my POC code, I can use source_var()/target_var() for all but one case of connecting the input voltage. If I uncomment line #233 (and remove 'pass'), then I am unable to run back to back simulations and get a segfault on reinitialization (stack trace below). The documentation warns against using section position 0 or 1 for variable step methods, but I am using fixed step. Could you help shed any light on whether this use is appropriate?

Thanks!

I'm using NEURON 7.7.1 (3078695) 2019-06-08

Code: Select all

(/users/bcaldwe2/anaconda/mne) [bcaldwe2@node1155 mne-neuron]$ mpiexec -np 1 nrniv -mpi -python -nobanner examples/validate_dipole.py                                numprocs=1
took 90.81s for simulation (avg=90.81s)
*** Error in `nrniv': corrupted double-linked list: 0x000000000813d9c0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7ada4)[0x7f1195ecdda4]
/lib64/libc.so.6(+0x7c595)[0x7f1195ecf595]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrniv.so.0(nrn_cache_prop_realloc+0x7f3)[0x7f1199a486e3]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrnoc.so.0(+0x34919)[0x7f1199d04919]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrnoc.so.0(v_setup_vectors+0x11ca)[0x7f1199d05b1a]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrnoc.so.0(verify_structure+0x45)[0x7f1199cfa4d5]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrnoc.so.0(nrn_finitialize+0x30)[0x7f1199cfab00]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrnoc.so.0(finitialize+0x3a)[0x7f1199cfb0ca]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrnoc.so.0(hoc_call+0x18e)[0x7f1199d0ceae]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrnpython.so.0(+0x11848)[0x7f1198223848]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrniv.so.0(_ZN10OcJumpImpl7fpycallEPFPvS0_S0_ES0_S0_+0x3e)[0x7f1199a248ce]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrnpython.so.0(+0x119ce)[0x7f11982239ce]
/users/bcaldwe2/anaconda/mne/lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x8a)[0x7f1197574fda]
/users/bcaldwe2/anaconda/mne/lib/libpython3.6m.so.1.0(+0x20ec87)[0x7f1197609c87]
/users/bcaldwe2/anaconda/mne/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x162b)[0x7f119756375b]
/users/bcaldwe2/anaconda/mne/lib/libpython3.6m.so.1.0(+0x20e833)[0x7f1197609833]
/users/bcaldwe2/anaconda/mne/lib/libpython3.6m.so.1.0(+0x20ede6)[0x7f1197609de6]
/users/bcaldwe2/anaconda/mne/lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x11a1)[0x7f11975632d1]
/users/bcaldwe2/anaconda/mne/lib/libpython3.6m.so.1.0(+0x20e833)[0x7f1197609833]
/users/bcaldwe2/anaconda/mne/lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x3e)[0x7f119760a23e]
/users/bcaldwe2/anaconda/mne/lib/libpython3.6m.so.1.0(PyEval_EvalCode+0x1c)[0x7f1197561e5c]
/users/bcaldwe2/anaconda/mne/lib/libpython3.6m.so.1.0(PyRun_FileExFlags+0xb7)[0x7f1197550317]
/users/bcaldwe2/anaconda/mne/lib/libpython3.6m.so.1.0(PyRun_SimpleFileExFlags+0xf4)[0x7f11975504f4]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrnpython.so.0(nrnpy_pyrun+0x2c)[0x7f119821feac]
/users/bcaldwe2/nrn/build/x86_64/lib/libnrnpython.so.0(nrnpython_start+0x268)[0x7f1198220428]
nrniv(ivocmain+0x35a)[0x40248a]
nrniv(main+0x56)[0x401cf6]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1195e74b35]
nrniv[0x401d7f]
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Using thread-safe source_var/target_var to replace POINTER variables

Post by hines »

use POINTER variables for (1) aggregation
I don't think you can use the source_var/target_var pattern for this purpose because there is no way for one target to receive an increment
from many sources. (the opposite is natural: a source can copy its value to many targets.) This assumes I correctly interpreted "aggregation" as
'sum += vars_from_many_places'.
However, in the context of dipoles, I don't think there needs to be a threadsafe problem with POINTER. The typical strategy is to
aggregate all the variables in each thread into a trajectory vector and sum the vectors in each process at the end of a run, and then allreduce them
into a single trajectory for each recording location. At least that is what is done for LFP. Apologies if I mistakenly imagined an analogy with
the computation of dipoles and LFP
bcaldwell

Re: Using thread-safe source_var/target_var to replace POINTER variables

Post by bcaldwell »

Thank you. Yes, you've correctly interpreted the need for aggregating dipole measurements across various points. In fact, I was able to use that scheme in this line of code:

Code: Select all

self.dpls.append(h.Vector().record(dpp._ref_Qtotal))
I'm not sure what the computational cost is at each time step of recording into numerous Vectors (for each segment), but it appears to be more than offset by the benefit of turning on cache_efficient().

However, my remaining question is on using this pattern for connecting segment voltages to the density mechanism specified in HOC. This statement seems to cause corruption in NEURON when i=0:

Code: Select all

pc.target_var(dpp, sect(loc[i]).dipole._ref_pv2, var_index)
If I run the simulation on a new network (re-created with different parameters), I get the error messages above. The results for the first simulation are different than using POINTER with the problematic line commented out. The difference could also (likely) be related to my implementation of aggregation using many vectors as you suggested. I'm still debugging that.

In general though, would you say like this is a reasonable pattern for replacing POINTER in non-gap junction scenarios?
hines
Site Admin
Posts: 1682
Joined: Wed May 18, 2005 3:32 pm

Re: Using thread-safe source_var/target_var to replace POINTER variables

Post by hines »

computational cost is at each time step of recording into numerous Vectors (for each segment
You definitely want only one vector per thread for storage for a recording location. A POINTER for each segment in a thread would all point to
the same scalar accumulator and the mod file would add a weighted value to the accumulator
at each time step and for each time step the accumulator would append to a Vector. I believe you would still obtain all the benefits of
cache efficiency
Post Reply