Error LoadBalance failure

General issues of interest both for network and
individual cell parallelization.

Moderator: hines

Post Reply
Keivan
Posts: 127
Joined: Sat Apr 22, 2006 4:28 am

Error LoadBalance failure

Post by Keivan »

I am experimenting new parallel neuron 7. I have just imported a dendritic tree to neuron without anything else. but I have encountered error activating multispilit from tools > parallel computing

Code: Select all

keivan@ubuntu:~/Desktop/TemporalGate$ nrngui main.hoc         
NEURON -- Release 7.0 (281:80827e3cd201) 2009-01-16           
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2008  
See http://www.neuron.yale.edu/credits.html                   

        1 
        1 
        1
oc>No children of somaA(0.7)
/home/keivan/neuron/nrn/x86_64/bin/nrniv: LoadBalance failure:
 near line 0
 {multisplit(multisplit_)}
                          ^
        LoadBalance[1].execerror("LoadBalance failure:")
      LoadBalance[1].ms_split(Vector[23], 0, 0.7, 0, 145.05)
    LoadBalance[1].multisplit(0, 145.05, Vector[23])
  ParallelComputeTool[0].use_thread()
and others
this is topology of my cell

Code: Select all

oc>topology()

|-----|       somaA(0-1)
       `----|       dendA1_0(0-1)
             `----|       dendA1_00(0-1)
             `----|       dendA1_01(0-1)
                   `----|       dendA1_010(0-1)
                         `----|       dendA1_0100(0-1)
                               `----|       dendA1_01000(0-1)
                               `----|       dendA1_01001(0-1)
                         `----|       dendA1_0101(0-1)
                               `----|       dendA1_01010(0-1)
                               `----|       dendA1_01011(0-1)
                   `----|       dendA1_011(0-1)
                         `----|       dendA1_0110(0-1)
                               `----|       dendA1_01100(0-1)
                               `----|       dendA1_01101(0-1)
                         `----|       dendA1_0111(0-1)
       `----|       dendA4_0(0-1)
             `----|       dendA4_00(0-1)
             `----|       dendA4_01(0-1)
                   `----|       dendA4_010(0-1)
                   `----|       dendA4_011(0-1)
                         `----|       dendA4_0110(0-1)
                               `----|       dendA4_01100(0-1)
                               `----|       dendA4_01101(0-1)
                         `----|       dendA4_0111(0-1)
                               `----|       dendA4_01110(0-1)
                                     `----|       dendA4_011100(0-1)
                                     `----|       dendA4_011101(0-1)
                               `----|       dendA4_01111(0-1)
     `----|       dendA2_0(0-1)
           `----|       dendA2_00(0-1)
           `----|       dendA2_01(0-1)
                 `----|       dendA2_010(0-1)
                       `----|       dendA2_0100(0-1)
                             `----|       dendA2_01000(0-1)
                             `----|       dendA2_01001(0-1)
                       `----|       dendA2_0101(0-1)
                             `----|       dendA2_01010(0-1)
                                   `----|       dendA2_010100(0-1)
                                   `----|       dendA2_010101(0-1)
                                         `----|       dendA2_0101010(0-1)
                                               `----|       dendA2_01010100(0-1)
                                               `----|       dendA2_01010101(0-1)
                                         `----|       dendA2_0101011(0-1)
                             `----|       dendA2_01011(0-1)
                                   `----|       dendA2_010110(0-1)
                                   `----|       dendA2_010111(0-1)
                                         `----|       dendA2_0101110(0-1)
                                         `----|       dendA2_0101111(0-1)
                 `----|       dendA2_011(0-1)
                       `----|       dendA2_0110(0-1)
                             `----|       dendA2_01100(0-1)
                             `----|       dendA2_01101(0-1)
                       `----|       dendA2_0111(0-1)
                             `----|       dendA2_01110(0-1)
                             `----|       dendA2_01111(0-1)
                                   `----|       dendA2_011110(0-1)
                                   `----|       dendA2_011111(0-1)
                                         `----|       dendA2_0111110(0-1)
                                         `----|       dendA2_0111111(0-1)
    `----|       hill(0-1)
          `----|       iseg(0-1)
                `----|       inode[0](0-1)
                      `----|       node[0](0-1)
                            `----|       inode[1](0-1)
                                  `----|       node[1](0-1)
                                        `----|       inode[2](0-1)
    `----|       dendA3_0(0-1)
          `----|       dendA3_00(0-1)
                `----|       dendA3_000(0-1)
                `----|       dendA3_001(0-1)
                      `----|       dendA3_0010(0-1)
                      `----|       dendA3_0011(0-1)
                            `----|       dendA3_00110(0-1)
                            `----|       dendA3_00111(0-1)
          `----|       dendA3_01(0-1)
                `----|       dendA3_010(0-1)
                      `----|       dendA3_0100(0-1)
                      `----|       dendA3_0101(0-1)
                `----|       dendA3_011(0-1)
                      `----|       dendA3_0110(0-1)
                      `----|       dendA3_0111(0-1)
 `----|       dendA5_0(0-1)
       `----|       dendA5_00(0-1)
       `----|       dendA5_01(0-1)
             `----|       dendA5_010(0-1)
                   `----|       dendA5_0100(0-1)
                   `----|       dendA5_0101(0-1)
             `----|       dendA5_011(0-1)
                   `----|       dendA5_0110(0-1)
                         `----|       dendA5_01100(0-1)
                         `----|       dendA5_01101(0-1)
                               `----|       dendA5_011010(0-1)
                               `----|       dendA5_011011(0-1)
                                     `----|       dendA5_0110110(0-1)
                                     `----|       dendA5_0110111(0-1)
                   `----|       dendA5_0111(0-1)
                         `----|       dendA5_01110(0-1)
                         `----|       dendA5_01111(0-1)
                               `----|       dendA5_011110(0-1)
                                     `----|       dendA5_0111100(0-1)
                                     `----|       dendA5_0111101(0-1)
                               `----|       dendA5_011111(0-1)
                                     `----|       dendA5_0111110(0-1)
                                           `----|       dendA5_01111100(0-1)
                                           `----|       dendA5_01111101(0-1)
                                                 `----|       dendA5_011111010(0-1)
                                                 `----|       dendA5_011111011(0-1)
                                                       `----|       dendA5_0111110110(0-1)
                                                       `----|       dendA5_0111110111(0-1)
                                     `----|       dendA5_0111111(0-1)
                                           `----|       dendA5_01111110(0-1)
                                           `----|       dendA5_01111111(0-1)
                                                 `----|       dendA5_011111110(0-1)
                                                 `----|       dendA5_011111111(0-1)
                                                       `----|       dendA5_0111111110(0-1)
                                                             `----|       dendA5_01111111100(0-1)
                                                             `----|       dendA5_01111111101(0-1)
                                                       `----|       dendA5_0111111111(0-1)
                                                             `----|       dendA5_01111111110(0-1)
                                                             `----|       dendA5_01111111111(0-1)
                                                                   `----|       dendA5_011111111110(0-1)
                                                                   `----|       dendA5_011111111111(0-1)
                                                                         `----|       dendA5_0111111111110(0-1)
                                                                               `----|       dendA5_01111111111100(0-1)
                                                                               `----|       dendA5_01111111111101(0-1)
                                                                         `----|       dendA5_0111111111111(0-1)
                                                                               `----|       dendA5_01111111111110(0-1)
                                                                                     `----|       dendA5_011111111111100(0-1)
                                                                                     `----|       dendA5_011111111111101(0-1)
                                                                                           `----|       dendA5_0111111111111010(0-1)
                                                                                                 `----|       dendA5_01111111111110100(0-1)
                                                                                                 `----|       dendA5_01111111111110101(0-1)
                                                                                           `----|       dendA5_0111111111111011(0-1)
                                                                               `----|       dendA5_01111111111111(0-1)
                                                                                     `----|       dendA5_011111111111110(0-1)
                                                                                     `----|       dendA5_011111111111111(0-1)
                                                                                           `----|       dendA5_0111111111111110(0-1)
                                                                                           `----|       dendA5_0111111111111111(0-1)
                                                                                                 `----|       dendA5_01111111111111110(0-1)
                                                                                                 `----|       dendA5_01111111111111111(0-1)
                                                                                                       `----|       dendA5_011111111111111110(0-1)
                                                                                                             `----|       dendA5_0111111111111111100(0-1)
                                                                                                             `----|       dendA5_0111111111111111101(0-1)
                                                                                                                   `----|       dendA5_01111111111111111010(0-1)
                                                                                                                   `----|       dendA5_01111111111111111011(0-1)
                                                                                                                         `----|       dendA5_011111111111111110110(0-1)
                                                                                                                         `----|       dendA5_011111111111111110111(0-1)
                                                                                                       `----|       dendA5_011111111111111111(0-1)
                                                                                                             `----|       dendA5_0111111111111111110(0-1)
                                                                                                             `----|       dendA5_0111111111111111111(0-1)
                                                                                                                   `----|       dendA5_01111111111111111110(0-1)
                                                                                                                         `----|       dendA5_011111111111111111100(0-1)
                                                                                                                         `----|       dendA5_011111111111111111101(0-1)
                                                                                                                               `----|       dendA5_0111111111111111111010(0-1)
                                                                                                                               `----|       dendA5_0111111111111111111011(0-1)
                                                                                                                   `----|       dendA5_01111111111111111111(0-1)
                                                                                                                         `----|       dendA5_011111111111111111110(0-1)
                                                                                                                         `----|       dendA5_011111111111111111111(0-1)
                                                                                                                               `----|       dendA5_0111111111111111111110(0-1)
                                                                                                                               `----|       dendA5_0111111111111111111111(0-1)

	1
Top
ted
Site Admin
Posts: 6300
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:
Contact ted

Re: Error LoadBalance failure

Post by ted »

To discover the cause of the failure, and recommend a fix, it will be necessary to reproduce the problem. Can you email the necessary hoc, ses, and mod files to
ted dot carnevale at yale dot edu
?
(see No rar, doc, or winmail.dat files, please! viewtopic.php?f=28&t=1048).
Top
Keivan
Posts: 127
Joined: Sat Apr 22, 2006 4:28 am

Re: Error LoadBalance failure

Post by Keivan »

I've sent required files
Top
ted
Site Admin
Posts: 6300
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:
Contact ted

Re: Error LoadBalance failure

Post by ted »

These tests were performed with the most recent alpha version of NEURON--7.1 (333:4aa881b7692d) 2009-08-20.

The problem comes from the way spatial discretization is specified.

If I import the complete model into a CellBuilder, save the CellBuilder to a ses file, exit NEURON, then use the CellBuilder to create a new model, multisplit succeeds when the entire cell is discretized according to:
the d_lambda rule with criterion of 0.1 lamba_100 (i.e. each section's nseg is an odd number just large enough that no segment is longer than 0.1 AC length constant at 100 Hz)
OR
the dX rule with criterion of 10 um (each section's nseg is an odd number just large enough that no segment is longer than 10 um).

Multisplit fails if I try it with just the "non-axon" part of the cell, i.e. the sections created by
nrngui Morphology.nrn
But it succeeds if I import the "non-axonal" stuff into a CellBuilder, save that to a ses file and use that ses fiel to create a new model as described above. And it succeeds when applied to a model created by combining the "non-axon" CellBuilder with the hoc-specified axon model like so:

Code: Select all

load_file("morphology.ses") // contains the non-axonal part of the cell
load_file("Axon.nrn")
So the nseg values specified by Morphology.nrn are such that NEURON's automatic load balancing algorithm is unable to find a parcellation that achieves load balance.

The nseg values seem more than slightly arbitrary. For example, somaA, which is 5 um in diameter at midpoint and 26 um long has nseg==42, which is probably 42 times larger than necessary for good spatial accuracy of electrical signals. Also, more than half of the sections (82 of 161) have even values for nseg, so they won't have nodes at the 0.5 location. Attempts to determine or set the value of any range variable at 0.5 will actually access the value of the range variable at one of the two adjacent nodes, i.e. 0.5 + 1/(2*nseg) or 0.5 - 1/(2*nseg). Which of these two locations will be accessed? Depends on roundoff error (will double precision arithmetic decide that 0.5 is closer to one node than the other?). Do you have a special reason for sticking with these nseg values, or can you change them?
Top
Keivan
Posts: 127
Joined: Sat Apr 22, 2006 4:28 am

Re: Error LoadBalance failure

Post by Keivan »

Do you have a special reason for sticking with these nseg values, or can you change them?
there is no special reason using these nseg values. I will try your recomendations on my machine.
Top
hines
Site Admin
Posts: 1691
Joined: Wed May 18, 2005 3:32 pm

Re: Error LoadBalance failure

Post by hines »

The best solution is to have one segment for the soma. It does not matter how many or few segments the other sections have. Multisplit works only when no subtree has more than two split points and, although the soma uniquely is allowed to have a split point in its interior, there can be only one of them. So it helps if all the branches from a soma come from not more than two locations.

By the way, it seems that balance is good with two threads (8 pieces, 0.1% balance) but the cell is too small (complexity 1548) to get any improvment with two threads. However there is a significant benefit (with the passsive model) of turning on cache efficiency (factor of two). Threads may help if you add a lot of channels.
Top
Keivan
Posts: 127
Joined: Sat Apr 22, 2006 4:28 am

Re: Error LoadBalance failure

Post by Keivan »

The best solution is to have one segment for the soma. It does not matter how many or few segments the other sections have. Multisplit works only when no subtree has more than two split points and, although the soma uniquely is allowed to have a split point in its interior, there can be only one of them. So it helps if all the branches from a soma come from not more than two locations.
Is this means that I should remove my somaA section and create an identical soma with hoc commands and attach it to the rest of the tree?
Top
ted
Site Admin
Posts: 6300
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:
Contact ted

Re: Error LoadBalance failure

Post by ted »

Just change its nseg to 1.
Top
Keivan
Posts: 127
Joined: Sat Apr 22, 2006 4:28 am

Re: Error LoadBalance failure {solved}

Post by Keivan »

thank you ted. I am now enjoying my parallel neuron.
Top
ted
Site Admin
Posts: 6300
Joined: Wed May 18, 2005 4:50 pm
Location: Yale University School of Medicine
Contact:
Contact ted

Re: Error LoadBalance failure

Post by ted »

Thanks for asking the original question. I didn't know this:
Multisplit works only when no subtree has more than two split points and, although the soma uniquely is allowed to have a split point in its interior, there can be only one of them.
Top
Post Reply

Return to “Parallel NEURON”