how to read from, and record data into binary files?

[itamarch]

I am simulating an IClamp through cells and recording Voltage vs time.
I have no problem recording and reading data from Ascii files, but the volume of these files becomes incredibly large very quickly. I want to save the data in binary format. Is it possible? How is it done?

[ted]

The most powerful and flexible way to deal with file i/o is to use the File class's
metods, which are described here
http://www.neuron.yale.edu/neuron/stati ... .html#File

Using binary formats risks generating future problems for oneself because of
cross-application and cross-platform incompatibilities.

[miller]

It works quite well with a method of the vector class called vwrite.
Using the precision argument you can save your data as 4 byte floats which will reduce needed space by half.
The first 4 bytes in the generated file interpreted as a 4 byte integer is the number of samples in the file and the second 4 bytes is the precision argument you passed to the vwrite method. From the 9th bytes on come the data.
You have to take care of the byte order. For example if you write the data on a unix machine and read them on a windows pc you have to reorder the bytes from big endian to little endian. (Matlab function fread can do it for you)

[miller]

Here is an example of code which read binary files written with Vector class method vwrite in Matlab:

Code: Select all

% [data,errmsg]=nrn_vread(FileName,machineformat)
% machineformat like in fopen (i.e. 'b' for big endian, 'l' for little endian)
%
% reads binary files, that was written with Vector.vwrite in NEURON

function [data,errmsg]=nrn_vread(FileName,machineformat)

data = [];

[fid,errmsg] = fopen(FileName,'r',machineformat);
if fid==-1
	return;
else
	errmsg = sprintf('File opened successfully');
end

[header,cnt]=fread(fid,2,'int32');
if cnt~=2
	errmsg = sprintf('Could not read the vwrite header');
	fclose(fid);
	return;
end

precision = 'double'; % to avoid Matlab warning
if header(2)==4
	precision = 'double';
elseif header(2)==3
	precision = 'float32';
elseif header(2)==5
	precision = 'int';
else
	errmsg = sprintf('Unsupported precision argument');
	fclose(fid);
	return;
end

[data,cnt]=fread(fid,inf,precision);
if cnt~=header(1)
	errmsg = sprintf('Only %d instead of %d Samples read',cnt,header(1));
	data = [];
	fclose(fid);
	return;
else
	errmsg = sprintf('Successfully read %d Samples',cnt);
end

fclose(fid);

[ted]

Excellent, Konstantin! Score a big point for portability of binary files.

[eacheon]

I am simulating an IClamp through cells and recording Voltage vs time.
I have no problem recording and reading data from Ascii files, but the volume of these files becomes incredibly large very quickly. I want to save the data in binary format. Is it possible? How is it done?

I am actually zipping the ascii files and am pretty comfortable with the results I get (may not be as efficient as binaries).

Then one can read in these zipped files in python (which is very simple with python's standard zipfile module), stub them in a more organized data structure that is pickled to one file saving everything produced in one run of NEURON simulation (from t=0 to t=tstop).

[mflynn]

I want to include metadata about the vectors I'm writing to a binary file. Can I include information like descriptions of the experiment, type of data(voltage, conductance, current), location of the recording, etc. for each vector? The corollary of this is would be can I use fread in matlab to read it out as plain text?

Thanks,
Mark

[eacheon]

I want to include metadata about the vectors I'm writing to a binary file. Can I include information like descriptions of the experiment, type of data(voltage, conductance, current), location of the recording, etc. for each vector? The corollary of this is would be can I use fread in matlab to read it out as plain text?

Thanks,
Mark

It is up to you to decide how to manage data outside of NEURON. If you decide to use fread in Matlab then you have to roll your own file format.

At the same time there're many packages helping you doing this kind of things. hdf5, just to name an example which I saw people are using. I myself is doing exactly the same thing as you said (metadata e.g. description, parameters, name of recorded vectors) in IPython, using a package called PyDSTool which supports these "annotated array with natural coordinate names"

In [6]: from cPickle import load

In [7]: trace = load(open("some_data_filename"))

In [8]: trace
Out[8]:
Pointset stimtime=90000,35,name=2grp (parameterized)
Independent variable:
t: [ 0. 0.025, ..., 199.975 200. ]
Coordinates:
v-soma: [-69.99999856 -69.99999856, ..., -69.96180949 -69.96182706]
v-55.9734;somewhere_in_dist55.973428: [-69.99999881 -69.99999881, ..., -69.96925701 -69.96927116]
v-41.0024;somewhere_in_dist41.002428: [-69.99999816 -69.99999816, ..., -69.95090783 -69.95093041]
g-AMPA[0]: [ 0. 0., ..., 0. 0.]
g-NMDA[0]: [ 4.25557790e-13 4.25557790e-13, ..., 4.27101522e-13 4.27100811e-13]
g-NMDA[224]: [ 1.06389450e-11 1.06389450e-11, ..., 2.95209464e-07 2.95073667e-07]
Labels by index: Empty

In [9]: trace.params.keys()
Out[9]:
['gabaa_gmax',
'stim_fn',
'view_stim',
'nmda_in_ampa_gmax',
'ampa_delay',
'spkdir',
'celsius',
'vm',
'nmda_gmax',
'verbose',
'ampa_tau2',
'G_pas',
'ampa_tau1',
'tstop',
'E_leak',
'spike_threshold',
'mpa_gmax',
'stiminfo']