Documenting one's work with NEURON (digital notebooks)

[kathryn]

Now I'm using NEURON on a daily basis, I've been forced to think about documenting my work.

I'm familiar with documenting wet experiments in a bound lab notebook, but what about in silico work? It seems wasteful (of my time, paper and printer toner) to repeatedly print and affix hard copies of .hoc and .mod files, and any generated data, to the bound notebook. It makes more sense to generate a digital notebook. In such a notebook I could document -- in one place -- my ideas and progress with NEURON-based experiments, including the model code as it evolves, and the subsequent data. [NB: Our lab uses Microsoft Windows exclusively, so I'm confined to that operating system.]

There is an increasing demand for lab notebook software; a Google search turned up a few options (and many discussions of this topic!), but I didn't find what I was looking for. Obviously the ideal software will rely on fairly universal (and cross-platform) file formats. We must be able to read the files years down the track, even when the original software has become obsolete.

A word processor (of some description) would probably offer the simplest solution; I could format my notes and easily paste in/keep track of the code as it changes. Upside: no need to separately archive the many superseded files of code. Downside: the document would be clogged with code, and difficult to follow.

It'd be great to create sections (i.e. of model code) in the document which only show the first line or two by default. The user would then click an inline button to expand the section of choice, and view the full body of code. Does any existing software offer this capability?

Finally:
Do you have any experience with digital lab notebooks? If so, what was your solution? Are you happy with it?

[ted]

Good questions, kathryn.

I'm familiar with documenting wet experiments in a bound lab notebook, but what about in silico work? It seems wasteful (of my time, paper and printer toner) to repeatedly print and affix hard copies of .hoc and .mod files, and any generated data, to the bound notebook. It makes more sense to generate a digital notebook. In such a notebook I could document -- in one place -- my ideas and progress with NEURON-based experiments, including the model code as it evolves, and the subsequent data. [NB: Our lab uses Microsoft Windows exclusively, so I'm confined to that operating system.]

Too bad you're stuck with MSWin. But all is not lost.

Before getting into anything at all sophisticated, let me suggest that you exploit the file system as a way to keep things straight. You can do this regardless of whatever other measures you take, and it will even work under MSWin. Make a separate directory tree for each of your projects. The root of each tree could contain your doc files, pdfs of relevant references, txt files that contain comments to yourself. It would also contain a subdirectory that contains all the source code for that project--the hoc, mod, and ses files, plus any data files that are needed or generated by your program.

Code management is best done with a version control system. See http://en.wikipedia.org/wiki/Revision_control, and also http://www.ericsink.com/scm/source_control.html for a fairly quick and easy "all about it" article.

It's a huge mistake to get locked into a proprietary, MSWin-specific package, since some of the best version control systems are open source, available at no charge, run on all popular operating systems, and aren't likely to become obsolete any time soon. The top names that come to my mind are subversion http://subversion.tigris.org/, git http://git-scm.com/, and mercurial http://www.selenic.com/mercurial/wiki/. Advantages of subversion are that it has a large user base, and O'Reilly has made the book "Version Control with Subversion" freely available on line (http://svnbook.red-bean.com/). NEURON's own source code is managed with subversion, and also takes advantage of Trac http://trac.edgewall.org/, a wiki that provides a very flexible interface for managing a subversion repository.

Notes related to the code can be embedded in the source files as comments, or kept in separate plain text files. I also use plain text files for notes to myself. Plain text is portable across operating systems and will never become obsolete. If you haven't already done so, abandon NotePad and get a "real" text editor like NoteTab or one of the other free text editors that is available for MSWin (mentioned elsewhere on the NEURON Forum, maybe in the Hot tips area, or maybe in the MSWin section). They can read and write MSWin, Mac, and UNIX text files, offer tabbed views of multiple files, many can do powerful multidocument search & replace, and some allow you to create macros to handle common tasks.

For "internal documentation" that requires text plus figures, html files are convenient--much more efficient of space and faster to load and save than doc files, easily edited by many text editing programs and even by word proc software like MSWord and OpenOffice, and, unlike MSWord's doc files, they can be viewed by anyone who has a browser. Also, html is plain text and can be searched with grep (there are greps for MSWin; one is included with NEURON's minimal cygwin installation) and managed by the same version control system that manages your code. The only time I bother with "word processor" files is if I have to include pretty looking equations or spreadsheets, or if I am writing a manuscript for publication.

It'd be great to create sections (i.e. of model code) in the document which only show the first line or two by default. The user would then click an inline button to expand the section of choice, and view the full body of code. Does any existing software offer this capability?

A standalone program that does this is head, which is part of UNIX/Linux and comes with cygwin.

Do you have any experience with digital lab notebooks?

Last time I looked (a couple of years ago), nothing relevant to my interests and activities seemed ready for prime time.

[kathryn]

Thank you, Ted! I appreciate the comprehensive reply and helpful suggestions. I'll immediately implement the file system idea.

Too bad you're stuck with MSWin. But all is not lost.

I have a dual-boot MSWin/Linux system at home, mostly for fun. The med. faculty IT staff might not be happy about a Linux box in the lab, but perhaps I should consider that option.

For "internal documentation" that requires text plus figures, html files are convenient--much more efficient of space and faster to load and save than doc files, easily edited by many text editing programs and even by word proc software like MSWord and OpenOffice, and, unlike MSWord's doc files, they can be viewed by anyone who has a browser.

Yes -- HTML immediately came to mind as the ideal format for documents with figures. Unfortunately MSWord (and similar software) generates HTML files with lots of code gunk. I've always hand-coded my HTML and CSS documents (for websites), but I might consider using an open-source WYSIWYG (or WYSIWYM) editor in this context. I'm about to have a play with Lyx, which I've recently installed.

[ted]

I have a dual-boot MSWin/Linux system at home, mostly for fun. The med. faculty IT staff might not be happy about a Linux box in the lab, but perhaps I should consider that option.

One alternative to a dual-boot configuration would be to stick with MSWin, but install cygwin--in which case, to avoid possible conflicts with the minimal cygwin that comes with NEURON's MSWin installation, you might have to compile NEURON from source code.

Yet another would be to install VirtualBox http://www.virtualbox.org/, which would allow your MSWin box to host a Linux installation--i.e. real Linux would be running inside a window on your MSWin desktop. For personal use VirtualBox is free of charge, and it's very easy to set up (IMHO easier than dealing with cygwin).

Unfortunately MSWord (and similar software) generates HTML files with lots of code gunk.

As does OpenOffice, but isn't the gunk in markup that is segregated from the substantive content (i.e. the words that you enter)? So you can grep them and do search & replace with a text editor or sed if necessary, but don't actually have to look at the contents of those html files.

[kathryn]

Yet another would be to install VirtualBox http://www.virtualbox.org/, which would allow your MSWin box to host a Linux installation--i.e. real Linux would be running inside a window on your MSWin desktop. For personal use VirtualBox is free of charge, and it's very easy to set up (IMHO easier than dealing with cygwin).

Now there's an idea! I've heard good things about VirtualBox, although I'm yet to try it.

As does OpenOffice, but isn't the gunk in markup that is segregated from the substantive content (i.e. the words that you enter)? So you can grep them and do search & replace with a text editor or sed if necessary, but don't actually have to look at the contents of those html files.

You're right. I'm just being fussy.

Thanks again for the helpful comments.

[kathryn]

It was in the back of my mind when I created this thread, though I didn't mention the issue of information integrity. That is, providing evidence for the integrity of digital content, and protecting it from manipulation.

Spoke to my supervisor today -- our university has strict requirements regarding laboratory documentation. I'll have to maintain some handwritten records, and specifically refer to accompanying digital material (whatever its form). Regular back-ups to read-only CDs will ensure I have protected copies of my work.

[ted]

Way back on September 20 I posted a message that contained this statement:

Code: Select all

NEURON's own source code is managed with subversion, and also takes advantage of Trac http://trac.edgewall.org/, a wiki that provides a very flexible interface for managing a subversion repository.

Effective immediately, the source code is now managed with Mercurial, and the root of the repository is
http://www.neuron.yale.edu/hg

For information about Mercurial itself, see http://www.selenic.com/mercurial/wiki/