Reinforcement Learning with NetPyNE: a closed-loop with external stimulation

A Python package that facilitates development and use of models of biological neural networks

Moderator: tom_morse

Post Reply
ziemek
Posts: 45
Joined: Thu May 23, 2019 8:02 am
Location: Warsaw, Poland
Contact:

Reinforcement Learning with NetPyNE: a closed-loop with external stimulation

Post by ziemek »

Hi there,

After a great tutorial of NetPyNE done by Salvador Dura I wanted to extend my question from the chat :)

I would like to make a simulation where I will stimulate a network in a closed-loop with external stimulation similar to Reinforcement Learning paradigm.

So let's say I have:
  • The network called the Agent
  • The Agent is implemented in NetPyNE
  • I would like to to simulate the network, on demand in 2 ways: 1) as a regular run for n ms; 2) as the observation run for n ms, during which the Agent receives an external stimulation
  • I would like to collect spikes from the output neurons

eg. pseudocode:

Code: Select all

# input (10,10) matrix which means 100 input cells, output -> 2 cells
agent = NetPyNEAgent(input_shape=(10,10), output_shape=2)

while True:
    obs = environment.get_observation()
    
    # make a external stimulation and run simulation for 100 ms
    agent.observe(observation=obs, time=100)
    
    # run the agent just for 100 ms without stimulation
    agent.run(time=100)

    # number of output spike 
    spikes = agent.get_output_spikes()
    environment.make_action(spikes)
I have 2 problems here:
  • The method observe() is the most critical one, since I don't know how to make an external stimulation in NetPyNE
  • I have seen only NetPyNE examples where simulation were entirely defined and run without interruption. I don't know how to make run(), stim() and another run()
Any simple code example would be sufficient for me to just get started.
salvadord
Posts: 86
Joined: Tue Aug 18, 2015 3:49 pm

Re: Reinforcement Learning with NetPyNE: a closed-loop with external stimulation

Post by salvadord »

Hi Ziemowit,

Thanks for your email and interest in the tool! Yes, everything you mention is possible in netpyne and we actually have a closely related example, where we control a simple virtual arm using RL and STDP: https://github.com/Neurosim-lab/netpyne ... les/RL_arm

Please have a look at the example and if you have any particular questions feel free to post them in the forum (or in this other forum www.netpyne.org/forum), or we could also have a quick videochat

Thanks
Salva
Post Reply