I am officially closing this blog down. I’ll actually be leaving it up, but I will not be making any more updates. All research related updates will be made on my personal page  here.  Later.

I’ve been struggling a bit to find some nice easy models and experiments to use to build up some momentum.  I was going to do another model for a contingency detector similar to the one here except that I was going to use DFT.  I may still do that, however, I had another idea that I had to explore immediately.

I’ve been thinking about how to show that many different learning algorithms can be implemented using DFT.  Some of the easier ones like habituation and sensitization are obvious.  There has already been some work showing that simple associative learning can be done.  So I got to thinking about reinforcement learning and how to implement that.  I managed to come up with the model itself and code it up in Matlab.

In a nutshell, the model consists of three DFT layers.  One is an excitatory layer, one is an inhibitory layer, and the other is a memory trace layer.  Now the excitatory layer is connected to itself with an excitatory connection and to the memory trace with an excitatory connection.  The inhibitory layer is connected to itself with an excitatory connection and to the memory trace with an inhibitory connection.  A stimulus is provided to both the excitatory and inhibitory layers and they lay down a memory trace based on the relative strength between them.  Currently I am adjusting their relative strengths by providing a homogenous boost to the resting level of both the excitatory layer and inhibitory layer.  This boost is equal in magnitude but opposite in sign i.e.  when the excitatory layer has been positively boosted (the resting level is closer to zero) the inhibitory layer is negatively boosted and vice versa.  The boosted resting level is the reward signal.

Here is a video of the model evolving in time.

Reinforcement Learning with DFT from Jesse Lane on Vimeo

The top, cyan colored graph is the stimulus signal.  You can see that it is instantaneously switched on and off.  There are two trials that last 250 time steps each.  The stimulus starts at time step 50 and ends at 150.  The second, green colored graph is the excitatory layer.  In the first trial, notice that the resting level is set to -5 and in the second trial it is set to -10.  The third, blue colored graph is the memory trace.  Finally, the fourth, red colored graph is the inhibitory layer.

During the first trial, the resting level of the excitatory layer is greater than the resting level of the inhibitory layer so when the activation of the excitatory layer increases above threshold (here threshold is zero indicated by the black axis) it induces an excitatory response in the memory trace.  Conversely, in the second trial the resting level of the inhibitory layer is now greater than the resting level of the excitatory layer and when it increases above threshold an inhibitory response is induced in the memory trace.  It is important to remember that the graphs depict the activations of each layer and that there is no adequate visualization of the connections between layers.  Therefore, in the second trial when the activation of the inhibitory layer is high the activation of the memory trace decreases.

Here are some images of the plots of the time course of the simulation.

As the plots above show, when the reward signal (homogeneous boost of the resting level) is positive the memory trace activation increases and vice versa.  When the reward signal is low the memory trace activation decreases.  When this memory trace field is used as an input or a modulatory component to another field the activation will be preshaped by the memory trace so that activations coinciding with positive memory traces will be increased and activations coinciding with negative memory traces will be decreased.

The true power of this model, that I hope to illustrate in future simulations, is that the reward signal can come from a homeostatic variable (I’m thinking a single neuron would do) that can be defined for particular embodiments.  In fact, the reward signal could be the combination of multiple homeostatic variables.  My next step is to scale this model and the homeostatic notion up.  I plan on creating a spatial representation for a high degree-of-freedom robotic mechanism and then trying out a heierarchical model of homeostatic rewards.

Imagine that the motor temperature of a joint is a homeostatic variable.  Once the temperature increases above a dangerous threshold, then the signal provided by a sensor that measures the temperature of a joint motor decreases the probability of another motion including the affected motor by reducing the activation of the memory trace.  If the homeostatic variable were connected to every field in a robotic control system with a small interaction parameter, then the global behavior being exhibited when the “punishment” was applied would be slightly decreased in its probability of remembered.  Each subsequent combination of fields that included the “painful” joint would have an increasing interaction with the “pain” response.  At the actual joint we could have a representation where the excitatory and inhibitory layers and their resting levels are only modulated by the “pain” signal from the joint itself.  Then we can create a hierarchical model that takes each “level” of field into account and assigns credit to this field.  I won’t go as far as to claim that I’ve found an answer to the Credit Assignment Problem, but it may be possible that this model gains some traction on the problem that has not been explored before.

Finally, it is also possible to reduce the computational load, at the possible expense of mathematical and biological elegance, by changing the homogeneous boost to the resting level into a weighting constant on the interaction kernel between one field and the memory trace.  In this case you could have the weight track the activation of the homeostatic variable such that the kernel has the usual excitatory influence on the memory trace when the variable is positive and the inverse, inhibitory influence on the memory trace when the homeostatic variable is negative.  This would allow for a reinforcement learning model consisting of a perceptive layer, a memory trace layer, and a single reward neuron.

So…I think that my last post was sort of a red herring.  I’m good at that.  Basically, what I wrote about, at length, is a potential problem that I could look at some time in the future.  However, the important thing now is to get something done.  In the interests of getting things done, here’s the plan.  I’m just going to concentrate on 2D fields of the same size.  I’ve chosen CUDA and the CUBLAS for the linear algebra.  At some point I want to add MPI so that I can get it to run on clusters.  First, I need some simple experiments to run so I can start churning out the publications.  Any ideas will be well received.

Much of this new motivation comes from a conversation I had with Alex Stoytchev yesterday.  The talk consisted of Alex asking me questions, me giving him lame answers followed by Alex looking disappointed.  He basically told me that I was a lazy ass and had better get on it or there was basically no point in me trying to get my degree.  All of these were great points.  Thanks Alex!  Now I am bound and determined to get myself moving and act like a super star again.

While I’ve been trying to decide on the languages and libraries to use in implementing my DFT framework, other implementation issues have been vying for my attention.

This issue actually combines implementation and theory.  I mentioned in a previous post that the equation

has a host of drawbacks, but I did not enumerate them. I won’t give an exhaustive list here either.  However, a major problem I am dealing with right now is that all of the fields in the equation above are pinned together by the position vector .  This means that each field must have the same dimensionality and is sampled at the exact same points and thus, the same resolution.

The issue of dimensionality is not terribly important theoretically because one can choose the dimensionality of the system to be the maximum required dimensionality of any one field.  Fields which do not require as many dimensions as are available in the system can simply set extra dimensional components to zero.  When implementing the system, however, we would like to reduce the amount of computation as much as possible.  Since the number of elements in a field is exponential in the dimensionality of the field, reducing the dimensionality reduces both the time and space complexity of computations on the field.  From an implementation standpoint it would be silly to model a simple univariate system such as a single neuron using 10 dimensions.  Therefore, it is important to create and use fields that have the right number of dimensions for the task.  Later I hope to attack how the right number of dimensions for a task is decided.

The other issue mentioned was that each field is sampled at the same resolution, indeed at the exact same points.  The sampling resolution also has a large impact on the computational complexity of the implementation, especially after taking the dimensionality of the fields into consideration.  Each task should use the minimum resolution needed to accurately perform its task without suffering from undersampling and the problems that arise from it.  A bistable switch does not need the resolution of a retina.

The two issues so far encountered reduce to the problem of mapping a field to another field where each field possibly has a different number of dimensions and different resolutions while preserving the dynamics between the two.  A side issue that needs to be mentioned is that the original dynamic field equations were formulated as infinite continuous fields.  In practice, bounded fields are used which can introduce boundary effects.  These effects can sometimes be ignored or treated.  The important fact is that since these fields are embedded in for some and mappings can be found such that , we can use unit hypercubes to begin with when deciding on these mappings between fields so that the actual boundaries of the fields are not a significant issue.

If we consider two fields to be the same “size”, i.e., a unit hypercube, the dimensions are the same, and their resolutions differ, we could simply interpolate between the components of one field making it the same resolution as the other and continuing.  We still need to decide on an interpolation method and each method has further computational tradeoffs.  The more difficult issue is when the fields have different dimensionalities.  There are some examples in the literature of models that combine fields having different dimensionalities that I should review, although I do not remember any general solutions suitable for a framework.

Another option, and the one that I will probably, initially pursue, is to just choose arbitrary dimensionality and resolution and try to make due.  There is much evidence for the brain using essentially 2D cortical sheets and so two dimensions could be a nice starting point.  0D and 1D stimulus signals can use population codings to map them to a 2D field.  Appropriate choices of resolution can be determined empirically or perhaps by using an adaptive process to optimally adjust the resolution.  Choosing this method makes the assumption that we can get by using heterarchies of 2D fields to approximate higher-dimensional fields.  This works well for separable dimensions, not so sure otherwise.

This post is long enough.  STOP

Every year the HCI department puts on a little shindig to show off the research, etc., that goes on there.  It used to be the HCI forum and then it was up-sized to the Emerging Technologies Conference.  At this conference the students are required to put up a poster on their research and stand by it to answer questions for an hour.  Here is my poster for this year.  Click on it to check it out in all of its glory.

Lately I’ve been searching for software libraries to build my dynamic field framework.  This process has been very frustrating.  First let me introduce my full dynamic field model and then go through the library issues.

The dynamic field model that I will probably be using is essentially the Amari-type model but expanded to include multiple fields.  The equation for the dynamics then becomes

.

Where

• position vector where the dimensionality defines the dimensionalities of the dynamic fields.
• vector field where each component of the vector at position is the activation of a field.
• diagonal matrix of time constants.
• matrix of field connections where diagonal components are “self” connections and off-diagonal components are “inter” connections.
• output function (typically a sigmoid or step function).
• vector field of external inputs.
• diagonal matrix of resting potentials.

This particular model has some drawbacks that I may go over in the future.  Regardless, the final form will look something like that above and will use the same operations which leads us to the software I’m searching for.

I need a mathematics library that has the following features

1. Data structures that can represent vectors, matrices, and vector fields (tensors).
2. Operations on these structures such as addition/subtraction, multiplication, and convolution.
3. Fast.
4. Prefer that it work under Mono/Linux so I can use C#.
5. Prefer to have a parallel implementation too.

So far, the biggest barrier to finding a library that I like is feature #4.  If I were to drop my language preference, then there are many libraries written in C/C++, Fortran, Python, etc.  I really don’t want to do that because I really like C#, I hate C++, I’ve never used Fortran, and I’m not very familiar with Python.  I guess if I had too I could go back to C++

So far the best candidates are ILNumerics, dnAnalytics, and Math.NET.  I like ILNumerics best at the moment, but they are concentrating on Windows .NET development right now.  This means that it hasn’t been tested much on Mono and that the downloads are all for Windows.  That means I’d have to build mine from source including all of the dependencies.  Both dnAnalytics and ILNumerics are able to use BLAS and LAPACK to speed up calculations.  Unfortunately, dnAnalytics defines their matrices explicitly and I didn’t get the impression that you could easily create vector fields (tensors).  Math.NET is written completely in C# (and F#?) so I’m skeptical about its speed vs. the other two (although I haven’t tried it) and it’s data types are similar to dnAnalytics.  Finally, all three of them are definitely “works in progress” so it is hard to gauge their potential lifetimes and available support.

If anyone who reads this can point me to a library or libraries that fulfill my requirements I would be very grateful!

I just got done reading a metric crap-tonne of research papers to get up-to-speed on the theoretical underpinnings of the dynamic models I am using in my research.  I am a little disappointed.  I’m not disappointed in the papers themselves, they are good papers for the most part and I learned a lot from them.  My problem is that since the mathematics for nonlinear systems is so difficult, many simplifying assumptions have to be made to make the mathematics tractable.  I appreciate this from the side of making theoretical progress toward understanding the behavior and limitations of these models.  However, from my research perspective, I plan to use these models to create robot control systems–essentially robot brains or parts of brains–so many of the simplifications make the results from these papers less useful for the actual construction of robot control systems (other than pointing out features that may be of interest).

Before continuing let me describe a simple model so that my detailed complaints make more sense.  I am currently using an Amari-type continuous neural field model (which, confusingly, goes under many names such as continous field, continuous neural field with lateral-inhibition, continous neural network, and dynamic field) for brevity lets call it the ACNF. A typical, simple equation for the dynamics of an ACNF is as follows.

Where u is the “activation” of the field, tau is a time constant, the function w is a connection or weighting function, f is a gain or output function, h is the resting activation, and s is an external input or stimulus function.  w typically of “Mexican hat” type like a Difference of Gaussians (DOG) and f is usually the Heaviside step function or sometimes a sigmoid.  Now we can move further into my particular set of issues.

The typical simplifications to this model are that f is a Heaviside step function which can only take on the values of 0 and 1 which makes analysis simpler and analyzing the solutions to the dynamics equation when s is either 0 or a vastly simplified input.  The first simplification mentioned is not expected to be  a very big deal in my control systems because there is evidence that the dynamics should be mostly the same in either case.  However, there is also evidence that some features of the solutions do not emerge when using a step function vs. a sigmoid such as certain stable n-bump solutions.  The second simplification is a much bigger deal because the input from the robots sensors will certainly not be a constant zero and will not necessarily conform to a simplified shape (think image sensors).

Another issue that I have is that the solutions that research up to this point has focused on (again to make the math tractable) are relatively simple shapes such as “bumps”, “rings”, “waves”, “pulses”, etc.  There is evidence that these forms of activation exist in the brain.  There have also been many useful psychological models of attention, spatial working memory, and robotic navigation where these shapes (mostly “bumps”) have been very successfully employed, in fact, this success is why I chose to work with ACNFs.  However, since I plan on working in a purely robotic domain, taking inspiration from nature, not trying to duplicate it, I have a bit more freedom in using the models.  Rather than matching the results of psychological experiments or neuroscientific data, I am taking an engineering approach where whatever works is what’s important.  Therefore, I would like to see what aspects of ACNFs a control system is likely to take advantage of instead of making that assumption beforehand.

My original, very rough, outline for my dissertation included three parts in this order: a theoretical section, a technical section, and an experimental section.  First, in the theoretical section I would attack some interesting and open theoretical problems in ACNFs that relate to the robot control systems.  Second, in the technical section I would use some of the theoretical results to create software for numerical solution of ACNFs including any new results from the theoretical section.  Finally, in the experimental section I would use the software to run simulations and control actual robots.

I have decided to reconsider due to the disillusionment discussed above.  Now I plan to change the order a bit and the nature of the work within some sections.  The new order will be: technical, experimental, then theoretical (probably how I should have approached it in the first place).  The technical section will be the same, the experimental section will be mostly the same but more exploratory, and the theoretical section will try to analyze the results of what is found in the experimental section and apply them to suitable open problems if possible.

Since my last post I’ve finally finished reading enough papers to feel up-to-date on the current state-of-the-art  in Amari-type continuous neural fields.  Here are the papers I’ve read since the last post.

• Waves and bumps in neuronal networks with axo-dendritic synaptic interactions by S. Coombes, G.J. Lord, and M.R. Owen
• Dynamic Approximation of Spatiotemporal Receptive Fields in Nonlinear Neural Field Models by Thomas Wennekers
• The Wave Packet:  An Action Potential for the 21st Century by Walter J. Freeman
• Two-bump solutions of Amari-type models of neuronal pattern formation by Carlo R. Lang and William C. Troy
• Waves, bumps, and patterns in neural field theories by S. Coombes  (A good review of techniques and models)
• Dynamic neural field with local inhibition by Nicolas P. Rougier
• Dynamics, Stability and Bifurcation Phenomena in the Nonlocal Model of Cortical Activity by Konstantin Doubrovinski
• Wave-front solution behaviour for continuous neural networks with lateral inhibition by Sittipong Ruktamatakul, Jonathan Bell, and Yongwimon Lenbury
• Bumps and rings in a two-dimensional neural field:  splitting and rotational instabilities by M.R. Owen, C.R. Laing, and S. Coombes
• Hardware/Software Codesign for Embedded Implementation of Neural Networks by Cesar Torres-Huitzil, Bernard Girau, and Adrien Gauffriau (decent paper, not as helpful as I had hoped)

Apparently it’s been a long time since I made a post here.  Haven’t done very much.  Back in the saddle now though.

Today I read:

• Dynamics and Formation of Self-Organizing Maps by Jun Zhang
• Circular stationary solutions in two-dimensional neural fields by Herrad Werner and Tim Richter
• Multiple Bumps in a Neuronal Model of Working Memory by Carlo R. Laing, William C. Troy, Boris Gutkin, and G. Bard Ermentrout

I have nine more papers to read until I’m current with the theoretical work in this field.  I’d better pick up the pace.  My task after that will be to make a rough thesis proposal/research outline (I promise John).

I was sick a couple of days last week, thus the lack of entries.  Slow day today, had some other stuff going on.  I read Formation of Topographic Maps and Columnar Microstructures in Nerve Fields by A. Takeuchi and S. Amari.  There was some good stuff in there about the equations used to induce self-organization in the nerve fields.