sYnergenz - research blog

Optimizing control signals

2006-02-22T18:42:00.000Z

After having had an initial look at fitness landscapes resulting from evolution of pulse control signals for single joint target movements, it is clear that certain optimal control criteria don't make much sense for this task. hence i'm thinking of a two-joint (elbow-shoulder) task that can be used to demonstrate the difference between muscle-models. ideally the task should involve an important dynamic aspect (no slow pointing e.g.) such as interjoint-coupling forces. underpowered weight-lifting could be an example. i'd then be looking for any signs of synergies and exploitation of muscle dynamics for solving the task...

Muscle model comparison again

2006-01-09T15:45:00.000Z

Setup: muscles symmetric, linear muscle lengths, no (i.e. fixed) moment arms. no gravity, no additional damping.

1. only a*aF:
a pulse accelerates the limb until joint limit is reached. to stop at a given point another antagonistic pulse has to be timed and scaled precisely. an external perturbation does exactly the same, i.e. accelerating the limb in the direction of perturbation. no resistance at all, i.e. no stability. even with coactivation of both muscles no stiffness, i.e. stability, can be achieved.

2. a*aF*vF;
with zero coactivation, still a pulse accelerates the limb until joint limit is reached. however, with some coactivation some stability to external perturbation is achieved. also pulses will now accelerate the limb but the movement will be damped and stops at some point (proportional to length and strength of pulse i guess, analyze...).

3. a*aF*vF + pF;
without activation the system behaves like an underdamped spring. a step activation is necessary to shift the setpoint of the spring, i.e. the equilibrium point of the limb. the higher the activations the more damped the oscillations around the EP. a pulse activation only, as an external perturbation, will after a short displacement be resisted by the "spring".

How to combine 2. and 3. ?
2. allows for triphasic pulse patterns and efficient movements. in 3. muscles have to work against each others passive resistance to stretch. control easier, as shift of equilibrium point or tri-phasic pulse-step pattern. but, oscillation around EP undesired. tuning could make the system critically damped maybe, but how realistic is that and is the property of 2. lost then? which is best for robot control, or more easily evolved?

An interesting study would evolve controllers for each setup and compare them in terms of control complexity, evolvability etc...

muscle functions

2006-01-06T21:46:00.000Z

no comparative study yet, but some observations that might be obvious but good to get down as i keep coming back to this wondering regularly:
in Inbar & Karniels work I sometimes saw pulse and other times pulse-step activations for non-linear models. now, pulse works as long as there are no passive restoring forces in the muscle (which exist in a Hill-type model but somehow are not modelled in their work). in this case however i find smooth movement both with hill-type viscosity and without. maybe it makes a difference only for VERY fast movements? with an added passive force-length relationship that resisd stretch there exists a natural equilibrium point. therefore one needs pulse step activations to move to a point different from this resting position. however, with this element added and reasonable maximum forces, the system acts like a pretty compliant spring. as a result, the sytem overshoots and oscillates around the equilibrium point specified by the step function. hmm... what is the passive damping good for then? In theory: 1) resisting external perturbation 2) keeping the muscle away from the extremes (for protection and because muscle can't efficiently produce force there); ...
question now, how to model the passive muscles? what's it supposed to be like without reflex signals and activation? like a loose spring, a stiff spring (as Inbar and Karniel seem to suggest) or not springy over most of workspace but only at extremes (shifting passive relationship)?

...meanwhile: muscle model revised

2006-01-04T16:33:00.000Z

while evolving the VITE CTRNN, i cleaned up my muscle model code and decided to do some comparative study (something i would need to do for the thesis anyway). the point of this exercise is to show the effect of the model's non-linearities. i will use three different simple input patterns (pulse, pulse-step and ramp) and compare the output of models of different complexity. this will involve reducing each of the elements of the hill-type model to linear relationships or scalar values and omitting the moment arms. i will also include a non-linear "PD muscle".

Evolution of VITE...

2006-01-01T18:49:00.000Z

... is impossible because of the neuron's linearity. for most of the parameter space the model just explodes to infinity. now, as with FLETE, i'm trying to evolve the core of the model as a ctrnn. without much success yet.

Vite and muscles

2005-10-18T20:24:00.000+01:00

one issue i haven't addressed yet is the relation between the control network's output and how it's scaled and damped to produce appropriate torques. scaling determines how fast the arm can implement the desired trajectory while damping influences oscillations and overshoot. these parameters can be tuned like a PD to produce nice, roughly critically damped, movement. the question is how much of this tuning should actually be done by the controller and not the plant? specifically how much of the damping is due to viscosity of the muscle and surrounding tissue and how much is due to reflexes?
at the moment i'm tuning the arm to produce nicely damped behavior with VITE in open-loop mode (i.e. no spindle feedback whatsoever). this makes it easier to study effects of changes in the network, because i know the behavior of the plant is simple.

VITE math vs. ctrnn

2005-09-28T17:57:00.000+01:00

for the purpose of debugging the VITE-CTRNN i plotted neural activations and kinematic variables for a simple unobstructed point-to-point movement. the following pictures compares these with the original model:

seems i'm in the right ball-park. now i have to adjust the parameters in the ctrnn (mostly default to weights of 1.0, biases of 0.0 etc.) to fit the model. might use evolution here.

VITE connectivity

2005-09-26T22:08:00.000+01:00

VITE inertial force feedback continued

2005-09-24T18:19:00.000+01:00

one problem. for fast response the desired velocity signal should be used as an efferent copy, not the signal gated by the go command. this works very well with a feedback gain of 2.0 e.g.. however, this way complete passiveness is lost with a go-command of zero (because the desired velocity vector is not zero). however, if the gated signal is used, feedback seems to slow to produce nicely damped movements. maybe it is possible to use the ungated signal along with pre-synaptic inhibition by the go-signal ...

VITE: inertial force feedback

2005-09-24T17:48:00.000+01:00

ah, i love it when a plans come together. while problems with static force feedback are not yet resolved, intertial force compensation seems to work. i implemented feedback resulting from the comparison between desired velocity and actual velocity of the muscles. due to intertia the actual velocity of a muscle/limb will always lag behind the desired value. the error can be added to the motorneurons though as positive feedback to compensate for this effect. neurobiologically this is implemented as a reafference principle consisting of velocity corollary discharge (dynamic gamma motorneuron, efferent copy) and primary spindle fibers (velocity measure, ex-afference).

VITE: static force feedback - continued

2005-09-23T15:47:00.000+01:00

actually, it only works for small static forces. for larger forces activity remains somewhere in the system where it doesn't belong. the system doesn't settle down to the desired position after the removal of the static force. something to be worked out ...

VITE: static force feedback

2005-09-23T14:28:00.000+01:00

including static force feedback (for gravity compensation e.g.) was not as trivial as i thought. the problem boils down to the interaction of three neurons: the sensory spindle neuron (S), an integrator of this sensory input (I) and the motorneuron (M) receiving compensatory input from the integrator. In this setup higher activity of the motorneuron can only reduce the error reported by the spindle. the goal is for the integrator to increase activity as long as incoming error-signals exist so as to add to motorneuron activity until the error is reduced to zero. a normal leaky integrator would stop building up activity though at a certain level at which leakage and input balance each other out. thus the arm wouldn't fully compensate for perturbations. without leakage though the activity wouldn't necessarily return to zero as the input goes to zero and overshoot results. hence, a fine balance has to be achieved of the following parameters: I's decay, I's time constant, I's transfer-function, weight I->M and weight I->I. what should be a case for optimization i tuned by hand, but still have undesirable underdamped oscillations...

2nd year report: Andy's feedback

2005-09-20T11:20:00.000+01:00

[...] the main point to be borne in mind is one of the thesis’ focus. That is, is it science or engineering? It can easily be both, but which is being addressed in which section must be borne in mind as this focuses the questions to be asked. In addition, it helps shape the literature review.

It is both, as is typical for biomimetics. Analysing and taking inspiration from organisms is the first scientific part. Here it is the study of human muscle and reflex systems. Using those ideas for building robots is the engineering bit. Analysing the resulting system and possibly making the step back to biology by explaining potential underlying mechanisms would be the final scientific step in this circular approach.

A related issue, which is also in part organisational is where the hexapod work fits into the overall story. I think this may be because of where it is placed in the thesis plan and also partly as Thomas did not detail all the experiments he has undertaken.

Experiments using the hexapod are preliminary so far. The idea is to prove that the main concept of the thesis, namely use of low-level muscle-reflex dynamics to make control as seen by higher levels easier, not only applies to human-like arm movement but to any complex movement involving many degrees of freedom. In the case of walking this could possibly take a form different from the VITE/FLETE networks. However, the idea of equilibrium-point control and coupled reflex networks still apply.

Organization: e.g. need 3 (?) solid Results chapters (each, in principle, a publication of sorts). The preliminary one is done (validation of muscle model and explorations), and the initial part of the VITE/FLETE one is done. One extra piece of work (as discussed in the report) would finish that chapter, and then the extensions discussed would be the 3rd. Alternatively, the existing VETE/FLETE bit could be a chapter and the 3rd chapter could be all the extensions. Given this, I am unclear as to where the hexapod fits into such an organisation: is it initial exploration/validation? Or is there enough (with maybe a bit of extension) for a chapter on its own? This is not a major problem but a decision should be taken as it will allow Thomas to know what work needs to be done and where best to focus his efforts.

Sounds right. 1 chapter muscle model validation, 1 chapter basic VITE/FLETE, 1 chapter Extensions. Ideally, there would be a separate chapter on the application of mentioned concepts to walking robots.

Given some such organisation, I feel there is only one more point to be borne in mind which is to have a contingency plan if the VITE/FLETE extension takes longer than expected or even doesn’t work. Again, not a major problem as there is enough work here, but it is worth thinking about the shape of the thesis given this eventuality.

In that case, I'll definitely have to make use of all 4 years. The extensions would be the major new stuff, as the muscle model and basic mathematical VITE/FLETE exist already. Contributions would mainly be in the extensions.

A final very minor issue is that the reasons/justification for using an evolutionary approach should be discussed.

1. no knowledge of what a solution to the coordination problem looks like (in terms of neural network connectivity e.g.)
2. although, typically linear, engineering solutions can be found to mathematically feasible problems, non-linear, mathematically non-feasible, solutions can be evolved that are more efficient, robust etc... (e.g. blip-PID-evolution)
...

VITE and compliance

2005-07-27T17:19:00.000+01:00

I tested the VITE circuit with feedback from spindle sensors for compliance. As it should be, when the go-signal is zero, it "passively" tracks externally imposed perturbations. With a go-signal present there is a tradeoff between tracking of imposed movement and tracking of commanded position. Inertia compensation next ...

VITE update

2005-07-26T22:00:00.000+01:00

I coupled my current neural VITE model to a physically simulated robot arm. Not surprisingly it couldn't cope with the non-linear muscle model and changing moment arms. Therefore I implemented a simpler symmetric muscle model with a square function of distance from resting position and some damping. Now it can produce smooth movements to a specified target with control over velocity. The feedback connection from perceived position to outflow position greatly reduces oscillation in the network. I haven't tested it yet with externally imposed perturbations. The next step will be to include inertia compensation as with fast movements the arm overshoots the neurally specified position. Once the VITE model is completed I will couple it to the already existing neural implementation of the FLETE model. The "final" step will be to then evolve the network using the engineered one as a starting point to also work with "real" muscles.

are complex control signals required?

2005-07-04T23:37:00.000+01:00

In VITE/FLETE sigmoidal control signals are required for sigmoidal velocity profiles. Perhaps, as in Gribble(1998), the non-linear muscle model allows for simple linear ramp signals to have the same effect.

concrete questions

2005-06-28T14:20:00.000+01:00

how can VITE/FLETE be extended for control of "real" arms, possibly exploiting non-linear muscle model properties? What effect does the use of a non-linear neural model have, and what additional elements may be necessary (e.g. joint feedback for moment arm compensation) ?
how can local per-joint VITE/FLETE networks be coupled to produce coordinated multi-joint movements?
how can error-feedback from spindles be used for lifetime adaptation of controller?

neuro-it and control of arm movements

2005-06-27T14:23:00.000+01:00

back from the neuro-it summer-school in venice. the talks by rolf pfeiffer and auke ijspeert were most interesting to me as their work is very related to what i'm doing at the moment. both emphasized the importance of decentralized mechanisms simplifying the coordinated control of limbs in animals and robots. Iijspeert, working mainly on models of lamprey/salamander locomotion, stressed the notion of CPG's and how far we can get without central planning of trajectories by studying the interaction of several CPGs. Pfeifer introduced the notion of "morphological computation", the idea that materials can take over some of the processes normally attributed to control. his examples included a tetrapod running without sensory feedback, a fisk-like robot using one degree of freedom to navigate in three dimensions, and a robot hand that grasps arbitrary objects without any control by using springy joints. Passive dynamic walking is another obvious example in this category. Now in my work I'm aiming at using the same ideas for more goal-directed movements, namely reaching. Materials are important in this case because the actuators are muscles with non-linear properties that can possibly be exploited by the controller. The controller itself is a decentralized dynamic neural network, a CPG. Coordinated movement including many joints makes necessary the right coupling of several of these local reflex-like networks. Hopefully, the right coupling between materials, controllers and environment (gravity, interia) allows for simple control signals that produce coordinated arm movements.

PID-like neurocontrollers

2005-06-16T16:19:00.000+01:00

by chance i came across this very recent publication from the animatlab: "Evolving PID-like Neurocontrollers for Non-Linear Control Problems". It nicely presents the approach I'm aiming at by evolving a neural VITE/FLETE model. They evolve a neural network that uses modules corresponding to proportional, derivative and integral terms in a PID for controlling a blimp with complex dynamics. They find that the ANN makes use of the non-linear property of neurons in order to improve performance if compared to classic linear PID control.

first VITE implementation

2005-06-10T17:33:00.000+01:00

I just created a modified CTRNN version of VITE model 1a (see below). It's bilateral and implements a reign-control version of a pId controller with voluntary control of speed. I'm gonna play around with it for a while, change non-linearities, weights etc to see how much it depends on those particular features of the network. I'll then add another interneuron to incorporate actual feedback (VITE 1b). happy for now...

science vs. engineering ?

2005-06-02T17:54:00.000+01:00

while trying to implement VITE/FLETE as a CTRNN I once more encountered an old friend of mine: the question of whether i'm doing, or want to do for that matter, science or engineering. In the current case, the question is what do i gain by fiddling around with my network until it mimics someone else's model. i figured how i can easily implement the VITE equations by adding multiplicative weights and positive linear activation functions to my network. which could actually be useful in many other cases of evolving ANNs for robotics. on the other hand it seems to fit into the category of models having too many free parameters, hence becoming kind of arbitrary. at the moment i think, as probably mentioned before, the strength lies in the fact that once the network does mimic VITE's behavior, it can easily be evolved for different robotic arms, can be extended through evolution to account for things like changing moment arms etc. In other words, it's easier to extend in many different directions. also, free parameters in the current model could easily be optimized for performance.

so, ... still the question of science or enginnering hasn't really be answered. i just hope that once my model is working it can be tested in more realistic scenarios than what's been done before. comparing it with previous results and actual human data could then reveal something of scientific interest...

VITE equations

2005-06-01T17:12:00.000+01:00

to make things complete:

so, d is the difference between target position (tp) and present position (pp); dv is the desired velocity and calculated as the difference d times the volitional signal go; op is the outflow position, integrating the desired velocity dv. pp, the present position, is estimated as an integration of commanded position op and spindle mediated error feedback s.

why VITE and FLETE?

2005-05-31T17:45:00.000+01:00

so why am i concerned that much with Bullock and Grossberg's VITE and FLETE models? Well, basically i think they could reconcile the idea of generic reflex-like low level behaviors that don't "know" about the controlled plant's dynamics, yet are able to produce stereotypic positioning movements, with the idea of centrally planned motor programmes (schemes) that are highly specialized for different tasks in different contexts. What's more, i hope to implement the models using generic differential equations, namely continuous time recurrent neural networks. This would have the advantage of allowing for easy evolution of the models to cope with different types of controlled plants (in the original models, all equations are set up by hand). it also makes adding new functionality, such as compensating for changing moment arms, easier.

FLETE

2005-05-31T17:15:00.000+01:00

the FLETE model, Factorization of LEngth and TEnsion, is a neural implementation of spinal circuitry responsible for transforming the desired trajectory produced by VITE into actual muscle commands. It's main purpose, according to Bullock and Grossberg, is to allow for independent control of angle and stiffness (which is problematic for several reasons, e.g. size-principle in motor units).

The type of neurons and their connectivity is modelled closely to actual spinal circuitry.

VITE 3: static and inertial loads

2005-05-30T17:13:00.000+01:00

In addition to coping with obstacles (position constraints), any real system has to be able to overcome static loads (gravity) and inertial effects. In VITE the former is achieved by simply integrating the spindle error and adding that signal to the outflow command (integral term in PID).

Inertia in a controlled system means that velocity will be smaller than the commanded movement at initiation, and bigger near the end. In VITE this is compensated for by adding an outflowing desired velocity command and corresponding velocity error (dynamic gammaneurons and primary spindle afferent). Hence, perceived velocity error can add a launching pulse in the beginning and a breaking pulse in the end.

VITE 2: compliance

2005-05-30T16:49:00.000+01:00

In the first step of VITE's construction one feature hasn't been mentioned: the feedback connection between perceived and commanded position. this is used to allow for compliance in the case of obstacles. it allows for the commanded position to track the sensed position if no volitional signal is present. hence, perturbations are not being opposed. a more precise neural implementation looks like the left picture.

An additional interneuron calculates the difference between perceived and commanded position, and feeds that error signal into the latter. For voluntary movement in the presence of an obstacle a trade-off results between tracking of external force and integration of desired position.

VITE 1: position tracking

2005-05-30T16:23:00.000+01:00

I'm gonna iterate through the construction of the Vector-Integration-To-Endpoint model as in Bullock et al.(1995). First step: variable speed movements to a target position can be achieved by multiplying the difference between a target position and present position with a volitional signal. In an ideal world, present position can be calculated by integrating the difference(=desired velocity). See first picture.

The basic idea is to use error integration for target tracking. However, if obstacles are taken into account, commanded and actual position may differ. Therefore, in order to allow for accurate estimation of current position, sensory feedback is used. In VITE, present position is calculated from an outflow command and a position error signal originating in muscle spindles (through use of ag-coactivation).

problem of abstraction: muscles and moment arms

2005-05-29T20:41:00.000+01:00

current problem: i've evolved a neural network that factors (independently controls) the difference and the sum of its two output neurons (non-trivial with sigmoidal activation functions). translating these outputs into muscle activation means it can independently control the angle and the stiffness of the limb(joint) whose movement it controls. a very desirable property. IF, that is, one assumes constant/symmetric moment arms and equal force-length tension curves and resting angles for both muscles. this is the case e.g. in bullock and grossberg's work (the FLETE model). it is not the case in more realistic setup's, like my simulation. so, ...
to account for changes in moment arm and different resting lengths, the network has to regulate depending on current joint angle. consequently i'm gonna try adding extra neurons and joint feedback to the current network. if this is enough to make it work, it would suggest a new role for joint feedback. namely modulating the angle-stiffnes factorization to compensate for changing moment arms. so far i haven't come across this idea anywhere ...

EP-theory continued

2005-05-29T19:30:00.000+01:00

... in other words: the spinal circuit ensures the existence of an equilibrium point to which the controlled limb converges. higher centers can modulate the circuit and influence details of the trajectory. for example to move to the same point faster, the equilibrium point itself doesn't have to be changed. instead, a little "push" in the beginning, and a little breaking in the end (or observed tri-phasic patterns) would achieve the desired speed-up. similarly, another mechanism could modulate the spinal circuit to account for gravity.

EP-theory and beyond

2005-05-29T18:57:00.000+01:00

conceptually: evolved spinal "reflex" circuits can implement flexible PID-style controllers. they could be seen as neural servos. coupled with the musculoskeletal system they create a controllable equilibrium point in workspace. this would be open to usual critique against EP-theory, but the story doesn't end here. basically, spinal circuits could solve the kinematic problem independent of the current context. higher centers then modulate the circuit to account for different situations and optimize/solve for dynamics (inertia, limb coupling ...)