[ODE] deterministic repeatability issues (cont'd) (i.e. resetting / restarting)

[Doron Tal]

Dear fellow ODE coders,

We need absolute perfect repeatability of each ODE simulation we run,
for teaching a simulated robot how to walk.  But I had unbelievable
troubles getting repeatability.  I have a server sitting running
simulations whenever it receives a request (over sockets) with the
next simulation's parameters.  The server does an ODE RESET before
running each simulation, as follows, in this exact order:

- dBodyDestroy                           (for all bodies)
- dGeomDestroy                           (for all geoms)
- dJointDestroy                          (for all joints)
- dJointGroupEmpty                       (for all joint groups)
- dJointGroupDestroy                     (for all joint groups)
- dSpaceDestroy                          (for all spaces)
- dWorldDestroy       

- dWorldCreate
- dHashSpaceCreate
- dJointGroupCreate
- dBodyCreate                            (all bodies)
- [ set up body same way every time ]
- dCreateBox                             (all geoms)
- [ set up geom same way every time ]
- dJointCreateX                          (all joints)
- [ set up joint same way every time ]

The non-repeatability shows up as follows:

left column     right column
-----------     ------------
-0.031869  ==?  -0.031869
-0.066889  ==?  -0.069010
-0.127048  ==?  -0.125755
-0.024693  ==?  -0.024693
-0.045233  ==?  -0.045233
-0.031346  ==?  -0.030807
-0.066436  ==?  -0.070823
-0.103483  ==?  -0.103483
-0.079873  ==?  -0.079873
-0.021944  ==?  -0.021944

Each number is a "score" obtained by a single simulation run.  Each
simulation run is a function of some parameters.  Each ROW above
corresponds to a different PARAMETER SET.  The left column shows the
simulation scores obtained as we go through a search algorithm and try
many different other parameter sets in between each paramter choice
(i.e.  many rows have been ommitted -- the simulator ran many times
between each row).

The only difference in obtaining right vs. left columns is that on the
left we performed many extra parameter-set evaluations of other
simulation runs in between the scores displayed, whereas we did the
entire left column in the order it appears (top to bottom).

Notice some of these entries are slightly *different*!

I tried following every suggestion in the ODE Wiki web page on this
topic ( http://ode.org/cgi-bin/wiki.pl?SaveAndRestoreHowTo ),
including using a dBodySetQuaternion() that doesn't normalize for
remembering the original state quaternions.  There was no need to take
care of geom orderings as suggested, because all geoms are destroyed
and then recreated on each reset.  I counted every call to ODE's
create/destroy functions and found that they match perfectly, i.e. one
does not occurr without the other.  The simulations themselves seem to
run nicely with no problems for hours on end, they are just not
perfectly repeatable.

dRandSetSeed() is called at the start of each simulation (but it
probably doesn't matter since dWorldStep() is the method for stepping
the world).

Using linux, debug, single-precision build.

This one number difference is just amazing: the code is totally
useless for AI type thins without ensuring full repeatability.

Any help or suggestions would be greatly appreciated, and if anything
interesting comes up on this, I will post it and try to append to the
Save/Restore page.

Best Regards,

-Doron