How the VPN works
The VPN is extremely simple, based almost entirely on the standard HTTPS and DTLS protocols. You connect to the secure web server, authenticate using certificates and/or arbitrary web forms, and you are rewarded with a standard HTTP cookie.
You then use this cookie in an HTTP CONNECT request, and can then pass traffic over that connection. IP addresses and routing information are passed back and forth in the headers of that CONNECT request.
Since TCP over TCP is very suboptimal, the VPN also attempts to use UDP datagrams, and will only actually pass traffic over the HTTPS connection if that fails. The UDP connectivity is done using Datagram TLS, which is supported by OpenSSL.
DTLS compatibility
Note: DTLS is optional and not required for basic connectivity, as explained above.
Unfortunately, Cisco used an old version of OpenSSL for their server, which predates the official RFC and has a few differences in the implementation of DTLS.
OpenSSL
Compatibility support for their "speshul" version of the protocol is in the 0.9.8m and later releases of OpenSSL (and 1.0.0-beta2 and later).
If you are using an older version of OpenSSL, DTLS will only work if you apply this patch from OpenSSL CVS:
For versions older than 0.9.8j, some generic DTLS bug fixes are also required:- http://cvs.openssl.org/chngview?cn=17500 (OpenSSL RT#1703)
- http://cvs.openssl.org/chngview?cn=17505 (OpenSSL RT#1752)
GnuTLS
Support for Cisco's version of DTLS was included in GnuTLS in June 2012, in commit fd5ca1af which will be part of GnuTLS 3.1.
The same patch will hopefully also be applied to the GnuTLS 3.0.x release branch for 3.0.21, or it can be applied manually from here.