COROSYNC_OVERVIEW(Corosync Cluster Engine Programmer's ManCOROSYNC_OVERVIEW(8)NAMEcorosync_overview - Corosync overview
OVERVIEW
The corosync project is a project to implement a production quality
"Revised BSD" licensed implementation of the most recent SA Forum's
Application Interface Specification. The Application Interface Speci‐
fication is a software API and policies which are used to develop
applications that maintain service during faults. The API consists of
Availability Management Framework (AMF) which provides application
failover, Cluster Membership (CLM), Checkpointing (CKPT), Eventing
(EVT), Messaging (MSG), and Distributed Locking (DLOCK).
Currently Messaging is unimplemented.
Faults occur for various reasons:
* Application Faults
* Middleware Faults
* Operating System Faults
* Hardware Faults
The major focus of high availability in the past has been to mask hard‐
ware faults. Faults in other components of the system have gone
unsolved until AIS. AIS can mask many types of faults in applications,
middleware, operating systems, or even hardware by providing a simple
framework for allowing developers to create redundant applications.
These redundant applications can be distributed over multiple nodes
such that if any one node faults, another node can recover.
Application programmers develop applications to periodically record
their state using the checkpointing service. When an active application
fails, a standby application recovers the state of the application.
This technique, called stateful application failover, provides the fun‐
damental difference between corosync and other systems that have come
before it. With stateful application failover, the end-application
user doesn't have to reload the application or redial a telephone. The
full state is recorded, so the end-application user sees no interrup‐
tion in service.
Because programmers can now distribute applications across multiple
processes or nodes, a mechanism must exist for them to communicate.
This mechanism is provided by two services. The event service provides
a publish/subscribe model for events. The messaging service provides
end to end messaging. Finally a mechanism to synchronize access is pro‐
vided by the distributed lock service.
The corosync project also provides a group messaging toolkit called
EVS. The EVS service implements a messaging model known as Extended
Virtual Synchrony. This model allows one sender to transmit to many
receivers. Certain guarantees are provided for message and membership
delivery which make virtual synchrony ideal for developing distributed
applications.
QUICKSTART
The corosync executive must be configured. In the directory conf in
the source distribution are several files that must be copied to the
/etc/corosync directory. If corosync is packaged by a distro, this may
be complete.
The directory contains the file corosync.conf. Please read the
corosync.conf(5) man page for details on the configuration options.
The corosync project will work out of the box with the default configu‐
ration options, although the administrator may desire different
options.
The corosync executive uses cryptographic techniques to ensure authen‐
ticity and privacy of the messages. In order for corosync to be secure
and operate, a private key must be generated and shared to all proces‐
sors.
First generate the key on one of the nodes:
unix# corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /etc/corosync/authkey.
After this operation, a private key will be in the file
/etc/corosync/authkey. This private key must be copied to every pro‐
cessor in the cluster. If the private key isn't the same for every
node, those nodes with nonmatching private keys will not be able to
join the same configuration.
Copy the key to some security transportable storage or use ssh to
transmit the key from node to node. Then install the key with the com‐
mand:
unix#: install -D --group=0 --owner=0 --mode=0400
/path_to_authkey/authkey /etc/corosync/authkey
If a message "Invalid digest" appears from the corosync executive, the
keys are not consistent between processors.
Finally run the corosync executive. If corosync is packaged from a
distro, it may be set to start on system start. It may also be turned
off by default in which case the init script for corosync must be
enabled.
After running aisexec, a list of all processors IP addresses running
the corosync executive and configured on the same multicast address
will appear. If they don't appear, there may be a problem with multi‐
cast in the distro or hardware. If this happens, participation in the
corosync mailing list may help solve the problem. The email address is
openais@lists.osdl.org.
USING LIBRARIES
The corosync AIS libraries have header files which must be included in
the developer's application. Once the header file is included, the
developer can reference the AIS interfaces.
The corosync project recommends to distros to place include files in
/usr/include/corosync. The following include lines must be added to
the application to use each of the following services:
#include <corosync/saClm.h> For the Cluster Membership B.01.01 service.
#include <corosync/saCkpt.h> For the Checkpointing B.01.01 service.
#include <corosync/saEvt.h> For the Eventing B.01.01 service.
#include <corosync/ais_amf.h> For the AMF A.01.01 service.
The corosync project recommends to distros to place library files in
/usr/lib. The following link lines must be added to the LDFLAGS sec‐
tion of the makefile.
-lsaClm For the Cluster Membership B.01.01 service
-lsaCkpt For the Checkpointing B.01.01 service
-lsaEvt For the Eventing B.01.01 service
-lsaAmf For the AMF A.01.01 service
-lais Specify this to get access to all AIS libraries without specify‐
ing each library individually.
IPv6
The corosync project supports both IPv4 and IPv6 network addresses.
The entire cluster must use either IPv4 or IPv6 for the cluster commu‐
nication mechanism. In order to use IPv6, IPv6 addresses must be spec‐
ified in the bindnetaddr and mcastaddr fields in the configuration
file. The nodeid field must also be set.
An example of this is: nodeid: 2 bindnetaddr: fec0::1:a800:4ff:fe00:20
mcastaddr: ff05::1
To configure a host for IPv6, use the ifconfig program to add inter‐
faces: box20: ifconfig eth0 add fec0::1:a800:4ff:fe00:20/64 box30:
ifconfig eth0 add fec0::1:a800:4ff:fe00:30/64
If the /64 is not specified, a route for the IPv6 network will not be
configured which will cause significant problems. Make sure a route is
available for IPv6 traffic.
ARCHITECTURE
The AIS libraries are a thin IPC interface to the corosync executive.
The corosync executive provides services for the SA Forum AIS libraries
as well as the EVS and CPG libraries.
The corosync executive uses the Totem extended virtual synchrony proto‐
col. The advantage to the end user is excellent performance character‐
istics and a proven protocol with excellent reliability. This protocol
connects the processors in a configuration together so they may commu‐
nicate.
ENVIRONMENT VARIABLES
The corosync executive process uses four environment variables during
startup. If these environment variables are not set, defaults will be
used.
COROSYNC_MAIN_CONFIG_FILE
This specifies the fully qualified path to the corosync configu‐
ration file.
The default is /etc/corosync/corosync.conf.
COROSYNC_AMF_CONFIG_FILE
This specifies the fully qualified path to the corosync Avail‐
ability Management Framework configuration file.
The default is /etc/corosync/amf.conf.
COROSYNC_DEFAULT_CONFIG_IFACE
This specifies the LCRSO that is used to parse the configuration
file. This allows other configuration file parsers to be imple‐
mented within the system.
The default is to use the default corosync configuration file
parser which parses the format specified in corosync.conf (5).
COROSYNC_TOTEM_AUTHKEY_FILE
This specifies the fully qualified path to the shared key used
to authenticate and encrypt data used within the Totem protocol.
The default is /etc/corosync/authkey.
SECURITY
The corosync executive optionally encrypts all messages sent over the
network using the SOBER-128 stream cipher. The corosync executive uses
HMAC and SHA1 to authenticate all messages. The corosync executive
library uses SOBER-128 as a pseudo random number generator. The EVS
library feeds the PRNG using the /dev/random Linux device.
If membership messages can be captured by intruders, it is possible to
execute a denial of service attack on the cluster. In this scenario,
the cluster is likely already compromised and a DOS attack is the least
of the administration's worries.
The security in corosync does not offer perfect forward secrecy because
the keys are reused. It may be possible for an intruder by capturing
packets in an automated fashion to determine the shared key. No such
automated attack has been published as of yet. In this scenario, the
cluster is likely already compromised to allow the long-term capture of
transmitted data.
For security reasons, the corosync executive binary should NEVER be
setuid or setgid in the filesystem.
SAFTEST COMPLIANCE
The corosync libraries are now nearly compliant with every aspect of
the SA Forum's AIS specification. The AMF service, however, is not
compliant with the B.01.01 specification. The remaining services pass
most of the tests of the saftest suite against the B.01.01 specifica‐
tion.
BUGS
The messaging service is partially implemented and not suitable for
deployment. The distributed locking service is buggy and not suitable
for deployment. The Availability Management Framework is under devel‐
opment and not suitable for deployment..
SEE ALSOcorosync.conf(5), corosync-keygen(8), evs_overview(8)corosync Man Page 2006-05-10 COROSYNC_OVERVIEW(8)
