BBFTP(1) User Contributed Perl Documentation BBFTP(1)NAMEbbftp - transfer files using compression and parallel streams
SYNOPSISbbftp-v
bbftp [Options] [-u RemoteUsername] -i ControlFile RemoteHost
bbftp [Options] [-u RemoteUsername] -e ControlCommands RemoteHost
DESCRIPTION
Use the bbftp command to transfer files between the local host and a
remote host. In order to get better performance on a loaded Wide Area
Network than transferring with the standard ftp command, use the bbftp
command. bbftp has been designed to take advantage of the RFC 1323 and
uses multiple streams is order to speed up transfer. It also implements
an automatic retry in case of failure of commands contained in the Con‐
trolFile or in the ControlCommands.
bbftp allows two methods of connection to the remote host, a direct
connection on the control port of the remote bbftpd daemon or the abil‐
ity to remotely start a bbftpd daemon through an ssh tunnel. For the
first method, the bbftpd daemon has to be started (by inetd or as a
standalone daemon) on the remote host and all user information (user‐
name, password) are transmitted encrypted to the daemon; for the second
the bbftpd binary has to be accessible somewhere on the remote host and
all control data are transmitted through the ssh tunnel.
A third additionnal authenticate mode allows to use certificates to log
on. This mode is based on the Grid Security Infrastructure and
requires Globus software to be installed. The client side needs a cer‐
tificate to identify itself and the daemon needs a host certificate.
Tshe behaviour of bbftp is controled by the ControlFile which contains
the commands to be executed (option -i) or by commands separated by
semicolons (option -e). The format of these commands are given in CON‐
TROL COMMANDS.
bbftp may be used in one of the following ways:
% bbftp
to print a short help
% bbftp-v
to write the version of the software and default values to standard
output.
% bbftp [Options] -i ControlFile [-u RemoteUsername] RemoteHost
% bbftp [Options] -e ControlCommands [-u RemoteUsername] RemoteHost
to request the execution of commands contained in the control file Con‐
trolFile or the ControlCommands using RemoteUsername on RemoteHost.
Depending on the Options a password for the RemoteUsername on Remote‐
Host may be requested interactively (see OPTIONS section, for a full
description of the Options).
The RemoteUsername value is ignored in the certificate authentication
mode. The remote user is automatically detected by the daemon using
the Globus gridmap file.
OPTIONS
Options can be separated into two classes, those describing the way
bbftp connects to the bbftpd daemon and those modifying the behaviour
of the control commands.
CONNECTION OPTIONS
-s Use this option to use ssh to remotely start a bbftpd daemon. It
usually starts the binary "bbftpd -s", but this can be changed
througth the -E option.
-S Same as -s but ask ssh to run without asking a question (no pass‐
word, no host key checking). It implies the usage of the identity
file (for those familiar to ssh, this in done in setting the ssh
options BatchMode to yes and the option StrictHostKeyChecking to
no).
-E Server command to run
This option has to be used if the binary to be started on the
remote host is not the default one (usually "bbftpd -s"). This
option also implies the usage of the ssh mode (no need to add the
-s option)
-I SSH identity file
This option has to be used if the ssh identity file is not the
default one (usually $HOME/.ssh/identity). This option also
implies the usage of the ssh mode (no need to add the -s option)
-L SSH command
If your ssh command is not the default one (usually "ssh -q"), use
this option to change it. This option also implies the usage of
the ssh mode (no need to add the -s option)
-P Private authentication string
If your client have been compiled with a private authentication
schema, this option allow to pass an arbitrary string to the
authentication module. You can determine if your client is using a
private authentication module with the -v option.
-w Control port number
Use this option to change the control port for connection to on the
RemoteHost. This option is meaningless in ssh mode.
PROTOCOL OPTIONS
-D[min:max] Domain for ephemeral ports
Use this option to use protocol V2 (non-passive mode) instead of V3
(passive mode). min and max are the range of ports used for data
sockets. If min and max are not specified, free ports will be used
(or built-in ports at compilation time).
BEHAVIOUR OPTIONS
-b Use this option to run bbftp in the background after all interac‐
tive requests.
-c Use this option to gzip the data during transmission. Compression
and uncompression are done "on the fly" and do not require any
additionnal disk space. Do not use it if the files to transmit are
not compressible, as this will only lead to a waste of CPU and
time. This option can be overridden by the control command setop‐
tion gzip
-f ErrorFile
Use this option to redirect the data generated on the standard
error to ErrorFile
-g Globus service name
Use this option in certificate mode if the server does not use a
host server. The service name will be the subject of the certifi‐
cate the server is actually using.
-m Use to special output on file transfer. The ouput is in the follow‐
ing format:
Direction NumberOfBytes NumberOfSeconds BuffersizePerStream Send‐
WinSize RecvWinSize NumberOfStreams Compression
BuffersizePerStream, SendWinSize, RecvWinSize are expressed in
KiloBytes
For example :
get 10000 10 256 256 256 2 gzip
means that 10000 Bytes have been transfered in 10 seconds in com‐
pressed mode using 2 streams, a buffer size per stream of 256
KBytes, a TCP send window size of 256 KBytes and a TCP receive win‐
dow size of 256 KBytes.
This option is useful when trying to choose the best parameters
between two sites.
When this option is set -V, -W and -t options are not available.
-n Use this option to simulate the transfer. All commands will be exe‐
cuted but no data will be transfered with the get, mget put and
mput commands. Output is the same as a real transfer. OutputFile
-o OutputFile
Use this option to redirect the data generated on the standard out‐
put to OutputFile
-q Use this option to mark packets for QBSS (QBone Scavenger Service)
-p NumberOfParallelStreams
Use this option to increase the number of streams to use during the
file transfer. Default is 1. This option can be overridden by the
control command setnbstream
-r NumberOfTries
Use this option to change the number of tries to use when a trans‐
fer fails. The default is usually 5.
-R .bbftprc file
After a successful connection to the daemon the client is going to
execute all control commands located in the $HOME/.bbftprc file.
The location of this file can be changed with this option. Take
care, not all control commands are allowed in the .bbftprc file
(See CONTROL COMMANDS to know the authorized one)
-t Use this option to have a timestamp on all output (overridden by -m
option).
-V Use this option to set the client in verbose mode (overridden by -m
option).
-W Use this option to print warnings to stderr (overridden by -m
option).
CONTROL COMMANDS
The control commands are either contained by an ASCII file ( -i option)
or written on the bbftp line ( -e option). They can be divided into two
classes, the "File related commands" and the "Behaviour commands".
All "Behaviour commands" may be put in a .bbftprc file, but all "File
related commands" are forbiden in that file.
IMPORTANT NOTE
Under the RFIO mode (see setoption remoterfio and setoption localrfio )
all file related commands have to be given in absolute mode.
FILE RELATED COMMANDS
cd "RemoteDirectory"
Change the current directory of the daemon on the remote host.
If the RemoteDirectory is given in relative mode (not beginning by
a /), it is supposed to be relative to the directory where the dae‐
mon is currently running. After the first connection, the current
directory is the home directory of the RemoteUsername .
The client keeps in mind the current remote directory so in case of
broken connection during a transfer, it can reset the current
directory of the daemon to the correct directory.
If the daemon has been set in RFIO mode (see setoption remoterfio )
this option is unavailable.
get " RemoteFile LocalFile"
Transfer the remote file RemoteFile to the local host with the name
LocalFile. If the local file already exists it is overwritten
(only in case of successful transfer if the setoption tmpfile has
been set). Under some circumstances (No space left on device,
Access denied, File is a directory ...), no retry is done and the
next command is processed.
If the RemoteFile is given in relative mode (not beginning by a /),
it is supposed to be relative to the current directory on the
remote host (which is set to the home directory of the RemoteUser‐
name at the beginning). If the LocalFile is given in relative mode
(not beginning by a /) the file is created relative to the direc‐
tory where the bbftp command is running (which may have been
changed with the lcd command).
get " RemoteFile LocalDir/"
Transfer the remote file RemoteFile to the local host with the name
RemoteFile (without path) in the LocalDir directory.
If the local file already exists it is overwritten (only in case of
successful transfer if the setoption tmpfile has been set). Under
some circumstances (No space left on device, Access denied, File is
a directory ...), no retry is done and the next command is pro‐
cessed.
If the RemoteFile is given in relative mode (not beginning by a /),
it is supposed to be relative to the current directory on the
remote host (which is set to the home directory of the RemoteUser‐
name at the beginning).
If the LocalDir is given in relative mode (not beginning by a /)
the file is created relative to the directory where the bbftp com‐
mand is running (which may have been changed with the lcd command).
get " RemoteFile"
Transfer the remote file RemoteFile to the local host with the name
RemoteFile.
If the RemoteFile is given in relative mode (not beginning by a /),
it is supposed to be relative to the current directory on the
remote host (which is set to the home directory of the RemoteUser‐
name at the beginning) and created on the local host relative to
the directory where the bbftp command is running (which may have
been changed with the lcd command).
lcd " LocalDirectory"
Change the current directory on the local host. If the LocalDirec‐
tory is given in relative mode (not beginning by a /), it is sup‐
posed to be relative to the directory where the client is cur‐
rently.
If the client has been set into RFIO mode (see setoption localrfio
) this option is unavailable.
mget " RemoteFiles LocalDirectory"
Expand the RemoteFiles on the remote machine and do a "get" for
each file name thus produced. Files are transferred into the
LocalDirectory.
mget " RemoteFiles"
Expand the RemoteFiles on the remote machine and do a "get" for
each file name thus produced. Files are transferred into the local
working directory, which can be changed with the lcd command.
mkdir " RemoteDirectory"
Create the directory RemoteDirectory on the remote host. If the
directory already exist no retry is done and the next command of
the file is processed. If the RemoteDirectory is given in relative
mode (not beginning by a /) the directory is created relative to
the current remote directory.
If one directory in the given path does not exist the command will
fail if the setoption nocreatedir is set. If the setoption create‐
dir has been set all unexisting directories will be created.
mput " LocalFiles RemoteDirectory"
Expand wild cards in the list of local files given as arguments
and do a "put" for each file in the resulting list. Files are
transfered into the RemoteDirectory.
mput " LocalFiles"
Expand wild cards in the list of local files given as arguments
and do a "put" for each file in the resulting list. Files are
transfered into the current remote directory which can be changed
with the cd command.
put " LocalFile RemoteFile"
Transfer the local file LocalFile to the remote host with the name
RemoteFile. If the remote file already exists it is overwritten.
Under some circumstances (No space left on device, Access denied
...), no retry is done and the next command of the file is pro‐
cessed.
If the LocalFile is given in relative mode (not beginning by a /)
the file is supposed to be relative to the directory where the
bbftp command is running (which may have been changed with the lcd
command).
If the RemoteFile is given in relative mode (not beginning by a /),
it is created relative to the current directory on the remote host
(which is set to the home directory of the RemoteUsername at the
beginning).
put " LocalFile RemoteDir/"
Transfer the local file LocalFile to the remote host with the name
LocalFile (without the path) in the RemoteDir directory
If the remote file already exists it is overwritten. Under some
circumstances (No space left on device, Access denied ...), no
retry is done and the next command of the file is processed.
If the LocalFile is given in relative mode (not beginning by a /)
the file is supposed to be relative to the directory where the
bbftp command is running (which may have been changed with the lcd
command).
If the RemoteDir is given in relative mode (not beginning by a /),
it is created relative to the current directory on the remote host
(which is set to the home directory of the RemoteUsername at the
beginning).
put " LocalFile"
Transfer the local file LocalFile to the remote host with the name
LocalFile.
If the LocalFile is given in relative mode (not beginning by a /)
the file is supposed to be relative to the directory where the
bbftp command is running (which may have been changed with the lcd
command) and created relative to the current directory on the
remote host (which is set to the home directory of the RemoteUser‐
name at the beginning).
BEHAVIOUR COMMANDS
setoption " Option"
To negate an option just add "no" before the option (ie setoption
nocreatedir). The options are the following :
createdir
All file-related commands will create missing directories if needed
(default createdir).
gzip
All file transfers will be compressed using the gzip algorythm
(default nogzip).
keepaccess
The access time and modify time will be kept on each file trans‐
ferred (default keepaccess).
keepmode
The file mode will be kept on each file transferred (default keep‐
mode).
localrfio
All local files will be created with RFIO functions (default nolo‐
calrfio).
remoterfio
All remote files will be created with RFIO functions (default nore‐
moterfio).
qbss
All the packets will be marked for QBSS (default noqbss).
tmpfile
All files will be created under a temporary name (File‐
Name.bbftp.tmp.HostName.Pid) and renamed to the correct file name
if transfer is successful (default tmpfile)
setbuffersize " Buffersize"
Set the size in Kbytes of the buffer used for reading or writing
the files. This command set the local and remote buffer size. (Each
stream will use the same buffer size)
setlocalcos " LocalCos"
Set the local COS to the value specified by LocalCos. This COS
will be used for further rfio funtions. It is used if the setop‐
tion localrfio has been set and if the file is a HPSS file. A
value of 0 allows to select the COS according to the file size. A
negative value allows to not set the COS. The default value is 0.
setlocalumask " LocalUmask"
Set the local umask to the value specified by LocalUmask. This
umask will be used for further i/o funtions. The LocalUmask has to
be given in OCTAL
setnbstream " NumberOfParallelStreams"
Set the number of parallel streams to NumberOfParallelStreams.
This number will be used for further transfer commands.
setremotecos " RemoteCos"
Set the remote COS to the value specified by RemoteCos. This COS
will be used for further rfio funtions. It is used if the setop‐
tion remoterfio has been set and if the file is a HPSS file. A
value of 0 allows to select the COS according to the file size. A
negative value allows to not set the COS. The default value is 0.
setremoteumask " RemoteUmask"
Set the remote umask to the value specified by RemoteUmask. This
remote umask will be used for further i/o funtions. The RemoteUmask
has to be given in OCTAL
setrecvwinsize " WindowSize"
Set size in Kbytes of the receive TCP window of each stream of the
bbftpd daemon. This also set the send window size of the client to
the same value.
setsendwinsize " WindowSize"
Set size in Kbytes of the send TCP window of each stream of the
bbftpd daemon. This also set the receive window size of the client
to the same value.
setackto " Acknowledge time-out" Set time-out (in seconds) to wait for
an acknowledge. Default value is 100
setrecvcontrolto " Input control time-out"
Set time-out (in seconds) to wait while reading on the control
socket. Default value is 180
setsendcontrolto " Output control time-out"
Set time-out (in seconds) to wait while writing on the control
socket. Default value is 180
setdatato " Data time-out"
Set time-out (in seconds) to wait while reading on the data socket.
Default value is 300
NOTES If the option tmpfile is used then if the new file ( RemoteFile
for a put or LocalFile for a get) did not exist before, bbftp ensures
that the file transfer was correct if the file exists.
In case of an already existing file, if the size, the last access and
modification time are correct (if option keepaccess has been set)
bbftp ensures that the file transfer was correct.
RETURN VALUES
The following exit values are returned:
"0" if all commands were successfuly executed
">0"
if one command failed.
It may happend that a non-zero value is returned even if all files were
correctly transfered, if during one transfer a retry was needed. This
will be corrected in future releases.
MESSAGES AND ERRORS
All informative messages are written to the standard ouput (or to the
OutputFile ). All error messages are written to the standard error (or
to the ErrorFile ).
WARNING
The bbftp client version 2.0.0 is unable to talk with a daemon in
release 1.x.x.
The rfioxxx or xxxrfio commands are no longer supported, use instead
the options localrfio or remoterfio in conjunction with put and get
commands to obtain the same result.
RESULT FILE
If the -i option was used a result file will be created in the same
directory as the ControlFile. Its name is ControlFile with the exten‐
sion ".res". It contains the same lines as the ControlFile plus the
keyword "OK", in case of success, or "FAILED", in case of failure.
If the -e option was used and the -V option was not used, the software
will print the command executed plus the keyword "OK", in case of suc‐
cess, or "FAILED", in case of failure to standard output.
CONNECTION EXAMPLESbbftp-i ctrlfile -u jon -p 5 -c cchost.in2p3.fr
means that bbftp is going to connect to remote host cchost.in2p3.fr
using username jon. If the connection is successful then the commands
in ctrlfile will be executed. All transfer commands will use five
streams and gzip compression.
bbftp-i ctrlfile -u phg -s cchost.in2p3.fr
means that bbftp is going to start a remote bbftp via sshd on host
cchost.in2p3.fr using username phg. ssh will first try an RSAAuthenti‐
cation if it is allowed by cchost.in2p3.fr; otherwise ssh will ask for
a password for user phg on cchost.in2p3.fr. Then the sshd on
cchost.in2p3.fr will log user phg and try to start the command "bbftpd
-s"
bbftp-i ctrlfile -u jon -E '/tmp/bbftpd -s' cchost.in2p3.fr
Same behaviour as preceding, except that the remote command will be
"/tmp/bbftpd -s"
bbftp-i ctrlfile -u gilles -S cchost.in2p3.fr
means that bbftp is going to start using ssh a remote bbftpd on host
cchost.in2p3.fr using username gilles. ssh will try an RSAAuthentica‐
tion if it is allowed by cchost.in2p3.fr, otherwise the connection will
be broken.
bbftp-e 'setrecvwinsize 1024 ; put file1 file2' -u jon
cchost.in2p3.fr
means that bbftp is going to connect to remote host cchost.in2p3.fr
using jon username. If the connection is successful then the commands
setrecvwinsize 1024 and put file1 file2 will be executed. All tranfer
commands will use one stream.
bbftp-e 'put file1 file2' cchost.in2p3.fr
means (in the certification authentication mode) that bbftp is going to
connect to remote host cchost.in2p3.fr using a certificate. The remote
user will be detected by the daemon which will check for the certifi‐
cate provided and will accept or not the connection.
Using SSH to start a BBFTPD daemon linked with dynamic libraries
If you have linked the daemon with dynamic libraries with
-L/path/to/lib option, you need to specify this location in
$LD_LIBRARY_PATH. To be taken into account by SSHD, this environment
variable must be modified in the $HOME/.ssh/environment file.
See your SSH or SSHD manual for more details.
EXAMPLES
User jon want to transfer files from host localhost to remotehost on
the account bbrdist. The bbrdist account has /home/babar/bbrdist as
default directory on remotehost but has no subdirectories. We are going
to study a control file in order to understand bbftp behaviour (we do
not care here about the connection method; see CONNECTION EXAMPLES for
that).
User jon on the local host is on the /home/babar/jon directory and has
the following control file (all lines have a number which must not
exists but which are there just for clarity) :
1 setnbstream 20
2 setremoteumask 022
3 setoption nocreatedir
4 put /home/babar/jon/f1 /home/babar/bbrdist/newfiles/f1
5 setoption createdir
6 put /home/babar/jon/f1 /home/babar/bbrdist/newfiles/f1
7 setnbstream 5
8 setrecvwinsize 1024
9 setoption gzip
10 put /home/babar/jon/f2 /home/babar/bbrdist/newfiles/f2
Command 1 just sets the number of parallel streams to 20 for further
get or put commands.
Command 2 sets the remote umask for further put commands.
Command 3 indicates that no directory has to be created on further put
or get commands.
Command 4 tries to send the local file /home/babar/jon/f1 to
/home/babar/bbrdist/newfiles/f1. This command will fail because bbrdist
has no subdirectory and directory creation is inhibed.
Command 5 resets the createdir option.
Command 6 will be successful (if the connection does not break),
because the creation of the directory /home/babar/bbrdist/newfiles has
been authorized by the createdir option.
Command 7 reduces the number of streams to 5
Command 8 sets the receive TCP window size to 1024 Kbytes on remotehost
and the send TCP window size to 1024 Kbytes on localhost.
Command 9 sets the gzip option for further get or put commands.
Command 10 will transfer file /home/babar/jon/f2 to
/home/babar/bbrdist/newfiles/f2 with 5 streams in compressed mode.
AUTHORSbbftp was developed by Gilles Farrache. It is now maintained by Lionel
Schwarz at IN2P3 Computing Center , Villeurbanne (FRANCE).
CONTRIBUTORS
Tim Adye (Idea and implementation of ssh mode) Paola Grosso (Idea and
implementation of the -q client option) Dan Schrager (Idea and imple‐
mentation of the -D client option) Rod Walker & Kostas Georgiou (Idea
and implementation of the -g client option) Shuwei Ye (Bug fix)
BUGS
Send bugs / comments to bbftp@in2p3.fr
SEE ALSObbftpd(1).
perl v5.8.0 2003-11-03 BBFTP(1)
