dd_rescue(1) Rescue copy tool dd_rescue(1)NAMEdd_rescue - Rescue copy tool
SYNOPSISdd_rescue [options] infile outfile
dd_rescue [options] [-2/-3/-4/-z/-Z seed/seedfile] outfile
dd_rescue [options] [--shred2/--shred3/--shred4/--random/--frandom
seed/seedfile] outfile
DESCRIPTIONdd_rescue is a tool that copies data from a source (file, block device,
pipe, ...) to one (or several) output file(s).
If input and output files are seekable (block devices or regular
files), dd_rescue does copy with large blocks (softbs) to increase
performance. When a read error is encountered, dd_rescue falls back to
reading smaller blocks (hardbs), to allow to recover the maximum amount
of data. If blocks can still not be read, dd_rescue by default skips
over them also in the output file, avoiding to overwrite data that
might have been copied there successfully in a previous run. (Option
-A / --alwayswrite changes this.).
dd_rescue can copy in reverse direction as well, allowing to approach a
bad spot from both directions. As trying to read over a bad spot of
significant size can take very long (and potentially cause further
damage), this is an important optimization when recovering data. The
dd_rhelp tool takes advantage of this and automates data recovery.
dd_rescue does not (by default) truncate the output file.
dd_rescue by default reports on progress, and optionally also writes
into a logfile. It has a progress bar and gives an estimate for the
remaining time. dd_rescue has a wealth of options that influence its
behaviour, such as the possibility to use direct IO for input/output,
to use fallocate() to preallocate space for the output file, using
splice copy (in kernel zerocopy) for efficiency, looking for empty
blocks to create sparse files, or using a pseudo random number
generator (PRNG) to quickly overwrite data with random numbers.
You can use "-" as infile or outfile, meaning stdin or stdout. Note
that this means that either file is not seekable, limiting the
usefulness of some of dd_rescues features.
OPTIONS
When parsing numbers, dd_rescue assumes bytes. It accepts the following
suffixes:
b -- 512 size units (blocks)
k -- 1024 size units (binary kilobytes, kiB)
M -- 1024^2 size units (binary megabytes, MiB)
G -- 1024^3 size units (binary gigabytes, GiB)
The following options may be used to modify the behaviour of dd_rescue
General options
-h, --help
This option tells dd_rescue to output a list of options and
exit.
-V, --version
Display version number and exit.
-q, --quiet
tells dd_rescue to be less verbose.
-v, --verbose
makes dd_rescue more verbose.
-c 0/1, --color=0/1
controls whether dd_rescue uses colors. By default it does,
unless the terminal type from TERM is unknown or dumb or ends
in -m or -mono.
-f, --force
makes dd_rescue skip some sanity checks (e.g. automatically
setting reverse direction when input and output file are the
same and ipos < opos).
-i, --interactive
tells dd_rescue to ask before overwriting existing files.
Block sizes
-b softbs, --softbs=softbs, --bs=softbs
sets the (larger) blocksize to softbs bytes. dd_rescue will
transfer chunks of that size unless a read error is encountered
(or the end of the input file or the maximum transfer size has
been reached). The default value for this is 64k for buffered
I/O and 1M for direct I/O.
-B hardbs, --hardbs=hardbs, --block-size=hardbs
sets the (smaller) fallback blocksize to hardbs bytes. When
dd_rescue encounters read errors, it will fall back to copying
data in chunks of this size. This value defaults to 4k for
buffered I/O and 512 bytes for direct I/O.
hardbs should be equal to or smaller than softbs. If both
block sizes are identical, no fallback mechanism (and thus no
retry) will take place on read errors.
-y syncsize, --syncfreq=syncsize
tells dd_rescue to call fsync() on the output file every
syncsize bytes (will be rounded to multiples of softbs sized
blocks). It will also update the progress indicator at least as
often. By default, syncsize is set to 0, meaning that fsync()
is only issued at the end of the copy operation.
Positions and length
-s ipos, --ipos=ipos, --input-position=ipos
sets the starting position of the infile to ipos. Note that
ipos is specified in bytes (but suffixes can be used, see
above), not in terms of softbs or hardbs sized blocks. The
default value for this is 0. When reverse direction copy is
used, an ipos of 0 is treated specially, meaning the end of
file.
Negative positions result in an error message.
-S opos, --opos=opos, --output-position=opos
sets the starting position of the outfile to opos. If not
specified, opos is set to ipos, so the file offsets in input
and output file are the same. For reverse direction copy, an
explicit opos of 0 will position at the end of the output file.
-x, --extend, --append
changes the interpretation of the output position to start at
the end of the existing output file, making appending to a file
convenient. If the output file does not exist, an error will
be reported and dd_rescue aborted.
-m maxxfer, --maxxfer=maxxfer, --max-size=maxxfer
specifies the maximum number of bytes (suffixes apply, but it's
NOT counted in blocks) that dd_rescue copies. If EOF is
encountered before maxxfer bytes have been transferred, this
option will be silently ignored.
-M, --noextend
tells dd_rescue to not extend the output file. This option is
particularly helpful when overwriting a file with random data
or zeroes for safe data destruction. If the output file does
not exist, an error message will be generated and the program
be aborted.
Error handling
-e maxerr, --maxerr=maxerr
tells dd_rescue to exit, after maxerr read errors have been
encountered. By default, this is set to 0, resulting in
dd_rescue trying to move on until it hits EOF (or maxxfer bytes
have been transferred).
-w, --abort_we
makes dd_rescue abort on any write errors. By default, write
errors are ignored. This is because for buffered writes,
errors can't be detected and located reliably anyway. Also,
write errors are not often reported to userspace, as hard
drives e.g. reallocate bad blocks on writes and the OS block
layer typically also does some retries on write operations.
Note that dd_rescue does exit if writes to the output file
result in the Operating System reporting that no space is left.
Sparse files and write avoidance
-A, --alwayswrite
changes the behaviour of dd_rescue to write zeroes to the
output file when the input file could not be read. By default,
it just skips over, leaving whatever content was in the output
file at the file position before. The default behaviour may be
desired, if e.g. previous copy operations may have resulted in
good data being in place; it may be undesired if the output
file may contain garbage (or sensitive information) that should
rather be overwritten with zeroes.
-a, --sparse
will make dd_rescue look for empty blocks (of softbs size),
i.e. blocks filled with zeroes. Rather than writing those
zeroes to the output file, it will then skip forward in the
output file, resulting in a sparse file, saving space in the
output filesystem (if it supports sparse files). Note that if
the output file does already exist and already has data stored
at the location where zeroes are skipped over, this will result
in an incomplete copy in that the output file is different from
the input file, where blocks of zeroes are skipped over.
dd_rescue tries to detect this and issue a warning, but it does
not prevent this from happening
-W, --avoidwrite
results in dd_rescue reading a block ( softbs sized) from the
output file prior to writing it. If it is already identical
with the data that would be written to it, the writes are
actually avoided. This option may be useful for devices, where
e.g. writes should be avoided (e.g. because they may impact the
reaining lifetime or because they are very slow compared to
reads).
Other optimizations
-R, --repeat
tells dd_rescue to only read one block ( softbs sized) and then
repeatedly write it to the output file. Note that this results
in never hitting EOF on the input file and should be used with
a limit for the transfer size (options -m or -M) or when
filling up an output device completely.
This option is automatically set, if the input file name equals
"/dev/zero".
-k, --splice
tells dd_rescue to use the Linux in-kernel zerocopy splice()
copy operation rather than reading blocks into a userspace
buffer. Note that this operation mode does prevent the support
of a number of dd_rescue features that can normally be used,
such as falling back to smaller block sizes, avoiding writes,
sparse mode, repeat optimization, reverse direction copy. A
warning is issued to make the user aware.
-P, --fallocate
results in dd_rescue calling fallocate() on the output file,
telling the filesystem how much space to preallocate for the
output file. (The size is determined by the expected last
position, as inferred from the input file length and maxxfer ).
On filesystems that support it, this results in them making
better allocation decisions, avoiding fragmentation. (Note that
it does not make sense to use sparse together with
fallocate().)
This option is only available if dd_rescue is compiled with
fallocate() support. For optimal support, it should be compiled
with the libfallocate library.
Misc options
-r, --reverse
tells dd_rescue to copy in reverse direction, starting at ipos
(with special case 0 meaning EOF) and working towards the
beginning of the file. This is especially helpful if the input
file has a bad spot which can be extremely slow to skip over,
so approaching it from both directions saves a lot of time (and
may prevent further damage).
Note that dd_rescue does automatically switch to reverse
direction copy, if input and output file are identical and the
input position is smaller than the output position, similar to
the intelligence that memmove() uses to prevent loss of data
when overlapping areas are copied. The option -f / --force does
prevent this intelligence from happening.
-p, --preserve
When copying files, this option does result in file metadata
(timestamps, ownership, access rights, xattrs) to be copied,
similar to the option with the same name in the cp program.
Note that dd_rescue at this point does not have support for
copying extended attributes or ACLs. (But the behaviour of -p
is still consistent with cp.)
-t, --truncate
tells dd_rescue to open the output file with O_TRUNC, resulting
in the output file (if it is a regular file) to be truncated to
0 bytes, removing all previous content that the file may have
contained. By default, dd_rescue does not remove previous
content.
-T, --trunclast
tells dd_rescue to truncate the output file to the highest
copied position, thus ensuring there's no data beyond the end
of the data that has been copied in this run.
-d, --odir_in
instructs dd_rescue to open infie with O_DIRECT, bypassing the
kernel buffers. While this option has a negative effect on
performance (the kernel does readahead for buffered I/O), it
will result in errors to be detected more quickly (kernel won't
retry) and allows for smaller I/O units (hardware sector size,
512bytes for most hard disks).
O_DIRECT may not be available on all platforms.
-D, --odir_out
tells dd_rescue to open outfile with O_DIRECT, bypassing kernel
buffers. This has a significant negative effect on performance,
as the program needs to wait for writes to hit the disks as
opposed to the asynchronous nature of buffered writeback. On
the flipside, the return status from writing is reliable this
way and smaller I/O chunks (hardware sector size, 512bytes) are
possible.
Logging
-l logfile, --logfile=logfile
Unless in quiet mode, dd_rescue does produce constant updates
on the status of the copy operation to stderr. With this
option, these updates are also written to the specified
logfile. The control characters (to move the cursor up to
overwrite the existing status lines) are not written to the
logfile.
-o bbfile, --bbfile=bbfile
instructs dd_rescue to write a list of bad blocks to bbfile.
The file will contain a list of numbers (ASCII), one per line,
where the numbers indicate the offset in terms of hardbs sized
blocks. The file format is compatible with that of badblocks.
Using dd_rescue on a block device (partition) and setting
hardbs to the block size of a filesystem that you want to
create, you should be able to feed the bbfile to mke2fs with
the option -l.
Multiple output files
-Y ofileX, --outfile=ofileX, --of=ofileX
If you want to copy data to multiple filess simultaneously, you
can specify this option. It can be specified multiple times, so
many copies can be made. Note that these files are secondary
output files; they share file position with the primary output
file outfile. Errors when writing to a secondary output file
are ignored.
Random data generation
-z RANDSEED, --random=RANDSEED
-Z RANDSEED, --frandom=RANDSEED
-2 RANDSEED, --shred2=RANDSEED
-3 RANDSEED, --shred3=RANDSEED
-4 RANDSEED, --shred4=RANDSEED
When you want to overwrite a file, partition or disk with
random data, using /dev/urandom (on Linux) as input is not a
very good idea; the interface has not been designed to yield a
high bandwidth. It's better to use a userspace Pseudo Random
Number Generator (PRNG). With option -z / --random, the C
library's PRNG is used. With -Z / --frandom and the -2/-3/-4 /
--shred2/3/4 options, an RC4 based PRNG is used.
Note that in this mode, there is no infile so the first non-
option argument is the output file.
The PRNG needs seeding; the C libraries PRNG takes a 32bit
integer (4 bytes); the RC4 based PRNG takes 256 bytes. If
RANDSEED is an integer, the integer number will be used to seed
the C library's PRNG. For the RC4 method, the C library's PRNG
then generates the 256 bytes to seed it. This creates
repeatable PRNG data. The RANDSEED value of 0 is special; it
will create a seedval that's based on the current time and the
process' PID and should be different for multiple runs of
dd_recue .
If RANDSEED is not an integer, it's assumed to be a filename
from which the seed values can be read. dd_rescue will read 4
or 256 bytes from the file to seed the C library's or the RC4
PRNG. For good pseudo random numbers, using /dev/urandom to
seed is a good idea.
The modes -2/-3/-4 resp. --shred2/--shred3/--shred4 will
overwrite the output file multiple times; after each pass,
fsync() will ensure that the data does indeed hit the file. The
last pass for these modes will overwrite the file with zeroes.
The rationale behind doing this is to make it easier to hide
that important data may have been overwritten, to make it
easier for intelligent storage systems (such as SSDs) to
recycle the empty blocks and to allow for better compression of
a filesystem image containing such data.
With -2 / --shred2, one pass with RC4 generated PRNG is
happening and then zeroes are written. With -3 / --shred3,
there are two passes with RC4 PRNG generated random numbers and
a zero pass; the second PRNG pass writes the inverse (bitwise
reversed) numbers from the first pass. -4 / --shred4 works like
-3 / --shred3, with an additional pass with independent random
numbers as third pass.
EXIT STATUS
On successful completion, dd_rescue returns an exit code of 0. Any
other exit code indicates that the program has aborted because of an
error condition or that copying of the file has not been entirely
successful.
EXAMPLESdd_rescue-k -P -p -t infile outfile
copies infile to outfile and does truncate the output file on
opening (so deleting any previous data in it), copies mode,
times, ownerships at the end, uses fallocate to reserve the
space for the output file and uses efficient in kernel splice
copy method.
dd_rescue-A -d -D -b 512 /dev/sda /dev/sda
reads the contents of every sector of disk sda and writes it
back to the same location. Typical hard disks reallocate flaky
and faulty sectors on writes, so this operation may result in
the complete disk being usable again when there were errors
before. Unreadable blocks however will contain zeroes after
this.
dd_rescue-2 /dev/urandom -M outfile
overwrites the file outfile twice; once with good pseudo random
numbers and then with zeroes.
dd_rescue-t -a image1.raw image2.raw
copies a filesystem image and looks for empty blocks to create a
sparse output file to save disk space. (If the source filesystem
has been used a bit, on that filesystem creating a large file
with zeroes and removing it again prior to this operation will
result in more sectors with zeroes. dd_rescue /dev/zero DUMMY;
rm DUMMY will achieve this ...)
See README.dd_rescue to learn about some more possibilities.
BUGS/LIMITATIONS
The source code does use the 64bit functions provided by glibc for file
positioning. However, your kernel might not support it, so you might be
unable to copy partitions larger then 2GB into a file.
This program has been written using Linux and only tested on a couple
of Linux systems. It should be possible to port it easily to other Un*x
platforms, and I occasionally get hints from users of other *nix like
systems that I take into account. Still, my own testing is limited to
Linux.
Currently, the escape sequence for moving the cursor up is hard coded
in the sources. It's fine for most terminal emulations (including vt100
and linux), but it should use the terminal description database
instead.
Since dd_rescue-1.10, non-seekable input or output files are supported,
but there's of course limitations to recover errors in such cases.
The modes for overwriting data with pseudo random numbers to securely
delete sensitive data on purpose only implement a limited number of
overwrites. While Peter Gutmann's classic analysis concludes that the
then current harddisk technology requires more overwrites to be really
secure, the author believes that modern hard disk technology does not
allow data restoration in sectors that have been overwritten with the
--shred4 mode. This is in compliance with the recommendations from BSI
GSDS M7.15.
However, users should be aware that depending on your filesystem,
overwriting files with this technology may not render file content
safely overwritten on hard disk. This is due to journals keeping data
or filesystems writing data to different locations every time. So here,
only overwriting a whole partition or disk will be effective. The fact
that the hard disk firmware has spare sectors to be allocated when
regular sectors become flaky or faulty may leave traces of files on the
medium. The Flash Translation Layer on SSDs may have similar effects.
dd_rescue does not automate the recovery of faulty files or partitions
by automatically keeping a list of copied sectors and approaching bad
spots from both sides. There is a helper script dd_rhelp from LAB
Valentin that does this. Integration of such a mode into dd_rescue is
non-trivial and due to the complexity of the source code probably will
never happen.
There also is a tool, GNU ddrescue, that is a reimplementation of this
tool and which contains the capabilities to automate recovery of bad
files in the way dd_rhelp does.
Please report bugs to me via email.
SEE ALSO
README.dd_rescue README.dd_rhelp wipe(1)shred(1)ddrescue(1)dd(1)AUTHOR
Kurt Garloff <kurt@garloff.de>
CREDITS
Many little issues were reported by Valentin LAB, the author of
dd_rhelp .
The RC4 PRNG (frandom) is a port from Eli Billauer's kernel mode PRNG.
A number of recent ideas and suggestions came from Thomas.
COPYRIGHT
This program is protected by the GNU General Public License (GPL) v2 or
v3 - at your option.
HISTORY
Since version 1.10, non seekable input and output files are supported.
Splice copy -k is supported since 1.15.
A progress bar exists since 1.17.
Support for preallocation (fallocate) -P exists since 1.19.
Since 1.23, we default to -y0, enhancing performance.
The Pseudo Random Number modes have been started with 1.29.
Write avoidance -W has been implemented in 1.30
Multiple output files -Y have been added in 1.32.
Long options and man page came with 1.33.
More detailed information can be found on
http://garloff.de/kurt/linux/ddrescue/
Kurt Garloff 2013-02-24 dd_rescue(1)
