DMAKE(1)DMAKE(1)NAMEdmake - Distributed Make
SYNOPSISdmake [-c dmake-rcfile] [-g dmake-group] [-j dmake-max-jobs] [-m
{serial | parallel | distributed | grid}] [-o dmake-odir]
DESCRIPTION
Note: This man page supplements the Solaris make utility man page. The
dmake utility will run under the Linux operating system, but the dmake
options will be independent of any other make utility.
Distributed Make (dmake) parses your makefiles and determines which
target can be built concurrently, and distributes the build of those
targets over a number of hosts set by you.
If you use the standard make utility, the transition to dmake requires
little if any alteration to your makefiles. dmake is a superset of the
make utility. With nested makes, if a top-level makefile calls "make",
you need to use $(MAKE).
You execute dmake on a "dmake host" and distribute "jobs" to "build
servers." You can also distribute jobs to the dmake host, in which
case the dmake host also becomes a build server. dmake distributes
jobs based on makefile targets that dmake determines (based on your
makefiles) can be built concurrently.
You can use any machine as a build server as long as it meets the fol‐
lowing requirements:
- From the dmake host (the machine you are using) you must be able to
use rsh, without being prompted for a password, to remotely execute
commands on the build server. The following example shows the result
of using the which command on dmake if the machine is running the
Solaris operating system:
demo% rsh <machine_name> which dmake
/bin/dmake
The rsh must be "clean", with no additional output.
- The bin directory in which the dmake software is installed must be
accessible from the build server. By default, dmake assumes that the
logical path to the dmake executables on the build server is the same
as the dmake host. This assumption can be overridden by specifying a
path name a an attribute of the host entry in the runtime configura‐
tion file.
From the dmake host you can control which build servers are used and
how many dmake jobs are allotted to each build server. The number of
dmake jobs that can run on a given build server can also be limited on
that server.
OPTIONS-c dmake-rcfile
Specifies an alternate runtime configuration file. The default
runtime configuration file is $(HOME)/.dmakerc.
-g dmake-group
Specifies the name of the build server group to which jobs are
distributed. You define server groups in the runtime configura‐
tion file. The default server group is the first group in the
runtime configuration file.
-j dmake-max-jobs
Specifies the maximum total number of jobs that are distributed to
the specified group of build servers in the runtime configuration
file. The default maximum number of jobs is the sum of all the
specified jobs in a build server group. The jobs are subtracted
from, or added to, hosts by 1 in the order they appear in the run‐
time configuration file. For example, if all jobs specified in
the runtime configuration file total 8:
host earth { jobs = 3 }
host mars { jobs = 5 }
and dmake-max-jobs is specified as 11, dmake adds three more jobs
to the current total maximum number of jobs (which is eight) as
follows:
host earth { jobs = 5 }
host mars { jobs = 6 }
Also, if dmake-max-jobs is specified as 4, dmake subtracts 4 jobs
(from the original eight) as follows:
host earth { jobs = 1 }
host mars { jobs = 3 }
-m {serial | parallel | distributed | grid}
Specify one of the following key words:
serial Causes dmake to behave like the standard serial ver‐
sion of make.
parallel Causes dmake to distribute jobs to only the dmake
host.
distributed Causes dmake to behave in fully distributed mode.
This is the dmake default.
grid Causes dmake to use SGE (Sun Grid Engine) to distrib‐
ute build jobs.
-o dmake-odir
Specifies a common physical directory that dmake can write tempo‐
rary output files to and read temporary output files from. The
directory used is $(HOME)/.dmake and this or whichever directory
is specified, must be visible to all build servers. Use this
option only if the $(HOME) directory on your local host and the
$(HOME) directory on all of your remote hosts are NOT the same
physical $(HOME) directory. For example, a root user would use
this option.
Note:
These options and the environment variables and makefile macros
described later in this man page modify the same behavior. Their
order of precedence is defined as:
1. Command-line options
2. Makefile macros
3. Environment variables
4. dmake default
USAGE
Special-purpose Targets
dmake allows targets to be built concurrently on a number of build
servers. Concurrent processing can greatly reduce the time required to
build a large system or project. dmake supplies the special makefile
targets .PARALLEL, .NO_PARALLEL, .LOCAL, and .WAIT for controlling con‐
currency and timing.
.NO_PARALLEL: Use this target to indicate which targets are to be
processed serially.
.PARALLEL: Use this target to indicate which targets are to be
processed in parallel.
.LOCAL: Use this target to indicate which targets are to be
processed serially on the local host.
.WAIT When you specify this target in a dependency list,
dmake waits until the dependencies that precede it are
finished before processing those that follow, even
when processing is parallel.
Makefiles that you write using these targets remain compatible with the
standard version of make distributed with Solaris 1.x and Solaris 2.x.
Standard make accepts these targets without error (and without action).
Controlling dmake Jobs
The distribution of dmake jobs is controlled in two ways:
1. A dmake user on a dmake host can specify the machines they
want to use as build servers and the number of jobs they want to
distribute to each build server.
2. The owner (a user that can alter the
/etc/opt/SPROdmake/dmake.conf file) on a build server can con‐
trol the maximum total number of dmake jobs that can be distrib‐
uted to that build server.
The dmake Host
When dmake begins execution it searches for a runtime configuration
file to know where to distribute jobs. Generally, this file is located
in your home directory on the dmake host and is named .dmakerc. dmake
searches for the runtime configuration file in the following locations
and in the following order:
1. The path name you specify on the command line using the -c option
2. The path name you specify using the DMAKE_RCFILE makefile macro
3. The path name you specify using the DMAKE_RCFILE environment vari‐
able
4. $(HOME)/.dmakerc
If a runtime configuration file is not found, dmake switches to paral‐
lel mode and distributes two jobs (the default) to the dmake host. You
can change this using the -j option, or DMAKE_MAX_JOBS.
The runtime configuration file may contain a list of build servers and
the number of jobs you want distributed to each build server. The fol‐
lowing is a sample of a simple runtime configuration file:
# My machine. This entry causes dmake to distribute to it
falcon { jobs = 1 }
hawk
eagle { jobs = 3 }
# Manager's machine. She's usually at meetings
heron { jobs = 4 }
avocet
The entries: falcon, hawk, eagle, heron, and avocet are listed as build
servers.
You can specify the number of jobs you want distributed to each build
server. The default number of jobs is two.
Any line that begins with the "#" character is interpreted as a com‐
ment.
Note: This list of build servers includes falcon which is also the
dmake host. The dmake host can also be specified as a build
server. If you do not include it in the runtime configuration
file, no dmake jobs are distributed to it.
You can also construct groups of build servers in the runtime configu‐
ration file. This provides you with the flexibility of easily switch‐
ing between different groups of build servers as circumstances warrant.
For instance you may define a different group of build servers for
builds under different operating systems, or on groups of build servers
that have special software installed on them. The build servers must
be all the same architecture and have the same SunOS installed.
The following runtime configuration file contains groups:
earth { jobs = 2 }
mars { jobs = 3 }
group lab1 {
host falcon { jobs = 3 }
host hawk
host eagle { jobs = 3 }
}
group lab2 {
host heron
host avocet { jobs = 3 }
host stilt { jobs = 2 }
group labs {
group lab1
group lab2
}
group sunos5.x {
group labs
host jupiter
host venus { jobs = 2 }
host pluto { jobs = 3 }
}
Formal groups are specified by the "group" directive and lists of their
constituents are delimited by braces ({}).
Build servers that are constituents of groups are specified by the
optional "host" directive.
Groups can be constituents of other groups.
Individual build servers can be listed in runtime configuration files
that also contain groups of build servers. In this case dmake treats
these build servers as constituents of the unnamed group.
dmake distributes jobs to a single group of hosts specified by the fol‐
lowing list and in precedence from 1 to 4.
1. The group specified on the command-line as an argument to the -g
option
2. The group specified by the DMAKE_GROUP makefile macro
3. The group specified by the DMAKE_GROUP environment variable
4. The first formal group listed in the runtime configuration file
The names of groups and hosts specified in the runtime configuration
file may be enclosed in double quotes. This is to allow more flexibil‐
ity with respect to the character sequences that may appear as part of
the group and host names. For example, if the name of the group starts
with a digit it should be double-quoted:
group "123_sparc"
As mentioned above, the bin directory in which the dmake software is
installed must be accessible from the build server. By default, dmake
assumes that the logical path to the dmake executables on the build
server is the same as the dmake host. This assumption can be overridden
by specifying a path name as an attribute of the host entry in the run‐
time configuration file. For example:
group sparc-cluster {
host wren { jobs = 10 , path = "/export/sunstudio/bin" }
host stimpy { path = "/bin" }
}
The Build Server
The /etc/opt/SPROdmake/dmake.conf file is located in the file system of
build servers. Use this file to specify the following:
Required The maximum total number of dmake jobs (from all users) that
can run concurrently on that build server.
Optional The /usr/bin/ priority under which all dmake jobs are to be
run.
The following is a sample of a dmake.conf file:
max_jobs: 8
nice_prio: 5
This file sets the maximum number of dmake jobs permitted to run on
that build server (from all dmake users) to be eight. You can change
the priority of the jobs to be run by using the nice_prio command. See
nice(1).
Note: If the /etc/opt/SPROdmake/dmake.conf file does not exist on a
build server, no dmake jobs will be allowed to run on that
server.
ENVIRONMENT/MACROS
The following can be defined as either environment variables or make‐
file macros:
DMAKE_RCFILE
Defines an alternate runtime configuration file. The default run‐
time configuration file is $(HOME)/.dmakerc
DMAKE_GROUP
Defines the name of the build server group to which jobs are dis‐
tributed. Server groups are defined in the runtime configuration
file. The default server group is the first group in the runtime
configuration file.
DMAKE_MAX_JOBS
Defines the maximum total number of jobs that are distributed to
the specified group of build servers in the runtime configuration
file. The default maximum number of jobs is the sum of all the
specified jobs in a build server group. The jobs are subtracted
from, or added to, hosts by 1 in the order they appear in the run‐
time configuration file. See the -j option in this man page for
an example.
DMAKE_ADJUST_MAX_JOBS
May contain one of the following key words:
YES Allows dmake to adjust the limit of parallel jobs according
to the current loading of the system. If the system is not
overloaded, dmake will use the limit defined by the user. If
the system is overloaded, dmake will set the "current" limit
to less than the limit defined by the user.
If this variable is not set, dmake will adjust the limit of
parallel jobs according to the current loading of the system.
This is the dmake default.
NO Causes dmake to switch off the autoadjustment mechanism.
DMAKE_MODE
May contain one of the following key words:
serial Causes dmake to behave like the standard serial ver‐
sion of make.
parallel Causes dmake to distribute jobs to only the dmake
host.
distributed Causes dmake to behave in fully distributed mode.
This is the dmake default.
grid Causes dmake to use SGE (Sun Grid Engine) to distrib‐
ute build jobs.
DMAKE_ODIR
Defines a common physical directory that dmake can write temporary
output files to and read temporary output files from. Use this
environment variable, or macro, only if the $(HOME) directory on
your local host and the $(HOME) directory on all of your remote
hosts are NOT the same physical $(HOME) directory. For example, a
root user would use this option:
DMAKE_OUTPUT_MODE
Defines the format of the log file. May contain one of the follow‐
ing keywords:
TEXT1 On starting each build job, dmake prints the name of the
system and command to the log file. Also, if the command
itself prints any output, then as the job finishes, dmake
prints the name of the system and command to the log file
again along with the command output.
Example:
host1 --> 1 job
echo "Done host1"
host2 --> 1 job
echo "Done host2"
host1 --> Job output
echo "Done host1"
Done host1
host2 --> Job output
echo "Done host2"
Done host2
This is the dmake default.
TEXT2 Allows dmake to serialize the output of parallel jobs,
which makes the log file more readable. In this mode,
dmake just prints the command once, immediately followed
by the command output, as each job finishes.
Example:
echo "Done host1"
Done host1
echo "Done host2"
Done host2
FILES
$(HOME)/.dmakerc The default runtime configuration file. Contains
the names of build servers and groups of build
servers.
/etc/opt/SPROdmake/dmake.conf
Located on build servers, this file is used to
specify the maximum total number of jobs that can
be distributed to it by all dmake users. It is
also used to specify the /usr/bin/nice priority all
dmake jobs are to be run under.
SEE ALSO
Access the complete Sun Studio documentation set by pointing an HTML
browser at:
http://developers.sun.com/sunstudio/documentation/
Consult the following man pages for additional information:
make(1), rsh(1), hosts(4), hosts.equiv(4), attributes(5), largefile(5),
qrsh(1)
2009/04/15 DMAKE(1)
