2 @mainpage Remake, a build system that bridges the gap between make and redo.
4 As with <b>make</b>, <b>remake</b> uses a centralized rule file, which is
5 named <b>Remakefile</b>. It contains rules with a <em>make</em>-like
9 target1 target2 ... : prerequisite1 prerequisite2 ...
15 A target is known to be up-to-date if all its prerequisites are. If it
16 has no known prerequisites yet the file already exits, it is assumed to
17 be up-to-date. Obsolete targets are rebuilt thanks to the shell script
20 As with <b>redo</b>, <b>remake</b> supports dynamic dependencies in
21 addition to these static dependencies. Whenever a script executes
22 <tt>remake prerequisite4 prerequisite5 ...</tt>, these prerequisites are
23 rebuilt if they are obsolete. (So <b>remake</b> acts like
24 <b>redo-ifchange</b>.) Moreover, all the dependencies are stored in file
25 <b>.remake</b> so that they are remembered in subsequent runs. Note that
26 dynamic dependencies from previous runs are only used to decide whether a
27 target is obsolete; they are not automatically rebuilt when they are
28 obsolete yet a target depends on them. They will only be rebuilt once the
29 dynamic call to <b>remake</b> is executed.
31 In other words, the following two rules have almost the same behavior.
34 target1 target2 ... : prerequisite1 prerequisite2 ...
38 remake prerequisite1 prerequisite2 ...
42 (There is a difference if the targets already exist, have never been
43 built before, and the prerequisites are either younger or obsolete, since
44 the targets will not be rebuilt in the second case.)
46 The above usage of dynamic dependencies is hardly useful. Their strength
47 lies in the fact that they can be computed on the fly:
51 gcc -MMD -MF $@.d -o $@ -c $<
56 ocamldep $< | remake -r $@
59 after.xml: before.xml rules.xsl
60 xsltproc --load-trace -o after.xml rules.xsl before.xml 2> deps
61 remake `sed -n -e "\\,//,! s,^.*URL=\"\\([^\"]*\\).*\$,\\1,p" deps`
65 Note that the first rule fails if any of the header files included by
66 a C source file has to be automatically generated. In that case, one
67 should perform a first call to <b>remake</b> them before calling the
68 compiler. (Dependencies from several calls to <b>remake</b> are
69 cumulative, so they will all be remembered the next time.)
71 \section sec-usage Usage
73 Usage: <tt>remake <i>options</i> <i>targets</i></tt>
77 - <tt>-d</tt>: Echo script commands.
78 - <tt>-f FILE</tt>: Read <tt>FILE</tt> as <b>Remakefile</b>.
79 - <tt>-j[N]</tt>, <tt>--jobs=[N]</tt>: Allow <tt>N</tt> jobs at once;
80 infinite jobs with no argument.
81 - <tt>-k</tt>, <tt>--keep-going</tt>: Keep going when some targets cannot be made.
82 - <tt>-r</tt>: Look up targets from the dependencies on standard input.
83 - <tt>-s</tt>, <tt>--silent</tt>, <tt>--quiet</tt>: Do not echo targets.
85 \section sec-syntax Syntax
87 Lines starting with a space character or a tabulation are assumed to be rule
88 scripts. They are only allowed after a rule header.
90 Lines starting with <tt>#</tt> are considered to be comments and are ignored.
91 They do interrupt rule scripts though.
93 Any other line is either a variable definition or a rule header. If such a
94 line ends with a backslash, the following line break is ignored and the line
95 extends to the next one.
97 Variable definitions are a single name followed by equal followed by a list
98 of names, possibly empty.
100 Rule headers are a nonempty list of names, followed by a colon, followed by
101 another list of names, possibly empty. Basically, the syntax of a rule is as
105 targets : prerequisites
109 List of names are space-separated sequences of names. If a name contains a
110 space character, it should be put into double quotes. Names can not be any
111 of the following special characters <tt>:$(),="</tt>. Again, quotation
112 should be used. Quotation marks can be escaped by a backslash inside
115 \subsection sec-variables Variables
117 Variables can be used to factor lists of targets or prerequisites. They are
118 expanded as they are encountered during <b>Remakefile</b> parsing.
127 Variable assignments can appear instead of prerequisites inside non-generic
128 rules with no script. They are then expanded inside the corresponding
132 foo.o: CFLAGS += -DBAR
135 gcc $(CFLAGS) -MMD -MF $@.d -o $@ -c $<
140 Note: contrarily to <b>make</b>, variable names have to be enclosed in
141 parentheses. For instance, <tt>$y</tt> is not a shorthand for <tt>\$(y)</tt> and
144 \subsection sec-autovars Automatic variables
146 The following special symbols can appear inside scripts:
148 - <tt>$<</tt> expands to the first static prerequisite of the rule.
149 - <tt>$^</tt> expands to all the static prerequisites of the rule, including
151 - <tt>$\@</tt> expands to the first target of the rule.
152 - <tt>$*</tt> expands to the string that matched <tt>%</tt> in a generic rule.
153 - <tt>$$</tt> expands to a single dollar symbol.
155 Note: contrarily to <b>make</b>, there are no corresponding variables. For
156 instance, <tt>$^</tt> is not a shorthand for <tt>$(^)</tt>. Another
157 difference is that <tt>$\@</tt> is always the first target, not the one that
160 \subsection sec-functions Built-in functions
162 <b>remake</b> also supports a few built-in functions inspired from <b>make</b>.
164 - <tt>$(addprefix <i>prefix</i>, <i>list</i>)</tt> returns the list obtained
165 by prepending its first argument to each element of its second argument.
166 - <tt>$(addsuffix <i>suffix</i>, <i>list</i>)</tt> returns the list obtained
167 by appending its first argument to each element of its second argument.
169 \subsection sec-order Order-only prerequisites
171 If the static prerequisites of a rule contain a pipe symbol, prerequisites
172 on its right do not cause the targets to become obsolete if they are newer
173 (unless they are also dynamically registered as dependencies). They are
174 meant to be used when the targets do not directly depend on them, but the
175 computation of their dynamic dependencies does.
179 gcc -MMD -MF $@.d -o $@ -c $<
183 parser.c parser.h: parser.y
184 yacc -d -o parser.c parser.y
187 \subsection sec-special-var Special variables
189 Variable <tt>.OPTIONS</tt> is handled specially. Its content enables some
190 features of <b>remake</b> that are not enabled by default.
192 - <tt>variable-propagation</tt>: When a variable is set in the prerequisite
193 part of a rule, it is propagated to the rules of all the targets this rule
194 depends on. This option also enables variables to be set on the command
195 line. Note that, as in <b>make</b>, this features introduces non-determinism:
196 the content of some variables will depend on the build order.
198 \section sec-semantics Semantics
200 \subsection src-obsolete When are targets obsolete?
202 A target is obsolete:
204 - if there is no file corresponding to the target, or to one of its siblings
205 in a multi-target rule,
206 - if any of its dynamic prerequisites from a previous run or any of its static
207 prerequisites is obsolete,
208 - if the latest file corresponding to its siblings or itself is older than any
209 of its dynamic prerequisites or static prerequisites.
211 In all the other cases, it is assumed to be up-to-date (and so are all its
212 siblings). Note that the last rule above says "latest" and not "earliest". While
213 it might cause some obsolete targets to go unnoticed in corner cases, it allows
214 for the following kind of rules:
217 config.h stamp-config_h: config.h.in config.status
218 ./config.status config.h
222 A <tt>config.status</tt> file generally does not update header files (here
223 <tt>config.h</tt>) if they would not change. As a consequence, if not for the
224 <tt>stamp-config_h</tt> file above, a header would always be considered obsolete
225 once one of its prerequisites is modified. Note that touching <tt>config.h</tt>
226 rather than <tt>stamp-config_h</tt> would defeat the point of not updating it
227 in the first place, since the program files would need to be rebuilt.
229 Once all the static prerequisites of a target have been rebuilt, <b>remake</b>
230 checks whether the target still needs to be built. If it was obsolete only
231 because its prerequisites needed to be rebuilt and none of them changed, the
232 target is assumed to be up-to-date.
234 \subsection sec-rules How are targets (re)built?
236 There are two kinds of rules. If any of the targets or prerequisites contains
237 a <tt>%</tt> character, the rule is said to be <em>generic</em>. All the
238 targets of the rule shall then contain a single <tt>%</tt> character. All the
239 other rules are said to be <em>specific</em>.
241 A rule is said to <em>match</em> a given target:
243 - if it is specific and the target appears inside its target list,
244 - if it is generic and there is a way to replace the <tt>%</tt> character
245 from one of its targets so that it matches the given target.
247 When <b>remake</b> tries to build a given target, it looks for a specific rule
248 that matches it. If there is one and its script is nonempty, it uses it to
251 Otherwise, it looks for a generic rule that matches the target. If there are
252 several matching rules, it chooses the one with the shortest pattern (and if
253 there are several ones, the earliest one). <b>remake</b> then looks for
254 specific rules that match each target of the generic rule. All the
255 prerequisites of these specific rules are added to those of the generic rule.
256 The script of the generic rule is used to build the target.
262 commands building t%1 and t2%
265 commands building t2z
269 # t2x is built by the first rule (which also builds tx1) and its prerequisites are p1, px2
270 # t2y is built by the first rule (which also builds ty1) and its prerequisites are p1, py2, p3
271 # t2z is built by the second rule and its prerequisite is p4
274 The set of rules from <b>Remakefile</b> is ill-formed:
276 - if any specific rule matching a target of the generic rule has a nonempty script,
277 - if any target of the generic rule is matched by a generic rule with a shorter pattern.
279 \section sec-compilation Compilation
281 - On Linux, MacOSX, and BSD: <tt>g++ -o remake remake.cpp</tt>
282 - On Windows: <tt>g++ -o remake.exe remake.cpp -lws2_32</tt>
284 Installing <b>remake</b> is needed only if <b>Remakefile</b> does not
285 specify the path to the executable for its recursive calls. Thanks to its
286 single source file, <b>remake</b> can be shipped inside other packages and
287 built at configuration time.
289 \section sec-differences Differences with other build systems
291 Differences with <b>make</b>:
293 - Dynamic dependencies are supported.
294 - For rules with multiple targets, the shell script is executed only once
295 and is assumed to build all the targets. There is no need for
296 convoluted rules that are robust enough for parallel builds. For generic
297 rules, this is similar to the behavior of pattern rules from <b>gmake</b>.
298 - As with <b>redo</b>, only one shell is run when executing a script,
299 rather than one per script line. Note that the shells are run with
300 option <tt>-e</tt>, thus causing them to exit as soon as an error is
302 - The prerequisites of generic rules (known as implicit rules in make lingo)
303 are not used to decide between several of them. <b>remake</b> does not
304 select one for which it could satisfy the dependencies.
305 - Variables and built-in functions are expanded as they are encountered
306 during <b>Remakefile</b> parsing.
307 - Target-specific variables are not propagated, unless specifically enabled,
308 since this causes non-deterministic builds. This is the same for variables
309 set on the command line.
311 Differences with <b>redo</b>:
313 - As with <b>make</b>, it is possible to write the following kind of rules
316 Remakefile: Remakefile.in ./config.status
317 ./config.status Remakefile
319 - If a target is already built the first time <b>remake</b> runs, it still
320 uses the static prerequisites of rules mentioning it to check whether it
321 needs to be rebuilt. It does not assume it to be up-to-date. As with
322 <b>redo</b> though, if its obsolete status would be due to a dynamic
323 prerequisite, it will go unnoticed; it should be removed beforehand.
324 - Multiple targets are supported.
325 - <b>remake</b> has almost no features: no checksum-based dependencies, no
326 compatibility with job servers, etc.
328 \section sec-limitations Limitations
330 - If a rule script calls <b>remake</b>, the current working directory should
331 be the directory containing <b>Remakefile</b> (or the working directory
332 from the original <b>remake</b> if it was called with option <b>-f</b>).
333 - As with <b>make</b>, variables passed on the command line should keep
334 the same values, to ensure deterministic builds.
335 - Some cases of ill-formed rules are not caught by <b>remake</b> and can
336 thus lead to unpredictable behaviors.
338 \section sec-links Links
340 @see http://cr.yp.to/redo.html for the philosophy of <b>redo</b> and
341 https://github.com/apenwarr/redo for an implementation and some comprehensive documentation.
343 \section sec-licensing Licensing
345 @author Guillaume Melquiond
349 This program is free software: you can redistribute it and/or modify
350 it under the terms of the GNU General Public License as published by
351 the Free Software Foundation, either version 3 of the License, or
352 (at your option) any later version.
354 This program is distributed in the hope that it will be useful,
355 but WITHOUT ANY WARRANTY; without even the implied warranty of
356 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
357 GNU General Public License for more details.
359 \section sec-internals Internals
361 The parent <b>remake</b> process acts as a server. The other ones have a
362 REMAKE_SOCKET environment variable that tells them how to contact the
363 server. They send the content of the REMAKE_JOB_ID environment variable,
364 so that the server can associate the child targets to the jobs that
365 spawned them. They then wait for completion and exit with the status
366 returned by the server. This is handled by #client_mode.
368 The server calls #load_dependencies and #save_dependencies to serialize
369 dynamic dependencies from <b>.remake</b>. It loads <b>Remakefile</b> with
370 #load_rules. It then runs #server_mode, which calls #server_loop.
372 When building a target, the following sequence of events happens:
374 - #start calls #find_rule (and #find_generic_rule) to get the rule.
375 - It then creates a pseudo-client if the rule has static dependencies, or
376 calls #run_script otherwise. In both cases, a new job is created and its
377 targets are put into #job_targets.
378 - #run_script creates a shell process and stores it in #job_pids. It
379 increases #running_jobs.
380 - The child process possibly calls <b>remake</b> with a list of targets.
381 - #accept_client receives a build request from a child process and adds
382 it to #clients. It also records the new dependencies of the job into
383 #dependencies. It increases #waiting_jobs.
384 - #handle_clients uses #get_status to look up the obsoleteness of the
386 - Once the targets of a request have been built or one of them has failed,
387 #handle_clients calls #complete_request and removes the request from
389 - If the build targets come from a pseudo-client, #complete_request calls
390 #run_script. Otherwise it sends the reply to the corresponding child
391 process and decreases #waiting_jobs.
392 - When a child process ends, #server_loop calls #finalize_job, which
393 removes the process from #job_pids, decreases #running_jobs, and calls
395 - #complete_job removes the job from #job_targets and calls #update_status
396 to change the status of the targets. It also removes the target files in
401 #define WIN32_LEAN_AND_MEAN
421 #include <sys/stat.h>
422 #include <sys/types.h>
435 #include <winsock2.h>
437 typedef SOCKET socket_t;
439 #include <sys/socket.h>
441 #include <sys/wait.h>
442 typedef int socket_t;
443 enum { INVALID_SOCKET = -1 };
444 extern char **environ;
447 #if defined(WINDOWS) || defined(MACOSX)
448 enum { MSG_NOSIGNAL = 0 };
451 typedef std::list<std::string> string_list;
453 typedef std::set<std::string> string_set;
456 * Reference-counted shared object.
457 * @note The default constructor delays the creation of the object until it
458 * is first dereferenced.
468 content(T const &t): cnt(1), val(t) {}
470 mutable content *ptr;
471 ref_ptr(): ptr(NULL) {}
472 ref_ptr(T const &t): ptr(new content(t)) {}
473 ref_ptr(ref_ptr const &p): ptr(p.ptr) { if (ptr) ++ptr->cnt; }
474 ~ref_ptr() { if (ptr && --ptr->cnt == 0) delete ptr; }
475 ref_ptr &operator=(ref_ptr const &p)
477 if (ptr == p.ptr) return *this;
478 if (ptr && --ptr->cnt == 0) delete ptr;
485 if (!ptr) ptr = new content;
488 T *operator->() const { return &**this; }
497 typedef std::map<std::string, ref_ptr<dependency_t> > dependency_map;
499 typedef std::map<std::string, string_list> variable_map;
502 * Build status of a target.
506 Uptodate, ///< Target is up-to-date.
507 Todo, ///< Target is missing or obsolete.
508 Recheck, ///< Target has an obsolete dependency.
509 Running, ///< Target is being rebuilt.
510 Remade, ///< Target was successfully rebuilt.
511 Failed ///< Build failed for target.
515 * Build status of a target.
519 status_e status; ///< Actual status.
520 time_t last; ///< Last-modified date.
523 typedef std::map<std::string, status_t> status_map;
526 * Delayed assignment to a variable.
534 typedef std::map<std::string, assign_t> assign_map;
537 * A rule loaded from Remakefile.
541 string_list targets; ///< Files produced by this rule.
542 string_list deps; ///< Dependencies used for an implicit call to remake at the start of the script.
543 string_list wdeps; ///< Like #deps, except that they are not registered as dependencies.
544 assign_map assigns; ///< Assignment of variables.
545 std::string script; ///< Shell script for building the targets.
548 typedef std::list<rule_t> rule_list;
550 typedef std::map<std::string, ref_ptr<rule_t> > rule_map;
553 * A job created from a set of rules.
558 rule_t rule; ///< Original rule.
559 std::string stem; ///< Pattern used to instantiate the generic rule, if any.
560 variable_map vars; ///< Values of local variables.
563 typedef std::map<int, job_t> job_map;
565 typedef std::map<pid_t, int> pid_job_map;
568 * Client waiting for a request to complete.
570 * There are two kinds of clients:
571 * - real clients, which are instances of remake created by built scripts,
572 * - pseudo clients, which are created by the server to build specific targets.
574 * Among pseudo clients, there are two categories:
575 * - original clients, which are created for the targets passed on the
576 * command line by the user or for the initial regeneration of the rule file,
577 * - dependency clients, which are created to handle rules that have
578 * explicit dependencies and thus to emulate a call to remake.
582 socket_t socket; ///< Socket used to reply to the client (invalid for pseudo clients).
583 int job_id; ///< Job for which the built script called remake and spawned the client (negative for original clients).
584 bool failed; ///< Whether some targets failed in mode -k.
585 string_list pending; ///< Targets not yet started.
586 string_set running; ///< Targets being built.
587 variable_map vars; ///< Variables set on request.
588 bool delayed; ///< Whether it is a dependency client and a script has to be started on request completion.
589 client_t(): socket(INVALID_SOCKET), job_id(-1), failed(false), delayed(false) {}
592 typedef std::list<client_t> client_list;
595 * Map from variable names to their content.
596 * Initialized with the values passed on the command line.
598 static variable_map variables;
601 * Map from targets to their known dependencies.
603 static dependency_map dependencies;
606 * Map from targets to their build status.
608 static status_map status;
611 * Set of generic rules loaded from Remakefile.
613 static rule_list generic_rules;
616 * Map from targets to specific rules loaded from Remakefile.
618 static rule_map specific_rules;
621 * Map of jobs being built.
626 * Map from jobs to shell pids.
628 static pid_job_map job_pids;
631 * List of clients waiting for a request to complete.
632 * New clients are put to front, so that the build process is depth-first.
634 static client_list clients;
637 * Maximum number of parallel jobs (non-positive if unbounded).
638 * Can be modified by the -j option.
640 static int max_active_jobs = 1;
643 * Whether to keep building targets in case of failure.
644 * Can be modified by the -k option.
646 static bool keep_going = false;
649 * Number of jobs currently running:
650 * - it increases when a process is created in #run_script,
651 * - it decreases when a completion message is received in #finalize_job.
653 * @note There might be some jobs running while #clients is empty.
654 * Indeed, if a client requested two targets to be rebuilt, if they
655 * are running concurrently, if one of them fails, the client will
656 * get a failure notice and might terminate before the other target
659 static int running_jobs = 0;
662 * Number of jobs currently waiting for a build request to finish:
663 * - it increases when a build request is received in #accept_client
664 * (since the client is presumably waiting for the reply),
665 * - it decreases when a reply is sent in #complete_request.
667 static int waiting_jobs = 0;
670 * Global counter used to produce increasing job numbers.
673 static int job_counter = 0;
676 * Socket on which the server listens for client request.
678 static socket_t socket_fd;
681 * Whether the request of an original client failed.
683 static bool build_failure;
687 * Name of the server socket in the file system.
689 static char *socket_name;
693 * Name of the first target of the first specific rule, used for default run.
695 static std::string first_target;
698 * Whether a short message should be displayed for each target.
700 static bool show_targets = true;
703 * Whether script commands are echoed.
705 static bool echo_scripts = false;
708 * Time at the start of the program.
710 static time_t now = time(NULL);
713 * Directory with respect to which command-line names are relative.
715 static std::string working_dir;
718 * Directory with respect to which targets are relative.
720 static std::string prefix_dir;
723 * Whether target-specific variables are propagated to prerequisites.
725 static bool propagate_vars = false;
728 static volatile sig_atomic_t got_SIGCHLD = 0;
730 static void sigchld_handler(int)
735 static void sigint_handler(int)
737 // Child processes will receive the signal too, so just prevent
738 // new jobs from starting and wait for the running jobs to fail.
747 log(): active(false), open(false), depth(0)
750 std::ostream &operator()()
752 if (open) std::cerr << std::endl;
754 std::cerr << std::string(depth * 2, ' ');
758 std::ostream &operator()(bool o)
760 if (o && open) std::cerr << std::endl;
763 if (o || !open) std::cerr << std::string(depth * 2, ' ');
772 struct log_auto_close
775 log_auto_close(): still_open(true)
780 if (debug.active && still_open) debug(false) << "done\n";
784 #define DEBUG if (debug.active) debug()
785 #define DEBUG_open log_auto_close auto_close; if (debug.active) debug(true)
786 #define DEBUG_close if ((auto_close.still_open = false), debug.active) debug(false)
789 * Strong typedef for strings that need escaping.
790 * @note The string is stored as a reference, so the constructed object is
791 * meant to be immediately consumed.
795 std::string const &input;
796 escape_string(std::string const &s): input(s) {}
800 * Write the string in @a se to @a out if it does not contain any special
801 * characters, a quoted and escaped string otherwise.
803 static std::ostream &operator<<(std::ostream &out, escape_string const &se)
805 std::string const &s = se.input;
806 char const *quoted_char = ",: '";
807 char const *escaped_char = "\"\\$!";
808 bool need_quotes = false;
810 size_t len = s.length(), last = 0, j = 0;
811 for (size_t i = 0; i < len; ++i)
813 if (strchr(escaped_char, s[i]))
816 if (!buf) buf = new char[len * 2];
817 memcpy(&buf[j], &s[last], i - last);
823 if (!need_quotes && strchr(quoted_char, s[i]))
826 if (!need_quotes) return out << s;
828 if (!buf) return out << s << '"';
830 out.write(&s[last], len - last);
836 * @defgroup paths Path helpers
842 * Initialize #working_dir.
844 static void init_working_dir()
847 char *res = getcwd(buf, sizeof(buf));
850 perror("Failed to get working directory");
855 for (size_t i = 0, l = working_dir.size(); i != l; ++i)
857 if (working_dir[i] == '\\') working_dir[i] = '/';
860 prefix_dir = working_dir;
864 * Initialize #prefix_dir and switch to it.
866 static void init_prefix_dir()
871 if (stat((prefix_dir + "/Remakefile").c_str(), &s) == 0)
873 chdir(prefix_dir.c_str());
876 size_t pos = prefix_dir.find_last_of('/');
877 if (pos == std::string::npos)
879 std::cerr << "Failed to locate Remakefile in the current directory or one of its parents" << std::endl;
882 prefix_dir.erase(pos);
887 * Normalize an absolute path with respect to @a p.
888 * Paths outside the subtree are left unchanged.
890 static std::string normalize_abs(std::string const &s, std::string const &p)
892 size_t l = p.length();
893 if (s.compare(0, l, p)) return s;
894 size_t ll = s.length();
895 if (ll == l) return ".";
898 size_t pos = s.rfind('/', l);
899 assert(pos != std::string::npos);
900 return s.substr(pos + 1);
902 if (ll == l + 1) return ".";
903 return s.substr(l + 1);
907 * Normalize path @a s (possibly relative to @a w) with respect to @a p.
909 * - If both @a p and @a w are empty, the function just removes ".", "..", "//".
910 * - If only @a p is empty, the function returns an absolute path.
912 static std::string normalize(std::string const &s, std::string const &w, std::string const &p)
915 char const *delim = "/\\";
919 size_t pos = s.find_first_of(delim);
920 if (pos == std::string::npos && w == p) return s;
921 bool absolute = pos == 0;
922 if (!absolute && w != p && !w.empty())
923 return normalize(w + '/' + s, w, p);
924 size_t prev = 0, len = s.length();
930 std::string n = s.substr(prev, pos - prev);
933 if (!l.empty()) l.pop_back();
934 else if (!absolute && !w.empty())
935 return normalize(w + '/' + s, w, p);
941 if (pos >= len) break;
943 pos = s.find_first_of(delim, prev);
944 if (pos == std::string::npos) pos = len;
946 string_list::const_iterator i = l.begin(), i_end = l.end();
947 if (i == i_end) return absolute ? "/" : ".";
949 if (absolute) n.push_back('/');
951 for (++i; i != i_end; ++i)
956 if (absolute && !p.empty()) return normalize_abs(n, p);
961 * Normalize the content of a list of targets.
963 static void normalize_list(string_list &l, std::string const &w, std::string const &p)
965 for (string_list::iterator i = l.begin(),
966 i_end = l.end(); i != i_end; ++i)
968 *i = normalize(*i, w, p);
975 * @defgroup lexer Lexer
983 static void skip_spaces(std::istream &in)
986 while (strchr(" \t", (c = in.get()))) {}
987 if (in.good()) in.putback(c);
993 static void skip_empty(std::istream &in)
996 while (strchr("\r\n", (c = in.get()))) {}
997 if (in.good()) in.putback(c);
1001 * Skip end of line. If @a multi is true, skip the following empty lines too.
1002 * @return true if there was a line to end.
1004 static bool skip_eol(std::istream &in, bool multi = false)
1007 if (c == '\r') c = in.get();
1008 if (c != '\n' && in.good()) in.putback(c);
1009 if (c != '\n' && !in.eof()) return false;
1010 if (multi) skip_empty(in);
1028 * Skip spaces and peek at the next token.
1029 * If it is one of @a mask, skip it (if it is not Word) and return it.
1030 * @note For composite tokens allowed by @a mask, input characters might
1031 * have been eaten even for an Unexpected result.
1033 static int expect_token(std::istream &in, int mask)
1039 if (!in.good()) return Unexpected;
1044 case '\n': return Unexpected;
1045 case ':': tok = Colon; break;
1046 case ',': tok = Comma; break;
1047 case '=': tok = Equal; break;
1048 case ')': tok = Rightpar; break;
1049 case '|': tok = Pipe; break;
1051 if (!(mask & Dollarpar)) return Unexpected;
1054 if (in.peek() != '(') return Unexpected;
1057 if (!(mask & Plusequal)) return Unexpected;
1060 if (in.peek() != '=') return Unexpected;
1064 if (skip_eol(in)) continue;
1066 return mask & Word ? Word : Unexpected;
1068 return mask & Word ? Word : Unexpected;
1070 if (!(tok & mask)) return Unexpected;
1077 * Read a (possibly quoted) word.
1079 static std::string read_word(std::istream &in)
1083 if (!in.good()) return res;
1084 char const *separators = " \t\r\n:$(),=+\"";
1085 bool quoted = c == '"';
1088 if (strchr(separators, c))
1098 if (!in.good()) return res;
1110 if (strchr(separators, c))
1123 * @defgroup stream Token streams
1129 * Possible results from word producers.
1139 * Interface for word producers.
1143 virtual ~generator() {}
1144 virtual input_status next(std::string &) = 0;
1148 * Generator for the words of a variable.
1150 struct variable_generator: generator
1153 string_list::const_iterator vcur, vend;
1154 variable_generator(std::string const &, variable_map const *);
1155 input_status next(std::string &);
1158 variable_generator::variable_generator(std::string const &n,
1159 variable_map const *local_variables): name(n)
1161 if (local_variables)
1163 variable_map::const_iterator i = local_variables->find(name);
1164 if (i != local_variables->end())
1166 vcur = i->second.begin();
1167 vend = i->second.end();
1171 variable_map::const_iterator i = variables.find(name);
1172 if (i == variables.end()) return;
1173 vcur = i->second.begin();
1174 vend = i->second.end();
1177 input_status variable_generator::next(std::string &res)
1189 * Generator for the words of an input stream.
1191 struct input_generator
1195 variable_map const *local_variables;
1196 bool earliest_exit, done;
1197 input_generator(std::istream &i, variable_map const *lv, bool e = false)
1198 : in(i), nested(NULL), local_variables(lv), earliest_exit(e), done(false) {}
1199 input_status next(std::string &);
1200 ~input_generator() { assert(!nested); }
1203 static generator *get_function(input_generator const &, std::string const &);
1205 input_status input_generator::next(std::string &res)
1210 input_status s = nested->next(res);
1211 if (s == Success) return Success;
1214 if (s == SyntaxError) return SyntaxError;
1216 if (done) return Eof;
1217 if (earliest_exit) done = true;
1218 switch (expect_token(in, Word | Dollarpar))
1221 res = read_word(in);
1225 std::string name = read_word(in);
1226 if (name.empty()) return SyntaxError;
1227 if (expect_token(in, Rightpar))
1228 nested = new variable_generator(name, local_variables);
1231 nested = get_function(*this, name);
1232 if (!nested) return SyntaxError;
1242 * Read a list of words from an input generator.
1243 * @return false if a syntax error was encountered.
1245 static bool read_words(input_generator &in, string_list &res)
1249 res.push_back(std::string());
1250 input_status s = in.next(res.back());
1251 if (s == Success) continue;
1257 static bool read_words(std::istream &in, string_list &res)
1259 input_generator gen(in, NULL);
1260 return read_words(gen, res);
1264 * Generator for the result of function addprefix.
1266 struct addprefix_generator: generator
1268 input_generator gen;
1270 string_list::const_iterator prei;
1273 addprefix_generator(input_generator const &, bool &);
1274 input_status next(std::string &);
1277 addprefix_generator::addprefix_generator(input_generator const &top, bool &ok)
1278 : gen(top.in, top.local_variables)
1280 if (!read_words(gen, pre)) return;
1281 if (!expect_token(gen.in, Comma)) return;
1287 input_status addprefix_generator::next(std::string &res)
1304 switch (gen.next(res))
1307 if (!prel) return Success;
1313 return expect_token(gen.in, Rightpar) ? Eof : SyntaxError;
1320 * Generator for the result of function addsuffix.
1322 struct addsuffix_generator: generator
1324 input_generator gen;
1326 string_list::const_iterator sufi;
1329 addsuffix_generator(input_generator const &, bool &);
1330 input_status next(std::string &);
1333 addsuffix_generator::addsuffix_generator(input_generator const &top, bool &ok)
1334 : gen(top.in, top.local_variables)
1336 if (!read_words(gen, suf)) return;
1337 if (!expect_token(gen.in, Comma)) return;
1343 input_status addsuffix_generator::next(std::string &res)
1355 switch (gen.next(res))
1358 if (!sufl) return Success;
1364 return expect_token(gen.in, Rightpar) ? Eof : SyntaxError;
1371 * Return a generator for function @a name.
1373 static generator *get_function(input_generator const &in, std::string const &name)
1376 generator *g = NULL;
1378 if (name == "addprefix") g = new addprefix_generator(in, ok);
1379 else if (name == "addsuffix") g = new addsuffix_generator(in, ok);
1380 if (!g || ok) return g;
1388 * @defgroup database Dependency database
1394 * Load dependencies from @a in.
1396 static void load_dependencies(std::istream &in)
1401 std::cerr << "Failed to load database" << std::endl;
1407 string_list targets;
1408 if (!read_words(in, targets)) goto error;
1409 if (in.eof()) return;
1410 if (targets.empty()) goto error;
1411 DEBUG << "reading dependencies of target " << targets.front() << std::endl;
1412 if (in.get() != ':') goto error;
1413 ref_ptr<dependency_t> dep;
1414 dep->targets = targets;
1416 if (!read_words(in, deps)) goto error;
1417 dep->deps.insert(deps.begin(), deps.end());
1418 for (string_list::const_iterator i = targets.begin(),
1419 i_end = targets.end(); i != i_end; ++i)
1421 dependencies[*i] = dep;
1428 * Load known dependencies from file <tt>.remake</tt>.
1430 static void load_dependencies()
1432 DEBUG_open << "Loading database... ";
1433 std::ifstream in(".remake");
1436 DEBUG_close << "not found\n";
1439 load_dependencies(in);
1444 * Save all the dependencies in file <tt>.remake</tt>.
1446 static void save_dependencies()
1448 DEBUG_open << "Saving database... ";
1449 std::ofstream db(".remake");
1450 while (!dependencies.empty())
1452 ref_ptr<dependency_t> dep = dependencies.begin()->second;
1453 for (string_list::const_iterator i = dep->targets.begin(),
1454 i_end = dep->targets.end(); i != i_end; ++i)
1456 db << escape_string(*i) << ' ';
1457 dependencies.erase(*i);
1460 for (string_set::const_iterator i = dep->deps.begin(),
1461 i_end = dep->deps.end(); i != i_end; ++i)
1463 db << ' ' << escape_string(*i);
1471 static void merge_rule(rule_t &dest, rule_t const &src);
1474 * @defgroup parser Rule parser
1480 * Register a specific rule with an empty script:
1482 * - Check that none of the targets already has an associated rule with a
1484 * - Create a new rule with a single target for each target, if needed.
1485 * - Add the prerequisites of @a rule to all these associated rules.
1487 static void register_transparent_rule(rule_t const &rule, string_list const &targets)
1489 assert(rule.script.empty());
1490 for (string_list::const_iterator i = targets.begin(),
1491 i_end = targets.end(); i != i_end; ++i)
1493 std::pair<rule_map::iterator, bool> j =
1494 specific_rules.insert(std::make_pair(*i, ref_ptr<rule_t>()));
1495 ref_ptr<rule_t> &r = j.first->second;
1498 r = ref_ptr<rule_t>(rule);
1499 r->targets = string_list(1, *i);
1502 if (!r->script.empty())
1504 std::cerr << "Failed to load rules: " << *i
1505 << " cannot be the target of several rules" << std::endl;
1508 assert(r->targets.size() == 1 && r->targets.front() == *i);
1509 merge_rule(*r, rule);
1512 for (string_list::const_iterator i = targets.begin(),
1513 i_end = targets.end(); i != i_end; ++i)
1515 ref_ptr<dependency_t> &dep = dependencies[*i];
1516 if (dep->targets.empty()) dep->targets.push_back(*i);
1517 dep->deps.insert(rule.deps.begin(), rule.deps.end());
1522 * Register a specific rule with a nonempty script:
1524 * - Check that none of the targets already has an associated rule.
1525 * - Create a single shared rule and associate it to all the targets.
1526 * - Merge the prerequisites of all the targets into a single set and
1527 * add the prerequisites of the rule to it. (The preexisting
1528 * prerequisites, if any, come from a previous run.)
1530 static void register_scripted_rule(rule_t const &rule)
1532 ref_ptr<rule_t> r(rule);
1533 for (string_list::const_iterator i = rule.targets.begin(),
1534 i_end = rule.targets.end(); i != i_end; ++i)
1536 std::pair<rule_map::iterator, bool> j =
1537 specific_rules.insert(std::make_pair(*i, r));
1538 if (j.second) continue;
1539 std::cerr << "Failed to load rules: " << *i
1540 << " cannot be the target of several rules" << std::endl;
1544 ref_ptr<dependency_t> dep;
1545 dep->targets = rule.targets;
1546 dep->deps.insert(rule.deps.begin(), rule.deps.end());
1547 for (string_list::const_iterator i = rule.targets.begin(),
1548 i_end = rule.targets.end(); i != i_end; ++i)
1550 ref_ptr<dependency_t> &d = dependencies[*i];
1551 dep->deps.insert(d->deps.begin(), d->deps.end());
1557 * Read a rule starting with target @a first, if nonempty.
1558 * Store into #generic_rules or #specific_rules depending on its genericity.
1560 static void load_rule(std::istream &in, std::string const &first)
1562 DEBUG_open << "Reading rule for target " << first << "... ";
1566 DEBUG_close << "failed\n";
1567 std::cerr << "Failed to load rules: syntax error" << std::endl;
1572 // Read targets and check genericity.
1573 string_list targets;
1574 if (!read_words(in, targets)) goto error;
1575 if (!first.empty()) targets.push_front(first);
1576 else if (targets.empty()) goto error;
1577 else DEBUG << "actual target: " << targets.front() << std::endl;
1578 bool generic = false;
1579 normalize_list(targets, "", "");
1580 for (string_list::const_iterator i = targets.begin(),
1581 i_end = targets.end(); i != i_end; ++i)
1583 if (i->empty()) goto error;
1584 if ((i->find('%') != std::string::npos) != generic)
1586 if (i == targets.begin()) generic = true;
1590 std::swap(rule.targets, targets);
1592 if (in.get() != ':') goto error;
1594 bool assignment = false;
1596 // Read dependencies.
1599 if (expect_token(in, Word))
1601 std::string d = read_word(in);
1602 if (int tok = expect_token(in, Equal | Plusequal))
1604 if (!read_words(in, v)) goto error;
1605 assign_t &a = rule.assigns[d];
1606 a.append = tok == Plusequal;
1614 if (!read_words(in, v)) goto error;
1615 normalize_list(v, "", "");
1618 if (expect_token(in, Pipe))
1620 if (!read_words(in, v)) goto error;
1621 normalize_list(v, "", "");
1628 if (!skip_eol(in, true)) goto error;
1631 std::ostringstream buf;
1635 if (!in.good()) break;
1636 if (c == '\t' || c == ' ')
1638 in.get(*buf.rdbuf());
1639 if (in.fail() && !in.eof()) in.clear();
1641 else if (c == '\r' || c == '\n')
1649 rule.script = buf.str();
1651 // Add generic rules to the correct set.
1654 if (assignment) goto error;
1655 generic_rules.push_back(rule);
1659 if (!rule.script.empty())
1661 if (assignment) goto error;
1662 register_scripted_rule(rule);
1666 // Swap away the targets to avoid costly copies when registering.
1667 string_list targets;
1668 std::swap(rule.targets, targets);
1669 register_transparent_rule(rule, targets);
1670 std::swap(rule.targets, targets);
1673 // If there is no default target yet, mark it as such.
1674 if (first_target.empty())
1675 first_target = rule.targets.front();
1679 * Load rules from @a remakefile.
1680 * If some rules have dependencies and non-generic targets, add these
1681 * dependencies to the targets.
1683 static void load_rules(std::string const &remakefile)
1685 DEBUG_open << "Loading rules... ";
1689 std::cerr << "Failed to load rules: syntax error" << std::endl;
1692 std::ifstream in(remakefile.c_str());
1695 std::cerr << "Failed to load rules: no Remakefile found" << std::endl;
1700 string_list options;
1708 while (in.get() != '\n') {}
1712 if (c == ' ' || c == '\t') goto error;
1713 if (expect_token(in, Word))
1715 std::string name = read_word(in);
1716 if (name.empty()) goto error;
1717 if (int tok = expect_token(in, Equal | Plusequal))
1719 DEBUG << "Assignment to variable " << name << std::endl;
1721 if (!read_words(in, value)) goto error;
1723 *(name == ".OPTIONS" ? &options : &variables[name]);
1724 if (tok == Equal) dest.swap(value);
1725 else dest.splice(dest.end(), value);
1726 if (!skip_eol(in, true)) goto error;
1728 else load_rule(in, name);
1730 else load_rule(in, std::string());
1733 // Set actual options.
1734 for (string_list::const_iterator i = options.begin(),
1735 i_end = options.end(); i != i_end; ++i)
1737 if (*i == "variable-propagation") propagate_vars = true;
1740 std::cerr << "Failed to load rules: unrecognized option" << std::endl;
1749 * @defgroup rules Rule resolution
1754 static void merge_rule(rule_t &dest, rule_t const &src)
1756 dest.deps.insert(dest.deps.end(), src.deps.begin(), src.deps.end());
1757 dest.wdeps.insert(dest.wdeps.end(), src.wdeps.begin(), src.wdeps.end());
1758 for (assign_map::const_iterator i = src.assigns.begin(),
1759 i_end = src.assigns.end(); i != i_end; ++i)
1761 if (!i->second.append)
1764 dest.assigns[i->first] = i->second;
1767 assign_map::iterator j = dest.assigns.find(i->first);
1768 if (j == dest.assigns.end()) goto new_assign;
1769 j->second.value.insert(j->second.value.end(),
1770 i->second.value.begin(), i->second.value.end());
1775 * Substitute a pattern into a list of strings.
1777 static void substitute_pattern(std::string const &pat, string_list const &src, string_list &dst)
1779 for (string_list::const_iterator i = src.begin(),
1780 i_end = src.end(); i != i_end; ++i)
1782 size_t pos = i->find('%');
1783 if (pos == std::string::npos) dst.push_back(*i);
1784 else dst.push_back(i->substr(0, pos) + pat + i->substr(pos + 1));
1789 * Find a generic rule matching @a target:
1790 * - the one leading to shorter matches has priority,
1791 * - among equivalent rules, the earliest one has priority.
1793 static void find_generic_rule(job_t &job, std::string const &target)
1795 size_t tlen = target.length(), plen = tlen + 1;
1796 for (rule_list::const_iterator i = generic_rules.begin(),
1797 i_end = generic_rules.end(); i != i_end; ++i)
1799 for (string_list::const_iterator j = i->targets.begin(),
1800 j_end = i->targets.end(); j != j_end; ++j)
1802 size_t len = j->length();
1803 if (tlen < len) continue;
1804 if (plen <= tlen - (len - 1)) continue;
1805 size_t pos = j->find('%');
1806 if (pos == std::string::npos) continue;
1807 size_t len2 = len - (pos + 1);
1808 if (j->compare(0, pos, target, 0, pos) ||
1809 j->compare(pos + 1, len2, target, tlen - len2, len2))
1811 plen = tlen - (len - 1);
1812 job.stem = target.substr(pos, plen);
1813 job.rule = rule_t();
1814 job.rule.script = i->script;
1815 substitute_pattern(job.stem, i->targets, job.rule.targets);
1816 substitute_pattern(job.stem, i->deps, job.rule.deps);
1817 substitute_pattern(job.stem, i->wdeps, job.rule.wdeps);
1824 * Find a specific rule matching @a target. Return a generic one otherwise.
1825 * If there is both a specific rule with an empty script and a generic rule, the
1826 * generic one is returned after adding the dependencies of the specific one.
1828 static void find_rule(job_t &job, std::string const &target)
1830 rule_map::const_iterator i = specific_rules.find(target),
1831 i_end = specific_rules.end();
1832 // If there is a specific rule with a script, return it.
1833 if (i != i_end && !i->second->script.empty())
1835 job.rule = *i->second;
1838 find_generic_rule(job, target);
1839 // If there is no generic rule, return the specific rule (no script), if any.
1840 if (job.rule.targets.empty())
1844 job.rule = *i->second;
1848 // Optimize the lookup when there is only one target (already looked up).
1849 if (job.rule.targets.size() == 1)
1851 if (i == i_end) return;
1852 merge_rule(job.rule, *i->second);
1855 // Add the dependencies of the specific rules of every target to the
1856 // generic rule. If any of those rules has a nonempty script, error out.
1857 for (string_list::const_iterator j = job.rule.targets.begin(),
1858 j_end = job.rule.targets.end(); j != j_end; ++j)
1860 i = specific_rules.find(*j);
1861 if (i == i_end) continue;
1862 if (!i->second->script.empty()) return;
1863 merge_rule(job.rule, *i->second);
1870 * @defgroup status Target status
1876 * Compute and memoize the status of @a target:
1877 * - if the file does not exist, the target is obsolete,
1878 * - if any dependency is obsolete or younger than the file, it is obsolete,
1879 * - otherwise it is up-to-date.
1881 * @note For rules with multiple targets, all the targets share the same
1882 * status. (If one is obsolete, they all are.) The second rule above
1883 * is modified in that case: the latest target is chosen, not the oldest!
1885 static status_t const &get_status(std::string const &target)
1887 std::pair<status_map::iterator,bool> i =
1888 status.insert(std::make_pair(target, status_t()));
1889 status_t &ts = i.first->second;
1890 if (!i.second) return ts;
1891 DEBUG_open << "Checking status of " << target << "... ";
1892 dependency_map::const_iterator j = dependencies.find(target);
1893 if (j == dependencies.end())
1896 if (stat(target.c_str(), &s) != 0)
1898 DEBUG_close << "missing\n";
1903 DEBUG_close << "up-to-date\n";
1904 ts.status = Uptodate;
1905 ts.last = s.st_mtime;
1908 dependency_t const &dep = *j->second;
1909 status_e st = Uptodate;
1911 for (string_list::const_iterator k = dep.targets.begin(),
1912 k_end = dep.targets.end(); k != k_end; ++k)
1915 if (stat(k->c_str(), &s) != 0)
1917 if (st == Uptodate) DEBUG_close << *k << " missing\n";
1921 status[*k].last = s.st_mtime;
1922 if (s.st_mtime > latest) latest = s.st_mtime;
1924 if (st != Uptodate) goto update;
1925 for (string_set::const_iterator k = dep.deps.begin(),
1926 k_end = dep.deps.end(); k != k_end; ++k)
1928 status_t const &ts_ = get_status(*k);
1929 if (latest < ts_.last)
1931 DEBUG_close << "older than " << *k << std::endl;
1935 if (ts_.status != Uptodate && st != Recheck)
1937 DEBUG << "obsolete dependency " << *k << std::endl;
1941 if (st == Uptodate) DEBUG_close << "all siblings up-to-date\n";
1943 for (string_list::const_iterator k = dep.targets.begin(),
1944 k_end = dep.targets.end(); k != k_end; ++k)
1946 status[*k].status = st;
1952 * Change the status of @a target to #Remade or #Uptodate depending on whether
1953 * its modification time changed.
1955 static void update_status(std::string const &target)
1957 DEBUG_open << "Rechecking status of " << target << "... ";
1958 status_map::iterator i = status.find(target);
1959 assert(i != status.end());
1960 status_t &ts = i->second;
1964 DEBUG_close << "possibly remade\n";
1968 if (stat(target.c_str(), &s) != 0)
1970 DEBUG_close << "missing\n";
1973 else if (s.st_mtime != ts.last)
1975 DEBUG_close << "remade\n";
1976 ts.last = s.st_mtime;
1980 DEBUG_close << "unchanged\n";
1981 ts.status = Uptodate;
1986 * Check whether all the prerequisites of @a target ended being up-to-date.
1988 static bool still_need_rebuild(std::string const &target)
1990 DEBUG_open << "Rechecking obsoleteness of " << target << "... ";
1991 status_map::const_iterator i = status.find(target);
1992 assert(i != status.end());
1993 if (i->second.status != Recheck) return true;
1994 dependency_map::const_iterator j = dependencies.find(target);
1995 assert(j != dependencies.end());
1996 dependency_t const &dep = *j->second;
1997 for (string_set::const_iterator k = dep.deps.begin(),
1998 k_end = dep.deps.end(); k != k_end; ++k)
2000 if (status[*k].status != Uptodate) return true;
2002 for (string_list::const_iterator k = dep.targets.begin(),
2003 k_end = dep.targets.end(); k != k_end; ++k)
2005 status[*k].status = Uptodate;
2007 DEBUG_close << "no longer obsolete\n";
2014 * @defgroup server Server
2020 * Handle job completion.
2022 static void complete_job(int job_id, bool success)
2024 DEBUG_open << "Completing job " << job_id << "... ";
2025 job_map::iterator i = jobs.find(job_id);
2026 assert(i != jobs.end());
2027 string_list const &targets = i->second.rule.targets;
2030 for (string_list::const_iterator j = targets.begin(),
2031 j_end = targets.end(); j != j_end; ++j)
2038 DEBUG_close << "failed\n";
2039 std::cerr << "Failed to build";
2040 for (string_list::const_iterator j = targets.begin(),
2041 j_end = targets.end(); j != j_end; ++j)
2043 status[*j].status = Failed;
2044 std::cerr << ' ' << *j;
2047 std::cerr << std::endl;
2053 * Return the script obtained by substituting variables.
2055 static std::string prepare_script(job_t const &job)
2057 std::string const &s = job.rule.script;
2058 std::istringstream in(s);
2059 std::ostringstream out;
2060 size_t len = s.size();
2064 size_t pos = in.tellg(), p = s.find('$', pos);
2065 if (p == std::string::npos || p == len - 1) p = len;
2066 out.write(&s[pos], p - pos);
2067 if (p == len) break;
2076 if (!job.rule.deps.empty())
2077 out << job.rule.deps.front();
2083 for (string_list::const_iterator i = job.rule.deps.begin(),
2084 i_end = job.rule.deps.end(); i != i_end; ++i)
2086 if (first) first = false;
2094 assert(!job.rule.targets.empty());
2095 out << job.rule.targets.front();
2106 input_generator gen(in, &job.vars, true);
2110 input_status s = gen.next(w);
2111 if (s == SyntaxError)
2116 if (s == Eof) break;
2117 if (first) first = false;
2124 // Let dollars followed by an unrecognized character
2125 // go through. This differs from Make, which would
2126 // use a one-letter variable.
2136 * Execute the script from @a rule.
2138 static status_e run_script(int job_id, job_t const &job)
2142 std::cout << "Building";
2143 for (string_list::const_iterator i = job.rule.targets.begin(),
2144 i_end = job.rule.targets.end(); i != i_end; ++i)
2146 std::cout << ' ' << *i;
2148 std::cout << std::endl;
2151 ref_ptr<dependency_t> dep;
2152 dep->targets = job.rule.targets;
2153 dep->deps.insert(job.rule.deps.begin(), job.rule.deps.end());
2154 for (string_list::const_iterator i = job.rule.targets.begin(),
2155 i_end = job.rule.targets.end(); i != i_end; ++i)
2157 dependencies[*i] = dep;
2160 std::string script = prepare_script(job);
2162 std::ostringstream job_id_buf;
2163 job_id_buf << job_id;
2164 std::string job_id_ = job_id_buf.str();
2166 DEBUG_open << "Starting script for job " << job_id << "... ";
2169 DEBUG_close << "no script\n";
2170 complete_job(job_id, true);
2177 DEBUG_close << "failed\n";
2178 complete_job(job_id, false);
2187 CloseHandle(pfd[0]);
2188 CloseHandle(pfd[1]);
2191 if (!CreatePipe(&pfd[0], &pfd[1], NULL, 0))
2193 if (!SetHandleInformation(pfd[0], HANDLE_FLAG_INHERIT, HANDLE_FLAG_INHERIT))
2196 ZeroMemory(&si, sizeof(STARTUPINFO));
2197 si.cb = sizeof(STARTUPINFO);
2198 si.hStdError = GetStdHandle(STD_ERROR_HANDLE);
2199 si.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
2200 si.hStdInput = pfd[0];
2201 si.dwFlags |= STARTF_USESTDHANDLES;
2202 PROCESS_INFORMATION pi;
2203 ZeroMemory(&pi, sizeof(PROCESS_INFORMATION));
2204 if (!SetEnvironmentVariable("REMAKE_JOB_ID", job_id_.c_str()))
2206 char const *argv = echo_scripts ? "SH.EXE -e -s -v" : "SH.EXE -e -s";
2207 if (!CreateProcess(NULL, (char *)argv, NULL, NULL,
2208 true, 0, NULL, NULL, &si, &pi))
2212 CloseHandle(pi.hThread);
2213 DWORD len = script.length(), wlen;
2214 if (!WriteFile(pfd[1], script.c_str(), len, &wlen, NULL) || wlen < len)
2215 std::cerr << "Unexpected failure while sending script to shell" << std::endl;
2216 CloseHandle(pfd[0]);
2217 CloseHandle(pfd[1]);
2219 job_pids[pi.hProcess] = job_id;
2230 if (pipe(pfd) == -1)
2232 if (setenv("REMAKE_JOB_ID", job_id_.c_str(), 1))
2234 if (pid_t pid = vfork())
2236 if (pid == -1) goto error2;
2237 ssize_t len = script.length();
2238 if (write(pfd[1], script.c_str(), len) < len)
2239 std::cerr << "Unexpected failure while sending script to shell" << std::endl;
2243 job_pids[pid] = job_id;
2246 // Child process starts here. Notice the use of vfork above.
2247 char const *argv[5] = { "sh", "-e", "-s", NULL, NULL };
2248 if (echo_scripts) argv[3] = "-v";
2255 execve("/bin/sh", (char **)argv, environ);
2256 _exit(EXIT_FAILURE);
2261 * Create a job for @a target according to the loaded rules.
2262 * Mark all the targets from the rule as running and reset their dependencies.
2263 * Inherit variables from @a current, if enabled.
2264 * If the rule has dependencies, create a new client to build them just
2265 * before @a current, and change @a current so that it points to it.
2267 static status_e start(std::string const &target, client_list::iterator ¤t)
2269 int job_id = job_counter++;
2270 DEBUG_open << "Starting job " << job_id << " for " << target << "... ";
2271 job_t &job = jobs[job_id];
2272 find_rule(job, target);
2273 if (job.rule.targets.empty())
2275 status[target].status = Failed;
2276 DEBUG_close << "failed\n";
2277 std::cerr << "No rule for building " << target << std::endl;
2280 for (string_list::const_iterator i = job.rule.targets.begin(),
2281 i_end = job.rule.targets.end(); i != i_end; ++i)
2283 status[*i].status = Running;
2285 if (propagate_vars) job.vars = current->vars;
2286 for (assign_map::const_iterator i = job.rule.assigns.begin(),
2287 i_end = job.rule.assigns.end(); i != i_end; ++i)
2289 std::pair<variable_map::iterator, bool> k =
2290 job.vars.insert(std::make_pair(i->first, string_list()));
2291 string_list &v = k.first->second;
2292 if (i->second.append)
2296 variable_map::const_iterator j = variables.find(i->first);
2297 if (j != variables.end()) v = j->second;
2300 else if (!k.second) v.clear();
2301 v.insert(v.end(), i->second.value.begin(), i->second.value.end());
2303 if (!job.rule.deps.empty() || !job.rule.wdeps.empty())
2305 current = clients.insert(current, client_t());
2306 current->job_id = job_id;
2307 current->pending = job.rule.deps;
2308 current->pending.insert(current->pending.end(),
2309 job.rule.wdeps.begin(), job.rule.wdeps.end());
2310 if (propagate_vars) current->vars = job.vars;
2311 current->delayed = true;
2314 return run_script(job_id, job);
2318 * Send a reply to a client then remove it.
2319 * If the client was a dependency client, start the actual script.
2321 static void complete_request(client_t &client, bool success)
2323 DEBUG_open << "Completing request from client of job " << client.job_id << "... ";
2326 assert(client.socket == INVALID_SOCKET);
2329 job_map::const_iterator i = jobs.find(client.job_id);
2330 assert(i != jobs.end());
2331 if (still_need_rebuild(i->second.rule.targets.front()))
2332 run_script(client.job_id, i->second);
2333 else complete_job(client.job_id, true);
2335 else complete_job(client.job_id, false);
2337 else if (client.socket != INVALID_SOCKET)
2339 char res = success ? 1 : 0;
2340 send(client.socket, &res, 1, MSG_NOSIGNAL);
2342 closesocket(client.socket);
2344 close(client.socket);
2349 if (client.job_id < 0 && !success) build_failure = true;
2353 * Return whether there are slots for starting new jobs.
2355 static bool has_free_slots()
2357 if (max_active_jobs <= 0) return true;
2358 return running_jobs - waiting_jobs < max_active_jobs;
2362 * Handle client requests:
2363 * - check for running targets that have finished,
2364 * - start as many pending targets as allowed,
2365 * - complete the request if there are neither running nor pending targets
2366 * left or if any of them failed.
2368 * @return true if some child processes are still running.
2370 * @post If there are pending requests, at least one child process is running.
2372 * @invariant New free slots cannot appear during a run, since the only way to
2373 * decrease #running_jobs is #finalize_job and the only way to
2374 * increase #waiting_jobs is #accept_client. None of these functions
2375 * are called during a run. So breaking out as soon as there no free
2376 * slots left is fine.
2378 static bool handle_clients()
2380 DEBUG_open << "Handling client requests... ";
2382 bool need_restart = false;
2384 for (client_list::iterator i = clients.begin(), i_next = i,
2385 i_end = clients.end(); i != i_end && has_free_slots(); i = i_next)
2388 DEBUG_open << "Handling client from job " << i->job_id << "... ";
2390 // Remove running targets that have finished.
2391 for (string_set::iterator j = i->running.begin(), j_next = j,
2392 j_end = i->running.end(); j != j_end; j = j_next)
2395 status_map::const_iterator k = status.find(*j);
2396 assert(k != status.end());
2397 switch (k->second.status)
2403 if (!keep_going) goto complete;
2407 i->running.erase(j);
2415 // Start pending targets.
2416 while (!i->pending.empty())
2418 std::string target = i->pending.front();
2419 i->pending.pop_front();
2420 switch (get_status(target).status)
2423 i->running.insert(target);
2428 if (!keep_going) goto complete;
2435 client_list::iterator j = i;
2436 switch (start(target, i))
2439 goto pending_failed;
2441 // A shell was started, check for free slots.
2442 j->running.insert(target);
2443 if (!has_free_slots()) return true;
2446 // Switch to the dependency client that was inserted.
2447 j->running.insert(target);
2452 need_restart = true;
2460 // Try to complete the request.
2461 // (This might start a new job if it was a dependency client.)
2462 if (i->running.empty() || i->failed)
2465 complete_request(*i, !i->failed);
2466 DEBUG_close << (i->failed ? "failed\n" : "finished\n");
2468 need_restart = true;
2472 if (running_jobs != waiting_jobs) return true;
2473 if (running_jobs == 0 && clients.empty()) return false;
2474 if (need_restart) goto restart;
2476 // There is a circular dependency.
2477 // Try to break it by completing one of the requests.
2478 assert(!clients.empty());
2479 std::cerr << "Circular dependency detected" << std::endl;
2480 client_list::iterator i = clients.begin();
2481 complete_request(*i, false);
2487 * Create a named unix socket that listens for build requests. Also set
2488 * the REMAKE_SOCKET environment variable that will be inherited by all
2491 static void create_server()
2496 perror("Failed to create server");
2502 DEBUG_open << "Creating server... ";
2505 // Prepare a windows socket.
2506 struct sockaddr_in socket_addr;
2507 socket_addr.sin_family = AF_INET;
2508 socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
2509 socket_addr.sin_port = 0;
2511 // Create and listen to the socket.
2512 socket_fd = socket(AF_INET, SOCK_STREAM, 0);
2513 if (socket_fd == INVALID_SOCKET) goto error;
2514 if (!SetHandleInformation((HANDLE)socket_fd, HANDLE_FLAG_INHERIT, 0))
2516 if (bind(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
2518 int len = sizeof(sockaddr_in);
2519 if (getsockname(socket_fd, (struct sockaddr *)&socket_addr, &len))
2521 std::ostringstream buf;
2522 buf << socket_addr.sin_port;
2523 if (!SetEnvironmentVariable("REMAKE_SOCKET", buf.str().c_str()))
2525 if (listen(socket_fd, 1000)) goto error;
2527 // Set signal handlers for SIGCHLD and SIGINT.
2528 // Block SIGCHLD (unblocked during select).
2530 sigemptyset(&sigmask);
2531 sigaddset(&sigmask, SIGCHLD);
2532 if (sigprocmask(SIG_BLOCK, &sigmask, NULL) == -1) goto error;
2533 struct sigaction sa;
2535 sigemptyset(&sa.sa_mask);
2536 sa.sa_handler = &sigchld_handler;
2537 if (sigaction(SIGCHLD, &sa, NULL) == -1) goto error;
2538 sa.sa_handler = &sigint_handler;
2539 if (sigaction(SIGINT, &sa, NULL) == -1) goto error;
2541 // Prepare a named unix socket in temporary directory.
2542 socket_name = tempnam(NULL, "rmk-");
2543 if (!socket_name) goto error2;
2544 struct sockaddr_un socket_addr;
2545 size_t len = strlen(socket_name);
2546 if (len >= sizeof(socket_addr.sun_path) - 1) goto error2;
2547 socket_addr.sun_family = AF_UNIX;
2548 strcpy(socket_addr.sun_path, socket_name);
2549 len += sizeof(socket_addr.sun_family);
2550 if (setenv("REMAKE_SOCKET", socket_name, 1)) goto error;
2552 // Create and listen to the socket.
2554 socket_fd = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
2555 if (socket_fd == INVALID_SOCKET) goto error;
2557 socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
2558 if (socket_fd == INVALID_SOCKET) goto error;
2559 if (fcntl(socket_fd, F_SETFD, FD_CLOEXEC) < 0) goto error;
2561 if (bind(socket_fd, (struct sockaddr *)&socket_addr, len))
2563 if (listen(socket_fd, 1000)) goto error;
2568 * Accept a connection from a client, get the job it spawned from,
2569 * get the targets, and mark them as dependencies of the job targets.
2571 static void accept_client()
2573 DEBUG_open << "Handling client request... ";
2575 // Accept connection.
2577 socket_t fd = accept(socket_fd, NULL, NULL);
2578 if (fd == INVALID_SOCKET) return;
2579 if (!SetHandleInformation((HANDLE)fd, HANDLE_FLAG_INHERIT, 0))
2582 std::cerr << "Unexpected failure while setting connection with client" << std::endl;
2586 // WSAEventSelect puts sockets into nonblocking mode, so disable it here.
2588 if (ioctlsocket(fd, FIONBIO, &nbio)) goto error2;
2589 #elif defined(LINUX)
2590 int fd = accept4(socket_fd, NULL, NULL, SOCK_CLOEXEC);
2593 int fd = accept(socket_fd, NULL, NULL);
2595 if (fcntl(fd, F_SETFD, FD_CLOEXEC) < 0) return;
2597 clients.push_front(client_t());
2598 client_list::iterator proc = clients.begin();
2603 DEBUG_close << "failed\n";
2604 std::cerr << "Received an ill-formed client message" << std::endl;
2610 clients.erase(proc);
2614 // Receive message. Stop when encountering two nuls in a row.
2615 std::vector<char> buf;
2617 while (len < sizeof(int) + 2 || buf[len - 1] || buf[len - 2])
2619 buf.resize(len + 1024);
2620 ssize_t l = recv(fd, &buf[0] + len, 1024, 0);
2621 if (l <= 0) goto error;
2625 // Parse job that spawned the client.
2627 memcpy(&job_id, &buf[0], sizeof(int));
2629 proc->job_id = job_id;
2630 job_map::const_iterator i = jobs.find(job_id);
2631 if (i == jobs.end()) goto error;
2632 DEBUG << "receiving request from job " << job_id << std::endl;
2633 if (propagate_vars) proc->vars = i->second.vars;
2635 // Parse the targets and the variable assignments.
2636 // Mark the targets as dependencies of the job targets.
2637 dependency_t &dep = *dependencies[i->second.rule.targets.front()];
2638 string_list *last_var = NULL;
2639 char const *p = &buf[0] + sizeof(int);
2652 if (len == 1) goto error;
2653 std::string target(p + 1, p + len);
2654 DEBUG << "adding dependency " << target << " to job\n";
2655 proc->pending.push_back(target);
2656 dep.deps.insert(target);
2661 if (len == 1) goto error;
2662 std::string var(p + 1, p + len);
2663 DEBUG << "adding variable " << var << " to job\n";
2664 last_var = &proc->vars[var];
2670 if (!last_var) goto error;
2671 last_var->push_back(std::string(p + 1, p + len));
2680 if (!propagate_vars && !proc->vars.empty())
2682 std::cerr << "Assignments are ignored unless 'variable-propagation' is enabled" << std::endl;
2688 * Handle child process exit status.
2690 static void finalize_job(pid_t pid, bool res)
2692 pid_job_map::iterator i = job_pids.find(pid);
2693 assert(i != job_pids.end());
2694 int job_id = i->second;
2697 complete_job(job_id, res);
2701 * Loop until all the jobs have finished.
2703 * @post There are no client requests left, not even virtual ones.
2705 static void server_loop()
2707 while (handle_clients())
2709 DEBUG_open << "Handling events... ";
2711 size_t len = job_pids.size() + 1;
2714 for (pid_job_map::const_iterator i = job_pids.begin(),
2715 i_end = job_pids.end(); i != i_end; ++i, ++num)
2719 WSAEVENT aev = WSACreateEvent();
2721 WSAEventSelect(socket_fd, aev, FD_ACCEPT);
2722 DWORD w = WaitForMultipleObjects(len, h, false, INFINITE);
2723 WSAEventSelect(socket_fd, aev, 0);
2734 bool res = GetExitCodeProcess(pid, &s) && s == 0;
2736 finalize_job(pid, res);
2739 sigemptyset(&emptymask);
2742 FD_SET(socket_fd, &fdset);
2743 int ret = pselect(socket_fd + 1, &fdset, NULL, NULL, NULL, &emptymask);
2744 if (ret > 0 /* && FD_ISSET(socket_fd, &fdset)*/) accept_client();
2745 if (!got_SIGCHLD) continue;
2749 while ((pid = waitpid(-1, &status, WNOHANG)) > 0)
2751 bool res = WIFEXITED(status) && WEXITSTATUS(status) == 0;
2752 finalize_job(pid, res);
2757 assert(clients.empty());
2761 * Load dependencies and rules, listen to client requests, and loop until
2762 * all the requests have completed.
2763 * If Remakefile is obsolete, perform a first run with it only, then reload
2764 * the rules, and perform a second with the original clients.
2766 static void server_mode(std::string const &remakefile, string_list const &targets)
2768 load_dependencies();
2769 load_rules(remakefile);
2771 if (get_status(remakefile).status != Uptodate)
2773 clients.push_back(client_t());
2774 clients.back().pending.push_back(remakefile);
2776 if (build_failure) goto early_exit;
2778 specific_rules.clear();
2779 generic_rules.clear();
2780 first_target.clear();
2781 load_rules(remakefile);
2783 clients.push_back(client_t());
2784 if (!targets.empty()) clients.back().pending = targets;
2785 else if (!first_target.empty())
2786 clients.back().pending.push_back(first_target);
2791 remove(socket_name);
2794 save_dependencies();
2795 exit(build_failure ? EXIT_FAILURE : EXIT_SUCCESS);
2801 * @defgroup client Client
2807 * Connect to the server @a socket_name, send a request for building @a targets
2808 * with some @a variables, and exit with the status returned by the server.
2810 static void client_mode(char *socket_name, string_list const &targets)
2815 perror("Failed to send targets to server");
2818 if (targets.empty()) exit(EXIT_SUCCESS);
2819 DEBUG_open << "Connecting to server... ";
2821 // Connect to server.
2823 struct sockaddr_in socket_addr;
2824 socket_fd = socket(AF_INET, SOCK_STREAM, 0);
2825 if (socket_fd == INVALID_SOCKET) goto error;
2826 socket_addr.sin_family = AF_INET;
2827 socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
2828 socket_addr.sin_port = atoi(socket_name);
2829 if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
2832 struct sockaddr_un socket_addr;
2833 size_t len = strlen(socket_name);
2834 if (len >= sizeof(socket_addr.sun_path) - 1) exit(EXIT_FAILURE);
2835 socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
2836 if (socket_fd == INVALID_SOCKET) goto error;
2837 socket_addr.sun_family = AF_UNIX;
2838 strcpy(socket_addr.sun_path, socket_name);
2839 if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(socket_addr.sun_family) + len))
2843 if (setsockopt(socket_fd, SOL_SOCKET, SO_NOSIGPIPE, &set_option, sizeof(set_option)))
2848 // Send current job id.
2849 char *id = getenv("REMAKE_JOB_ID");
2850 int job_id = id ? atoi(id) : -1;
2851 if (send(socket_fd, (char *)&job_id, sizeof(job_id), MSG_NOSIGNAL) != sizeof(job_id))
2855 for (string_list::const_iterator i = targets.begin(),
2856 i_end = targets.end(); i != i_end; ++i)
2858 DEBUG_open << "Sending target " << *i << "... ";
2859 std::string s = 'T' + *i;
2860 ssize_t len = s.length() + 1;
2861 if (send(socket_fd, s.c_str(), len, MSG_NOSIGNAL) != len)
2866 for (variable_map::const_iterator i = variables.begin(),
2867 i_end = variables.end(); i != i_end; ++i)
2869 DEBUG_open << "Sending variable " << i->first << "... ";
2870 std::string s = 'V' + i->first;
2871 ssize_t len = s.length() + 1;
2872 if (send(socket_fd, s.c_str(), len, MSG_NOSIGNAL) != len)
2874 for (string_list::const_iterator j = i->second.begin(),
2875 j_end = i->second.end(); j != j_end; ++j)
2877 std::string s = 'W' + *j;
2878 len = s.length() + 1;
2879 if (send(socket_fd, s.c_str(), len, MSG_NOSIGNAL) != len)
2884 // Send terminating nul and wait for reply.
2886 if (send(socket_fd, &result, 1, MSG_NOSIGNAL) != 1) goto error;
2887 if (recv(socket_fd, &result, 1, 0) != 1) exit(EXIT_FAILURE);
2888 exit(result ? EXIT_SUCCESS : EXIT_FAILURE);
2894 * @defgroup ui User interface
2900 * Display usage and exit with @a exit_status.
2902 static void usage(int exit_status)
2904 std::cerr << "Usage: remake [options] [target] ...\n"
2906 " -d Echo script commands.\n"
2907 " -d -d Print lots of debugging information.\n"
2908 " -f FILE Read FILE as Remakefile.\n"
2909 " -h, --help Print this message and exit.\n"
2910 " -j[N], --jobs=[N] Allow N jobs at once; infinite jobs with no arg.\n"
2911 " -k Keep going when some targets cannot be made.\n"
2912 " -r Look up targets from the dependencies on stdin.\n"
2913 " -s, --silent, --quiet Do not echo targets.\n";
2918 * This program behaves in two different ways.
2920 * - If the environment contains the REMAKE_SOCKET variable, the client
2921 * connects to this socket and sends to the server its build targets.
2922 * It exits once it receives the server reply.
2924 * - Otherwise, it creates a server that waits for build requests. It
2925 * also creates a pseudo-client that requests the targets passed on the
2928 int main(int argc, char *argv[])
2932 std::string remakefile;
2933 string_list targets;
2934 bool literal_targets = false;
2935 bool indirect_targets = false;
2937 // Parse command-line arguments.
2938 for (int i = 1; i < argc; ++i)
2940 std::string arg = argv[i];
2941 if (arg.empty()) usage(EXIT_FAILURE);
2942 if (literal_targets) goto new_target;
2943 if (arg == "-h" || arg == "--help") usage(EXIT_SUCCESS);
2945 if (echo_scripts) debug.active = true;
2946 else echo_scripts = true;
2947 else if (arg == "-k" || arg =="--keep-going")
2949 else if (arg == "-s" || arg == "--silent" || arg == "--quiet")
2950 show_targets = false;
2951 else if (arg == "-r")
2952 indirect_targets = true;
2953 else if (arg == "-f")
2955 if (++i == argc) usage(EXIT_FAILURE);
2956 remakefile = argv[i];
2958 else if (arg == "--")
2959 literal_targets = true;
2960 else if (arg.compare(0, 2, "-j") == 0)
2961 max_active_jobs = atoi(arg.c_str() + 2);
2962 else if (arg.compare(0, 7, "--jobs=") == 0)
2963 max_active_jobs = atoi(arg.c_str() + 7);
2966 if (arg[0] == '-') usage(EXIT_FAILURE);
2967 if (arg.find('=') != std::string::npos)
2969 std::istringstream in(arg);
2970 std::string name = read_word(in);
2971 if (name.empty() || !expect_token(in, Equal)) usage(EXIT_FAILURE);
2972 read_words(in, variables[name]);
2976 targets.push_back(normalize(arg, working_dir, working_dir));
2977 DEBUG << "New target: " << arg << '\n';
2981 if (indirect_targets)
2983 load_dependencies(std::cin);
2986 if (l.empty() && !dependencies.empty())
2988 l.push_back(dependencies.begin()->second->targets.front());
2990 for (string_list::const_iterator i = l.begin(),
2991 i_end = l.end(); i != i_end; ++i)
2993 dependency_map::const_iterator j = dependencies.find(*i);
2994 if (j == dependencies.end()) continue;
2995 dependency_t const &dep = *j->second;
2996 for (string_set::const_iterator k = dep.deps.begin(),
2997 k_end = dep.deps.end(); k != k_end; ++k)
2999 targets.push_back(normalize(*k, working_dir, working_dir));
3002 dependencies.clear();
3007 if (WSAStartup(MAKEWORD(2,2), &wsaData))
3009 std::cerr << "Unexpected failure while initializing Windows Socket" << std::endl;
3014 // Run as client if REMAKE_SOCKET is present in the environment.
3015 if (char *sn = getenv("REMAKE_SOCKET")) client_mode(sn, targets);
3017 // Otherwise run as server.
3018 if (remakefile.empty())
3020 remakefile = "Remakefile";
3023 normalize_list(targets, working_dir, prefix_dir);
3024 server_mode(remakefile, targets);