2 @mainpage Remake, a build system that bridges the gap between make and redo.
4 As with <b>make</b>, <b>remake</b> uses a centralized rule file, which is
5 named <b>Remakefile</b>. It contains rules with a <em>make</em>-like
9 target1 target2 ... : prerequisite1 prerequisite2 ...
15 A target is known to be up-to-date if all its prerequisites are. If it
16 has no known prerequisites yet the file already exits, it is assumed to
17 be up-to-date. Obsolete targets are rebuilt thanks to the shell script
20 As with <b>redo</b>, <b>remake</b> supports dynamic dependencies in
21 addition to these static dependencies. Whenever a script executes
22 <tt>remake prerequisite4 prerequisite5 ...</tt>, these prerequisites are
23 rebuilt if they are obsolete. (So <b>remake</b> acts like
24 <b>redo-ifchange</b>.) Moreover, all the dependencies are stored in file
25 <b>.remake</b> so that they are remembered in subsequent runs. Note that
26 dynamic dependencies from previous runs are only used to decide whether a
27 target is obsolete; they are not automatically rebuilt when they are
28 obsolete yet a target depends on them. They will only be rebuilt once the
29 dynamic call to <b>remake</b> is executed.
31 In other words, the following two rules have almost the same behavior.
34 target1 target2 ... : prerequisite1 prerequisite2 ...
38 remake prerequisite1 prerequisite2 ...
42 (There is a difference if the targets already exist, have never been
43 built before, and the prerequisites are either younger or obsolete, since
44 the targets will not be rebuilt in the second case.)
46 The above usage of dynamic dependencies is hardly useful. Their strength
47 lies in the fact that they can be computed on the fly:
51 gcc -MMD -MF $@.d -o $@ -c $<
56 ocamldep $< | remake -r $@
59 after.xml: before.xml rules.xsl
60 xsltproc --load-trace -o after.xml rules.xsl before.xml 2> deps
61 remake `sed -n -e "\\,//,! s,^.*URL=\"\\([^\"]*\\).*\$,\\1,p" deps`
65 Note that the first rule fails if any of the header files included by
66 a C source file has to be automatically generated. In that case, one
67 should perform a first call to <b>remake</b> them before calling the
68 compiler. (Dependencies from several calls to <b>remake</b> are
69 cumulative, so they will all be remembered the next time.)
71 \section sec-usage Usage
73 Usage: <tt>remake <i>options</i> <i>targets</i></tt>
77 - <tt>-d</tt>: Echo script commands.
78 - <tt>-f FILE</tt>: Read <tt>FILE</tt> as <b>Remakefile</b>.
79 - <tt>-j[N]</tt>, <tt>--jobs=[N]</tt>: Allow <tt>N</tt> jobs at once;
80 infinite jobs with no argument.
81 - <tt>-k</tt>, <tt>--keep-going</tt>: Keep going when some targets cannot be made.
82 - <tt>-r</tt>: Look up targets from the dependencies on standard input.
83 - <tt>-s</tt>, <tt>--silent</tt>, <tt>--quiet</tt>: Do not echo targets.
85 \section sec-syntax Syntax
87 Lines starting with a space character or a tabulation are assumed to be rule
88 scripts. They are only allowed after a rule header.
90 Lines starting with <tt>#</tt> are considered to be comments and are ignored.
91 They do interrupt rule scripts though.
93 Any other line is either a rule header or a variable definition. If such a
94 line ends with a backslash, the following line break is ignored and the line
95 extends to the next one.
97 Rule headers are a nonempty list of names, followed by a colon, followed by
98 another list of names, possibly empty. Variable definitions are a single
99 name followed by equal followed by a list of names, possibly empty. Basically,
100 the syntax of a rule is as follows:
103 targets : prerequisites
107 List of names are space-separated sequences of names. If a name contains a
108 space character, it should be put into double quotes. Names can not be any
109 of the following special characters <tt>:$(),="</tt>. Again, quotation
110 should be used. Quotation marks can be escaped by a backslash inside
113 \subsection sec-variables Variables
115 Variables can be used to factor lists of targets or prerequisites. They are
116 expanded as they are encountered during <b>Remakefile</b> parsing.
125 Variable assignments can appear instead of prerequisites inside non-generic
126 rules with no script. They are then expanded inside the corresponding
130 foo.o: CFLAGS += -DBAR
133 gcc $(CFLAGS) -MMD -MF $@.d -o $@ -c $<
138 Note: contrarily to <b>make</b>, variable names have to be enclosed in
139 parentheses. For instance, <tt>$y</tt> is not a shorthand for <tt>\$(y)</tt> and
142 \subsection sec-autovars Automatic variables
144 The following special symbols can appear inside scripts:
146 - <tt>$<</tt> expands to the first static prerequisite of the rule.
147 - <tt>$^</tt> expands to all the static prerequisites of the rule, including
149 - <tt>$\@</tt> expands to the first target of the rule.
150 - <tt>$*</tt> expands to the string that matched <tt>%</tt> in a generic rule.
151 - <tt>$$</tt> expands to a single dollar symbol.
153 Note: contrarily to <b>make</b>, there are no corresponding variables. For
154 instance, <tt>$^</tt> is not a shorthand for <tt>$(^)</tt>. Another
155 difference is that <tt>$\@</tt> is always the first target, not the one that
158 \subsection sec-functions Built-in functions
160 <b>remake</b> also supports a few built-in functions inspired from <b>make</b>.
162 - <tt>$(addprefix <i>prefix</i>, <i>list</i>)</tt> returns the list obtained
163 by prepending its first argument to each element of its second argument.
164 - <tt>$(addsuffix <i>suffix</i>, <i>list</i>)</tt> returns the list obtained
165 by appending its first argument to each element of its second argument.
167 \section sec-semantics Semantics
169 \subsection src-obsolete When are targets obsolete?
171 A target is obsolete:
173 - if there is no file corresponding to the target, or to one of its siblings
174 in a multi-target rule,
175 - if any of its dynamic prerequisites from a previous run or any of its static
176 prerequisites is obsolete,
177 - if the latest file corresponding to its siblings or itself is older than any
178 of its dynamic prerequisites or static prerequisites.
180 In all the other cases, it is assumed to be up-to-date (and so are all its
181 siblings). Note that the last rule above says "latest" and not "earliest". While
182 it might cause some obsolete targets to go unnoticed in corner cases, it allows
183 for the following kind of rules:
186 config.h stamp-config_h: config.h.in config.status
187 ./config.status config.h
191 A <tt>config.status</tt> file generally does not update header files (here
192 <tt>config.h</tt>) if they would not change. As a consequence, if not for the
193 <tt>stamp-config_h</tt> file above, a header would always be considered obsolete
194 once one of its prerequisites is modified. Note that touching <tt>config.h</tt>
195 rather than <tt>stamp-config_h</tt> would defeat the point of not updating it
196 in the first place, since the program files would need to be rebuilt.
198 Once all the static prerequisites of a target have been rebuilt, <b>remake</b>
199 checks whether the target still needs to be built. If it was obsolete only
200 because its prerequisites needed to be rebuilt and none of them changed, the
201 target is assumed to be up-to-date.
203 \subsection sec-rules How are targets (re)built?
205 There are two kinds of rules. If any of the targets or prerequisites contains
206 a <tt>%</tt> character, the rule is said to be <em>generic</em>. All the
207 targets of the rule shall then contain a single <tt>%</tt> character. All the
208 other rules are said to be <em>specific</em>.
210 A rule is said to <em>match</em> a given target:
212 - if it is specific and the target appears inside its target list,
213 - if it is generic and there is a way to replace the <tt>%</tt> character
214 from one of its targets so that it matches the given target.
216 When <b>remake</b> tries to build a given target, it looks for a specific rule
217 that matches it. If there is one and its script is nonempty, it uses it to
220 Otherwise, it looks for a generic rule that match the target. If there are
221 several matching rules, it chooses the one with the shortest pattern (and if
222 there are several ones, the earliest one). <b>remake</b> then looks for
223 specific rules that match each target of the generic rule. All the
224 prerequisites of these specific rules are added to those of the generic rule.
225 The script of the generic rule is used to build the target.
231 commands building t%1 and t2%
234 commands building t2z
238 # t2x is built by the first rule (which also builds tx1) and its prerequisites are p1, px2
239 # t2y is built by the first rule (which also builds ty1) and its prerequisites are p1, py2, p3
240 # t2z is built by the second rule and its prerequisite is p4
243 The set of rules from <b>Remakefile</b> is ill-formed:
245 - if any specific rule matching a target of the generic rule has a nonempty script,
246 - if any target of the generic rule is matched by a generic rule with a shorter pattern.
248 \section sec-compilation Compilation
250 - On Linux, MacOSX, and BSD: <tt>g++ -o remake remake.cpp</tt>
251 - On Windows: <tt>g++ -o remake.exe remake.cpp -lws2_32</tt>
253 Installing <b>remake</b> is needed only if <b>Remakefile</b> does not
254 specify the path to the executable for its recursive calls. Thanks to its
255 single source file, <b>remake</b> can be shipped inside other packages and
256 built at configuration time.
258 \section sec-differences Differences with other build systems
260 Differences with <b>make</b>:
262 - Dynamic dependencies are supported.
263 - For rules with multiple targets, the shell script is executed only once
264 and is assumed to build all the targets. There is no need for
265 convoluted rules that are robust enough for parallel builds. For generic
266 rules, this is similar to the behavior of pattern rules from <b>gmake</b>.
267 - As with <b>redo</b>, only one shell is run when executing a script,
268 rather than one per script line. Note that the shells are run with
269 option <tt>-e</tt>, thus causing them to exit as soon as an error is
271 - The prerequisites of generic rules (known as implicit rules in make lingo)
272 are not used to decide between several of them. <b>remake</b> does not
273 select one for which it could satisfy the dependencies.
274 - Variables and built-in functions are expanded as they are encountered
275 during <b>Remakefile</b> parsing.
277 Differences with <b>redo</b>:
279 - As with <b>make</b>, it is possible to write the following kind of rules
282 Remakefile: Remakefile.in ./config.status
283 ./config.status Remakefile
285 - If a target is already built the first time <b>remake</b> runs, it still
286 uses the static prerequisites of rules mentioning it to check whether it
287 needs to be rebuilt. It does not assume it to be up-to-date. As with
288 <b>redo</b> though, if its obsolete status would be due to a dynamic
289 prerequisite, it will go unnoticed; it should be removed beforehand.
290 - Multiple targets are supported.
291 - <b>remake</b> has almost no features: no checksum-based dependencies, no
292 compatibility with job servers, etc.
294 \section sec-limitations Limitations
296 - When the user calls <b>remake</b>, the current working directory should be
297 the one containing <b>.remake</b>. Rules are understood relatively to this
298 directory. If a rule script calls <b>remake</b>, the current working
299 directory should be the same as the one from the original <b>remake</b>.
300 - Some cases of ill-formed rules are not caught by <b>remake</b> and can
301 thus lead to unpredictable behaviors.
303 \section sec-links Links
305 @see http://cr.yp.to/redo.html for the philosophy of <b>redo</b> and
306 https://github.com/apenwarr/redo for an implementation and some comprehensive documentation.
308 \section sec-licensing Licensing
310 @author Guillaume Melquiond
314 This program is free software: you can redistribute it and/or modify
315 it under the terms of the GNU General Public License as published by
316 the Free Software Foundation, either version 3 of the License, or
317 (at your option) any later version.
319 This program is distributed in the hope that it will be useful,
320 but WITHOUT ANY WARRANTY; without even the implied warranty of
321 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
322 GNU General Public License for more details.
324 \section sec-internals Internals
326 The parent <b>remake</b> process acts as a server. The other ones have a
327 REMAKE_SOCKET environment variable that tells them how to contact the
328 server. They send the content of the REMAKE_JOB_ID environment variable,
329 so that the server can associate the child targets to the jobs that
330 spawned them. They then wait for completion and exit with the status
331 returned by the server. This is handled by #client_mode.
333 The server calls #load_dependencies and #save_dependencies to serialize
334 dynamic dependencies from <b>.remake</b>. It loads <b>Remakefile</b> with
335 #load_rules. It then runs #server_mode, which calls #server_loop.
337 When building a target, the following sequence of events happens:
339 - #start calls #find_rule (and #find_generic_rule) to get the rule.
340 - It then creates a pseudo-client if the rule has static dependencies, or
341 calls #run_script otherwise. In both cases, a new job is created and its
342 targets are put into #job_targets.
343 - #run_script creates a shell process and stores it in #job_pids. It
344 increases #running_jobs.
345 - The child process possibly calls <b>remake</b> with a list of targets.
346 - #accept_client receives a build request from a child process and adds
347 it to #clients. It also records the new dependencies of the job into
348 #dependencies. It increases #waiting_jobs.
349 - #handle_clients uses #get_status to look up the obsoleteness of the
351 - Once the targets of a request have been built or one of them has failed,
352 #handle_clients calls #complete_request and removes the request from
354 - If the build targets come from a pseudo-client, #complete_request calls
355 #run_script. Otherwise it sends the reply to the corresponding child
356 process and decreases #waiting_jobs.
357 - When a child process ends, #server_loop calls #finalize_job, which
358 removes the process from #job_pids, decreases #running_jobs, and calls
360 - #complete_job removes the job from #job_targets and calls #update_status
361 to change the status of the targets. It also removes the target files in
366 #define WIN32_LEAN_AND_MEAN
385 #include <sys/stat.h>
386 #include <sys/types.h>
399 #include <winsock2.h>
401 typedef SOCKET socket_t;
403 #include <sys/socket.h>
405 #include <sys/wait.h>
406 typedef int socket_t;
407 enum { INVALID_SOCKET = -1 };
410 #if defined(WINDOWS) || defined(MACOSX)
411 enum { MSG_NOSIGNAL = 0 };
414 typedef std::list<std::string> string_list;
416 typedef std::set<std::string> string_set;
419 * Reference-counted shared object.
420 * @note The default constructor delays the creation of the object until it
421 * is first dereferenced.
431 content(T const &t): cnt(1), val(t) {}
433 mutable content *ptr;
434 ref_ptr(): ptr(NULL) {}
435 ref_ptr(T const &t): ptr(new content(t)) {}
436 ref_ptr(ref_ptr const &p): ptr(p.ptr) { if (ptr) ++ptr->cnt; }
437 ~ref_ptr() { if (ptr && --ptr->cnt == 0) delete ptr; }
438 ref_ptr &operator=(ref_ptr const &p)
440 if (ptr == p.ptr) return *this;
441 if (ptr && --ptr->cnt == 0) delete ptr;
448 if (!ptr) ptr = new content;
451 T *operator->() const { return &**this; }
460 typedef std::map<std::string, ref_ptr<dependency_t> > dependency_map;
462 typedef std::map<std::string, string_list> variable_map;
465 * Build status of a target.
469 Uptodate, ///< Target is up-to-date.
470 Todo, ///< Target is missing or obsolete.
471 Recheck, ///< Target has an obsolete dependency.
472 Running, ///< Target is being rebuilt.
473 Remade, ///< Target was successfully rebuilt.
474 Failed ///< Build failed for target.
478 * Build status of a target.
482 status_e status; ///< Actual status.
483 time_t last; ///< Last-modified date.
486 typedef std::map<std::string, status_t> status_map;
489 * Delayed assignment to a variable.
498 typedef std::list<assign_t> assign_list;
501 * A rule loaded from Remakefile.
505 string_list targets; ///< Files produced by this rule.
506 string_list deps; ///< Files used for an implicit call to remake at the start of the script.
507 assign_list vars; ///< Values of variables.
508 std::string script; ///< Shell script for building the targets.
509 std::string stem; ///< String used to instantiate a generic rule.
512 typedef std::list<rule_t> rule_list;
514 typedef std::map<std::string, ref_ptr<rule_t> > rule_map;
516 typedef std::map<int, string_list> job_targets_map;
518 typedef std::map<int, variable_map> job_variables_map;
520 typedef std::map<pid_t, int> pid_job_map;
523 * Client waiting for a request to complete.
525 * There are two kinds of clients:
526 * - real clients, which are instances of remake created by built scripts,
527 * - pseudo clients, which are created by the server to build specific targets.
529 * Among pseudo clients, there are two categories:
530 * - original clients, which are created for the targets passed on the
531 * command line by the user or for the initial regeneration of the rule file,
532 * - dependency clients, which are created to handle rules that have
533 * explicit dependencies and thus to emulate a call to remake.
537 socket_t socket; ///< Socket used to reply to the client (invalid for pseudo clients).
538 int job_id; ///< Job for which the built script called remake and spawned the client (negative for original clients).
539 bool failed; ///< Whether some targets failed in mode -k.
540 string_list pending; ///< Targets not yet started.
541 string_set running; ///< Targets being built.
542 rule_t *delayed; ///< Rule that implicitly created a dependency client, and which script has to be started on request completion.
543 client_t(): socket(INVALID_SOCKET), job_id(-1), failed(false), delayed(NULL) {}
546 typedef std::list<client_t> client_list;
549 * Map from variable names to their content.
551 static variable_map variables;
554 * Map from targets to their known dependencies.
556 static dependency_map dependencies;
559 * Map from targets to their build status.
561 static status_map status;
564 * Set of generic rules loaded from Remakefile.
566 static rule_list generic_rules;
569 * Map from targets to specific rules loaded from Remakefile.
571 static rule_map specific_rules;
574 * Map from jobs to targets being built.
576 static job_targets_map job_targets;
579 * Map from jobs to job specific variables
581 static job_variables_map job_variables;
584 * Map from jobs to shell pids.
586 static pid_job_map job_pids;
589 * List of clients waiting for a request to complete.
590 * New clients are put to front, so that the build process is depth-first.
592 static client_list clients;
595 * Maximum number of parallel jobs (non-positive if unbounded).
596 * Can be modified by the -j option.
598 static int max_active_jobs = 1;
601 * Whether to keep building targets in case of failure.
602 * Can be modified by the -k option.
604 static bool keep_going = false;
607 * Number of jobs currently running:
608 * - it increases when a process is created in #run_script,
609 * - it decreases when a completion message is received in #finalize_job.
611 * @note There might be some jobs running while #clients is empty.
612 * Indeed, if a client requested two targets to be rebuilt, if they
613 * are running concurrently, if one of them fails, the client will
614 * get a failure notice and might terminate before the other target
617 static int running_jobs = 0;
620 * Number of jobs currently waiting for a build request to finish:
621 * - it increases when a build request is received in #accept_client
622 * (since the client is presumably waiting for the reply),
623 * - it decreases when a reply is sent in #complete_request.
625 static int waiting_jobs = 0;
628 * Global counter used to produce increasing job numbers.
631 static int job_counter = 0;
634 * Socket on which the server listens for client request.
636 static socket_t socket_fd;
639 * Whether the request of an original client failed.
641 static bool build_failure;
645 * Name of the server socket in the file system.
647 static char *socket_name;
651 * Name of the first target of the first specific rule, used for default run.
653 static std::string first_target;
656 * Whether a short message should be displayed for each target.
658 static bool show_targets = true;
661 * Whether script commands are echoed.
663 static bool echo_scripts = false;
665 static time_t now = time(NULL);
667 static std::string working_dir;
670 static volatile sig_atomic_t got_SIGCHLD = 0;
672 static void sigchld_handler(int)
677 static void sigint_handler(int)
679 // Child processes will receive the signal too, so just prevent
680 // new jobs from starting and wait for the running jobs to fail.
689 log(): active(false), open(false), depth(0)
692 std::ostream &operator()()
694 if (open) std::cerr << std::endl;
696 std::cerr << std::string(depth * 2, ' ');
700 std::ostream &operator()(bool o)
702 if (o && open) std::cerr << std::endl;
705 if (o || !open) std::cerr << std::string(depth * 2, ' ');
714 struct log_auto_close
717 log_auto_close(): still_open(true)
722 if (debug.active && still_open) debug(false) << "done\n";
726 #define DEBUG if (debug.active) debug()
727 #define DEBUG_open log_auto_close auto_close; if (debug.active) debug(true)
728 #define DEBUG_close if ((auto_close.still_open = false), debug.active) debug(false)
731 * Strong typedef for strings that need escaping.
732 * @note The string is stored as a reference, so the constructed object is
733 * meant to be immediately consumed.
737 std::string const &input;
738 escape_string(std::string const &s): input(s) {}
742 * Write the string in @a se to @a out if it does not contain any special
743 * characters, a quoted and escaped string otherwise.
745 static std::ostream &operator<<(std::ostream &out, escape_string const &se)
747 std::string const &s = se.input;
748 char const *quoted_char = ",: '";
749 char const *escaped_char = "\"\\$!";
750 bool need_quotes = false;
752 size_t len = s.length(), last = 0, j = 0;
753 for (size_t i = 0; i < len; ++i)
755 if (strchr(escaped_char, s[i]))
758 if (!buf) buf = new char[len * 2];
759 memcpy(&buf[j], &s[last], i - last);
765 if (!need_quotes && strchr(quoted_char, s[i]))
768 if (!need_quotes) return out << s;
770 if (!buf) return out << s << '"';
772 out.write(&s[last], len - last);
778 * @defgroup paths Path helpers
784 * Initialize #working_dir.
786 void init_working_dir(char *argv0, bool client_mode)
789 std::string cmd(argv0);
798 size_t pos = cmd.find_last_of(dirsep);
799 if (pos != std::string::npos)
801 std::string dir = cmd.substr(0, pos+1);
802 if (chdir(dir.c_str()) != 0)
804 perror("Failed to change working directory");
809 std::cout << "Entering directory `"
810 << dir << "'" << std::endl;
814 char *res = getcwd(buf, sizeof(buf));
817 perror("Failed to get working directory");
824 * Normalize an absolute path with respect to the working directory.
825 * Paths outside the working subtree are left unchanged.
827 static std::string normalize_abs(std::string const &s)
829 size_t l = working_dir.length();
830 if (s.compare(0, l, working_dir)) return s;
831 size_t ll = s.length();
832 if (ll == l) return ".";
835 size_t pos = s.rfind('/', l);
836 assert(pos != std::string::npos);
837 return s.substr(pos + 1);
839 if (ll == l + 1) return ".";
840 return s.substr(l + 1);
844 * Normalize a target name.
846 static std::string normalize(std::string const &s)
849 char const *delim = "/\\";
853 size_t prev = 0, len = s.length();
854 size_t pos = s.find_first_of(delim);
855 if (pos == std::string::npos) return s;
856 bool absolute = pos == 0;
862 std::string n = s.substr(prev, pos - prev);
865 if (!l.empty()) l.pop_back();
867 return normalize(working_dir + '/' + s);
873 if (pos >= len) break;
875 pos = s.find_first_of(delim, prev);
876 if (pos == std::string::npos) pos = len;
878 string_list::const_iterator i = l.begin(), i_end = l.end();
879 if (i == i_end) return absolute ? "/" : ".";
881 if (absolute) n.push_back('/');
883 for (++i; i != i_end; ++i)
888 if (absolute) return normalize_abs(n);
893 * Normalize the content of a list of targets.
895 static void normalize_list(string_list &l)
897 for (string_list::iterator i = l.begin(),
898 i_end = l.end(); i != i_end; ++i)
907 * @defgroup lexer Lexer
915 static void skip_spaces(std::istream &in)
918 while (strchr(" \t", (c = in.get()))) {}
919 if (in.good()) in.putback(c);
925 static void skip_empty(std::istream &in)
928 while (strchr("\r\n", (c = in.get()))) {}
929 if (in.good()) in.putback(c);
933 * Skip end of line. If @a multi is true, skip the following empty lines too.
934 * @return true if there was a line to end.
936 static bool skip_eol(std::istream &in, bool multi = false)
939 if (c == '\r') c = in.get();
940 if (c != '\n' && in.good()) in.putback(c);
941 if (c != '\n' && !in.eof()) return false;
942 if (multi) skip_empty(in);
959 * Skip spaces and peek at the next token.
960 * If it is one of @a mask, skip it (if it is not Word) and return it.
961 * @note For composite tokens allowed by @a mask, input characters might
962 * have been eaten even for an Unexpected result.
964 static int expect_token(std::istream &in, int mask)
970 if (!in.good()) return Unexpected;
975 case '\n': return Unexpected;
976 case ':': tok = Colon; break;
977 case ',': tok = Comma; break;
978 case '=': tok = Equal; break;
979 case ')': tok = Rightpar; break;
981 if (!(mask & Dollarpar)) return Unexpected;
984 if (in.peek() != '(') return Unexpected;
987 if (!(mask & Plusequal)) return Unexpected;
990 if (in.peek() != '=') return Unexpected;
994 if (skip_eol(in)) continue;
996 return mask & Word ? Word : Unexpected;
998 return mask & Word ? Word : Unexpected;
1000 if (!(tok & mask)) return Unexpected;
1007 * Read a (possibly quoted) word.
1009 static std::string read_word(std::istream &in)
1013 if (!in.good()) return res;
1014 char const *separators = " \t\r\n:$(),=+\"";
1015 bool quoted = c == '"';
1018 if (strchr(separators, c))
1028 if (!in.good()) return res;
1040 if (strchr(separators, c))
1053 * @defgroup stream Token streams
1059 * Possible results from word producers.
1069 * Interface for word producers.
1073 virtual ~generator() {}
1074 virtual input_status next(std::string &) = 0;
1078 * Generator for the words of a variable.
1080 struct variable_generator: generator
1083 string_list::const_iterator cur1, end1;
1084 assign_list::const_iterator cur2, end2;
1085 variable_generator(std::string const &, assign_list const *);
1086 input_status next(std::string &);
1089 variable_generator::variable_generator(std::string const &n,
1090 assign_list const *local_variables): name(n)
1093 if (local_variables)
1095 // Set cur2 to the last variable overwriter, if any.
1096 cur2 = local_variables->begin();
1097 end2 = local_variables->end();
1098 for (assign_list::const_iterator i = cur2; i != end2; ++i)
1100 if (i->name == name && !i->append)
1109 static assign_list dummy;
1110 cur2 = dummy.begin();
1113 static string_list dummy;
1114 cur1 = dummy.begin();
1118 variable_map::const_iterator i = variables.find(name);
1119 if (i == variables.end()) return;
1120 cur1 = i->second.begin();
1121 end1 = i->second.end();
1125 input_status variable_generator::next(std::string &res)
1134 while (cur2 != end2)
1136 if (cur2->name == name)
1138 cur1 = cur2->value.begin();
1139 end1 = cur2->value.end();
1149 * Generator for the words of an input stream.
1151 struct input_generator
1155 assign_list const *local_variables;
1156 bool earliest_exit, done;
1157 input_generator(std::istream &i, assign_list const *lv, bool e = false)
1158 : in(i), nested(NULL), local_variables(lv), earliest_exit(e), done(false) {}
1159 input_status next(std::string &);
1160 ~input_generator() { assert(!nested); }
1163 static generator *get_function(input_generator const &, std::string const &);
1165 input_status input_generator::next(std::string &res)
1170 input_status s = nested->next(res);
1171 if (s == Success) return Success;
1174 if (s == SyntaxError) return SyntaxError;
1176 if (done) return Eof;
1177 if (earliest_exit) done = true;
1178 switch (expect_token(in, Word | Dollarpar))
1181 res = read_word(in);
1185 std::string name = read_word(in);
1186 if (name.empty()) return SyntaxError;
1187 if (expect_token(in, Rightpar))
1188 nested = new variable_generator(name, local_variables);
1191 nested = get_function(*this, name);
1192 if (!nested) return SyntaxError;
1202 * Read a list of words from an input generator.
1203 * @return false if a syntax error was encountered.
1205 static bool read_words(input_generator &in, string_list &res)
1209 res.push_back(std::string());
1210 input_status s = in.next(res.back());
1211 if (s == Success) continue;
1217 static bool read_words(std::istream &in, string_list &res)
1219 input_generator gen(in, NULL);
1220 return read_words(gen, res);
1225 * Serialize a variable map.
1227 std::string serialize_variables(variable_map &vars, char sep = ' ')
1229 std::ostringstream buf;
1230 for(variable_map::const_iterator i = vars.begin(),
1231 i_end = vars.end(); i != i_end; ++i)
1233 buf << i->first << '=';
1236 for (string_list::const_iterator j = i->second.begin(),
1237 j_end = i->second.end(); j != j_end; ++j)
1239 if (first) first = false;
1243 buf << escape_string(val) << sep;
1245 std::string s = buf.str();
1250 * Generator for the result of function addprefix.
1252 struct addprefix_generator: generator
1254 input_generator gen;
1256 string_list::const_iterator prei;
1259 addprefix_generator(input_generator const &, bool &);
1260 input_status next(std::string &);
1263 addprefix_generator::addprefix_generator(input_generator const &top, bool &ok)
1264 : gen(top.in, top.local_variables)
1266 if (!read_words(gen, pre)) return;
1267 if (!expect_token(gen.in, Comma)) return;
1273 input_status addprefix_generator::next(std::string &res)
1290 switch (gen.next(res))
1293 if (!prel) return Success;
1299 return expect_token(gen.in, Rightpar) ? Eof : SyntaxError;
1306 * Generator for the result of function addsuffix.
1308 struct addsuffix_generator: generator
1310 input_generator gen;
1312 string_list::const_iterator sufi;
1315 addsuffix_generator(input_generator const &, bool &);
1316 input_status next(std::string &);
1319 addsuffix_generator::addsuffix_generator(input_generator const &top, bool &ok)
1320 : gen(top.in, top.local_variables)
1322 if (!read_words(gen, suf)) return;
1323 if (!expect_token(gen.in, Comma)) return;
1329 input_status addsuffix_generator::next(std::string &res)
1341 switch (gen.next(res))
1344 if (!sufl) return Success;
1350 return expect_token(gen.in, Rightpar) ? Eof : SyntaxError;
1357 * Return a generator for function @a name.
1359 generator *get_function(input_generator const &in, std::string const &name)
1362 generator *g = NULL;
1364 if (name == "addprefix") g = new addprefix_generator(in, ok);
1365 else if (name == "addsuffix") g = new addsuffix_generator(in, ok);
1366 if (!g || ok) return g;
1374 * @defgroup database Dependency database
1380 * Load dependencies from @a in.
1382 static void load_dependencies(std::istream &in)
1387 std::cerr << "Failed to load database" << std::endl;
1393 string_list targets;
1394 if (!read_words(in, targets)) goto error;
1395 if (in.eof()) return;
1396 if (targets.empty()) goto error;
1397 DEBUG << "reading dependencies of target " << targets.front() << std::endl;
1398 if (in.get() != ':') goto error;
1399 ref_ptr<dependency_t> dep;
1400 dep->targets = targets;
1402 if (!read_words(in, deps)) goto error;
1403 dep->deps.insert(deps.begin(), deps.end());
1404 for (string_list::const_iterator i = targets.begin(),
1405 i_end = targets.end(); i != i_end; ++i)
1407 dependencies[*i] = dep;
1414 * Load known dependencies from file <tt>.remake</tt>.
1416 static void load_dependencies()
1418 DEBUG_open << "Loading database... ";
1419 std::ifstream in(".remake");
1422 DEBUG_close << "not found\n";
1425 load_dependencies(in);
1430 * Save all the dependencies in file <tt>.remake</tt>.
1432 static void save_dependencies()
1434 DEBUG_open << "Saving database... ";
1435 std::ofstream db(".remake");
1436 while (!dependencies.empty())
1438 ref_ptr<dependency_t> dep = dependencies.begin()->second;
1439 for (string_list::const_iterator i = dep->targets.begin(),
1440 i_end = dep->targets.end(); i != i_end; ++i)
1442 db << escape_string(*i) << ' ';
1443 dependencies.erase(*i);
1446 for (string_set::const_iterator i = dep->deps.begin(),
1447 i_end = dep->deps.end(); i != i_end; ++i)
1449 db << ' ' << escape_string(*i);
1458 * @defgroup parser Rule parser
1464 * Register a specific rule with an empty script:
1466 * - Check that none of the targets already has an associated rule with a
1468 * - Create a new rule with a single target for each target, if needed.
1469 * - Add the prerequisites of @a rule to all these associated rules.
1471 static void register_transparent_rule(rule_t const &rule, string_list const &targets)
1473 assert(rule.script.empty());
1474 for (string_list::const_iterator i = targets.begin(),
1475 i_end = targets.end(); i != i_end; ++i)
1477 std::pair<rule_map::iterator, bool> j =
1478 specific_rules.insert(std::make_pair(*i, ref_ptr<rule_t>()));
1479 ref_ptr<rule_t> &r = j.first->second;
1482 r = ref_ptr<rule_t>(rule);
1483 r->targets = string_list(1, *i);
1486 if (!r->script.empty())
1488 std::cerr << "Failed to load rules: " << *i
1489 << " cannot be the target of several rules" << std::endl;
1492 assert(r->targets.size() == 1 && r->targets.front() == *i);
1493 r->deps.insert(r->deps.end(), rule.deps.begin(), rule.deps.end());
1494 r->vars.insert(r->vars.end(), rule.vars.begin(), rule.vars.end());
1497 for (string_list::const_iterator i = targets.begin(),
1498 i_end = targets.end(); i != i_end; ++i)
1500 ref_ptr<dependency_t> &dep = dependencies[*i];
1501 if (dep->targets.empty()) dep->targets.push_back(*i);
1502 dep->deps.insert(rule.deps.begin(), rule.deps.end());
1507 * Register a specific rule with a nonempty script:
1509 * - Check that none of the targets already has an associated rule.
1510 * - Create a single shared rule and associate it to all the targets.
1511 * - Merge the prerequisites of all the targets into a single set and
1512 * add the prerequisites of the rule to it. (The preexisting
1513 * prerequisites, if any, come from a previous run.)
1515 static void register_scripted_rule(rule_t const &rule)
1517 ref_ptr<rule_t> r(rule);
1518 for (string_list::const_iterator i = rule.targets.begin(),
1519 i_end = rule.targets.end(); i != i_end; ++i)
1521 std::pair<rule_map::iterator, bool> j =
1522 specific_rules.insert(std::make_pair(*i, r));
1523 if (j.second) continue;
1524 std::cerr << "Failed to load rules: " << *i
1525 << " cannot be the target of several rules" << std::endl;
1529 ref_ptr<dependency_t> dep;
1530 dep->targets = rule.targets;
1531 dep->deps.insert(rule.deps.begin(), rule.deps.end());
1532 for (string_list::const_iterator i = rule.targets.begin(),
1533 i_end = rule.targets.end(); i != i_end; ++i)
1535 ref_ptr<dependency_t> &d = dependencies[*i];
1536 dep->deps.insert(d->deps.begin(), d->deps.end());
1542 * Read a rule starting with target @a first, if nonempty.
1543 * Store into #generic_rules or #specific_rules depending on its genericity.
1545 static void load_rule(std::istream &in, std::string const &first)
1547 DEBUG_open << "Reading rule for target " << first << "... ";
1551 DEBUG_close << "failed\n";
1552 std::cerr << "Failed to load rules: syntax error" << std::endl;
1557 // Read targets and check genericity.
1558 string_list targets;
1559 if (!read_words(in, targets)) goto error;
1560 if (!first.empty()) targets.push_front(first);
1561 else if (targets.empty()) goto error;
1562 else DEBUG << "actual target: " << targets.front() << std::endl;
1563 bool generic = false;
1564 normalize_list(targets);
1565 for (string_list::const_iterator i = targets.begin(),
1566 i_end = targets.end(); i != i_end; ++i)
1568 if (i->empty()) goto error;
1569 if ((i->find('%') != std::string::npos) != generic)
1571 if (i == targets.begin()) generic = true;
1575 std::swap(rule.targets, targets);
1577 if (in.get() != ':') goto error;
1579 bool assignment = false;
1581 // Read dependencies.
1582 if (expect_token(in, Word))
1584 std::string d = read_word(in);
1585 if (int tok = expect_token(in, Equal | Plusequal))
1587 rule.vars.push_back(assign_t());
1589 if (!read_words(in, v)) goto error;
1590 assign_t &a = rule.vars.back();
1592 a.append = tok == Plusequal;
1599 if (!read_words(in, v)) goto error;
1608 if (!read_words(in, v)) goto error;
1613 if (!skip_eol(in, true)) goto error;
1616 std::ostringstream buf;
1620 if (!in.good()) break;
1621 if (c == '\t' || c == ' ')
1623 in.get(*buf.rdbuf());
1624 if (in.fail() && !in.eof()) in.clear();
1626 else if (c == '\r' || c == '\n')
1634 rule.script = buf.str();
1636 // Add generic rules to the correct set.
1639 if (assignment) goto error;
1640 generic_rules.push_back(rule);
1644 if (!rule.script.empty())
1646 if (assignment) goto error;
1647 register_scripted_rule(rule);
1651 // Swap away the targets to avoid costly copies when registering.
1652 string_list targets;
1653 std::swap(rule.targets, targets);
1654 register_transparent_rule(rule, targets);
1655 std::swap(rule.targets, targets);
1658 // If there is no default target yet, mark it as such.
1659 if (first_target.empty())
1660 first_target = rule.targets.front();
1664 * Load rules from @a remakefile.
1665 * If some rules have dependencies and non-generic targets, add these
1666 * dependencies to the targets.
1668 static void load_rules(std::string const &remakefile)
1670 DEBUG_open << "Loading rules... ";
1674 std::cerr << "Failed to load rules: syntax error" << std::endl;
1677 std::ifstream in(remakefile.c_str());
1680 std::cerr << "Failed to load rules: no Remakefile found" << std::endl;
1691 while (in.get() != '\n') {}
1695 if (c == ' ' || c == '\t') goto error;
1696 if (expect_token(in, Word))
1698 std::string name = read_word(in);
1699 if (name.empty()) goto error;
1700 if (int tok = expect_token(in, Equal | Plusequal))
1702 DEBUG << "Assignment to variable " << name << std::endl;
1704 if (!read_words(in, value)) goto error;
1705 string_list &dest = variables[name];
1706 if (tok == Equal) dest.swap(value);
1707 else dest.splice(dest.end(), value);
1708 if (!skip_eol(in, true)) goto error;
1710 else load_rule(in, name);
1712 else load_rule(in, std::string());
1719 * @defgroup rules Rule resolution
1725 * Substitute a pattern into a list of strings.
1727 static void substitute_pattern(std::string const &pat, string_list const &src, string_list &dst)
1729 for (string_list::const_iterator i = src.begin(),
1730 i_end = src.end(); i != i_end; ++i)
1732 size_t pos = i->find('%');
1733 if (pos == std::string::npos)dst.push_back(*i);
1734 else dst.push_back(i->substr(0, pos) + pat + i->substr(pos + 1));
1739 * Find a generic rule matching @a target:
1740 * - the one leading to shorter matches has priority,
1741 * - among equivalent rules, the earliest one has priority.
1743 static rule_t find_generic_rule(std::string const &target)
1745 size_t tlen = target.length(), plen = tlen + 1;
1747 for (rule_list::const_iterator i = generic_rules.begin(),
1748 i_end = generic_rules.end(); i != i_end; ++i)
1750 for (string_list::const_iterator j = i->targets.begin(),
1751 j_end = i->targets.end(); j != j_end; ++j)
1753 size_t len = j->length();
1754 if (tlen < len) continue;
1755 if (plen <= tlen - (len - 1)) continue;
1756 size_t pos = j->find('%');
1757 if (pos == std::string::npos) continue;
1758 size_t len2 = len - (pos + 1);
1759 if (j->compare(0, pos, target, 0, pos) ||
1760 j->compare(pos + 1, len2, target, tlen - len2, len2))
1762 plen = tlen - (len - 1);
1764 rule.stem = target.substr(pos, plen);
1765 rule.script = i->script;
1766 substitute_pattern(rule.stem, i->targets, rule.targets);
1767 substitute_pattern(rule.stem, i->deps, rule.deps);
1775 * Find a specific rule matching @a target. Return a generic one otherwise.
1776 * If there is both a specific rule with an empty script and a generic rule, the
1777 * generic one is returned after adding the dependencies of the specific one.
1779 static rule_t find_rule(std::string const &target)
1781 rule_map::const_iterator i = specific_rules.find(target),
1782 i_end = specific_rules.end();
1783 // If there is a specific rule with a script, return it.
1784 if (i != i_end && !i->second->script.empty()) return *i->second;
1785 rule_t grule = find_generic_rule(target);
1786 // If there is no generic rule, return the specific rule (no script), if any.
1787 if (grule.targets.empty())
1789 if (i != i_end) return *i->second;
1792 // Optimize the lookup when there is only one target (already looked up).
1793 if (grule.targets.size() == 1)
1795 if (i == i_end) return grule;
1796 grule.deps.insert(grule.deps.end(),
1797 i->second->deps.begin(), i->second->deps.end());
1798 grule.vars.insert(grule.vars.end(),
1799 i->second->vars.begin(), i->second->vars.end());
1802 // Add the dependencies of the specific rules of every target to the
1803 // generic rule. If any of those rules has a nonempty script, error out.
1804 for (string_list::const_iterator j = grule.targets.begin(),
1805 j_end = grule.targets.end(); j != j_end; ++j)
1807 i = specific_rules.find(*j);
1808 if (i == i_end) continue;
1809 if (!i->second->script.empty()) return rule_t();
1810 grule.deps.insert(grule.deps.end(),
1811 i->second->deps.begin(), i->second->deps.end());
1812 grule.vars.insert(grule.vars.end(),
1813 i->second->vars.begin(), i->second->vars.end());
1821 * @defgroup status Target status
1827 * Compute and memoize the status of @a target:
1828 * - if the file does not exist, the target is obsolete,
1829 * - if any dependency is obsolete or younger than the file, it is obsolete,
1830 * - otherwise it is up-to-date.
1832 * @note For rules with multiple targets, all the targets share the same
1833 * status. (If one is obsolete, they all are.) The second rule above
1834 * is modified in that case: the latest target is chosen, not the oldest!
1836 static status_t const &get_status(std::string const &target)
1838 std::pair<status_map::iterator,bool> i =
1839 status.insert(std::make_pair(target, status_t()));
1840 status_t &ts = i.first->second;
1841 if (!i.second) return ts;
1842 DEBUG_open << "Checking status of " << target << "... ";
1843 dependency_map::const_iterator j = dependencies.find(target);
1844 if (j == dependencies.end())
1847 if (stat(target.c_str(), &s) != 0)
1849 DEBUG_close << "missing\n";
1854 DEBUG_close << "up-to-date\n";
1855 ts.status = Uptodate;
1856 ts.last = s.st_mtime;
1859 dependency_t const &dep = *j->second;
1860 status_e st = Uptodate;
1862 for (string_list::const_iterator k = dep.targets.begin(),
1863 k_end = dep.targets.end(); k != k_end; ++k)
1866 if (stat(k->c_str(), &s) != 0)
1868 if (st == Uptodate) DEBUG_close << *k << " missing\n";
1872 status[*k].last = s.st_mtime;
1873 if (s.st_mtime > latest) latest = s.st_mtime;
1875 if (st == Todo) goto update;
1876 for (string_set::const_iterator k = dep.deps.begin(),
1877 k_end = dep.deps.end(); k != k_end; ++k)
1879 status_t const &ts_ = get_status(*k);
1880 if (latest < ts_.last)
1882 DEBUG_close << "older than " << *k << std::endl;
1886 if (ts_.status == Uptodate) continue;
1888 DEBUG << "obsolete dependency " << *k << std::endl;
1891 if (st == Uptodate) DEBUG_close << "all siblings up-to-date\n";
1893 for (string_list::const_iterator k = dep.targets.begin(),
1894 k_end = dep.targets.end(); k != k_end; ++k)
1896 status[*k].status = st;
1902 * Change the status of @a target to #Remade or #Uptodate depending on whether
1903 * its modification time changed.
1905 static void update_status(std::string const &target)
1907 DEBUG_open << "Rechecking status of " << target << "... ";
1908 status_map::iterator i = status.find(target);
1909 assert(i != status.end());
1910 status_t &ts = i->second;
1914 DEBUG_close << "possibly remade\n";
1918 if (stat(target.c_str(), &s) != 0)
1920 DEBUG_close << "missing\n";
1923 else if (s.st_mtime != ts.last)
1925 DEBUG_close << "remade\n";
1926 ts.last = s.st_mtime;
1930 DEBUG_close << "unchanged\n";
1931 ts.status = Uptodate;
1936 * Check whether all the prerequisites of @a target ended being up-to-date.
1938 static bool still_need_rebuild(std::string const &target)
1940 DEBUG_open << "Rechecking obsoleteness of " << target << "... ";
1941 status_map::const_iterator i = status.find(target);
1942 assert(i != status.end());
1943 if (i->second.status != Recheck) return true;
1944 dependency_map::const_iterator j = dependencies.find(target);
1945 assert(j != dependencies.end());
1946 dependency_t const &dep = *j->second;
1947 for (string_set::const_iterator k = dep.deps.begin(),
1948 k_end = dep.deps.end(); k != k_end; ++k)
1950 if (status[*k].status != Uptodate) return true;
1952 for (string_list::const_iterator k = dep.targets.begin(),
1953 k_end = dep.targets.end(); k != k_end; ++k)
1955 status[*k].status = Uptodate;
1957 DEBUG_close << "no longer obsolete\n";
1964 * @defgroup server Server
1970 * Handle job completion.
1972 static void complete_job(int job_id, bool success)
1974 DEBUG_open << "Completing job " << job_id << "... ";
1975 job_targets_map::iterator i = job_targets.find(job_id);
1976 assert(i != job_targets.end());
1977 string_list const &targets = i->second;
1980 for (string_list::const_iterator j = targets.begin(),
1981 j_end = targets.end(); j != j_end; ++j)
1988 DEBUG_close << "failed\n";
1989 std::cerr << "Failed to build";
1990 for (string_list::const_iterator j = targets.begin(),
1991 j_end = targets.end(); j != j_end; ++j)
1993 status[*j].status = Failed;
1994 std::cerr << ' ' << *j;
1997 std::cerr << std::endl;
1999 job_targets.erase(i);
2000 job_variables.erase(job_variables.find(job_id));
2004 * Return the script obtained by substituting variables.
2006 static std::string prepare_script(rule_t const &rule)
2008 std::string const &s = rule.script;
2009 std::istringstream in(s);
2010 std::ostringstream out;
2011 size_t len = s.size();
2015 size_t pos = in.tellg(), p = s.find('$', pos);
2016 if (p == std::string::npos || p == len - 1) p = len;
2017 out.write(&s[pos], p - pos);
2018 if (p == len) break;
2027 if (!rule.deps.empty())
2028 out << rule.deps.front();
2034 for (string_list::const_iterator i = rule.deps.begin(),
2035 i_end = rule.deps.end(); i != i_end; ++i)
2037 if (first) first = false;
2045 assert(!rule.targets.empty());
2046 out << rule.targets.front();
2057 input_generator gen(in, &rule.vars, true);
2061 input_status s = gen.next(w);
2062 if (s == SyntaxError)
2067 if (s == Eof) break;
2068 if (first) first = false;
2075 // Let dollars followed by an unrecognized character
2076 // go through. This differs from Make, which would
2077 // use a one-letter variable.
2087 * Execute the script from @a rule.
2089 static bool run_script(int job_id, rule_t /*const */ &rule)
2093 std::cout << "Building";
2094 for (string_list::const_iterator i = rule.targets.begin(),
2095 i_end = rule.targets.end(); i != i_end; ++i)
2097 std::cout << ' ' << *i;
2099 std::cout << std::endl;
2102 ref_ptr<dependency_t> dep;
2103 dep->targets = rule.targets;
2104 dep->deps.insert(rule.deps.begin(), rule.deps.end());
2105 for (string_list::const_iterator i = rule.targets.begin(),
2106 i_end = rule.targets.end(); i != i_end; ++i)
2108 dependencies[*i] = dep;
2110 variable_map job_vars = job_variables[job_id];
2111 for(variable_map::const_iterator i = job_vars.begin(),
2112 i_end = job_vars.end(); i != i_end; ++i)
2114 assign_t a = assign_t();
2117 a.value = i->second;
2118 rule.vars.push_back(a);
2121 std::string script = prepare_script(rule);
2123 std::ostringstream job_id_buf;
2124 job_id_buf << job_id;
2125 std::string job_id_ = job_id_buf.str();
2127 DEBUG_open << "Starting script for job " << job_id << "... ";
2131 DEBUG_close << "failed\n";
2132 complete_job(job_id, false);
2141 CloseHandle(pfd[0]);
2142 CloseHandle(pfd[1]);
2145 if (!CreatePipe(&pfd[0], &pfd[1], NULL, 0))
2147 if (!SetHandleInformation(pfd[0], HANDLE_FLAG_INHERIT, HANDLE_FLAG_INHERIT))
2150 ZeroMemory(&si, sizeof(STARTUPINFO));
2151 si.cb = sizeof(STARTUPINFO);
2152 si.hStdError = GetStdHandle(STD_ERROR_HANDLE);
2153 si.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
2154 si.hStdInput = pfd[0];
2155 si.dwFlags |= STARTF_USESTDHANDLES;
2156 PROCESS_INFORMATION pi;
2157 ZeroMemory(&pi, sizeof(PROCESS_INFORMATION));
2158 if (!SetEnvironmentVariable("REMAKE_JOB_ID", job_id_.c_str()))
2160 char const *argv = echo_scripts ? "SH.EXE -e -s -v" : "SH.EXE -e -s";
2161 if (!CreateProcess(NULL, (char *)argv, NULL, NULL,
2162 true, 0, NULL, NULL, &si, &pi))
2166 CloseHandle(pi.hThread);
2167 DWORD len = script.length(), wlen;
2168 if (!WriteFile(pfd[1], script.c_str(), len, &wlen, NULL) || wlen < len)
2169 std::cerr << "Unexpected failure while sending script to shell" << std::endl;
2170 CloseHandle(pfd[0]);
2171 CloseHandle(pfd[1]);
2173 job_pids[pi.hProcess] = job_id;
2184 if (pipe(pfd) == -1)
2186 if (setenv("REMAKE_JOB_ID", job_id_.c_str(), 1))
2188 if (pid_t pid = vfork())
2190 if (pid == -1) goto error2;
2191 ssize_t len = script.length();
2192 if (write(pfd[1], script.c_str(), len) < len)
2193 std::cerr << "Unexpected failure while sending script to shell" << std::endl;
2197 job_pids[pid] = job_id;
2200 // Child process starts here. Notice the use of vfork above.
2201 char const *argv[5] = { "sh", "-e", "-s", NULL, NULL };
2202 if (echo_scripts) argv[3] = "-v";
2209 execve("/bin/sh", (char **)argv, environ);
2210 _exit(EXIT_FAILURE);
2215 * Create a job for @a target according to the loaded rules.
2216 * Mark all the targets from the rule as running and reset their dependencies.
2217 * If the rule has dependencies, create a new client to build them just
2218 * before @a current, and change @a current so that it points to it.
2220 static bool start(std::string const &target, client_list::iterator ¤t)
2222 DEBUG_open << "Starting job " << job_counter << " for " << target << "... ";
2223 rule_t rule = find_rule(target);
2224 if (rule.targets.empty())
2226 status[target].status = Failed;
2227 DEBUG_close << "failed\n";
2228 std::cerr << "No rule for building " << target << std::endl;
2231 for (string_list::const_iterator i = rule.targets.begin(),
2232 i_end = rule.targets.end(); i != i_end; ++i)
2234 status[*i].status = Running;
2236 int job_id = job_counter++;
2237 job_targets[job_id] = rule.targets;
2238 DEBUG << "Setting variables of job: " << job_id << " (spawn by: "
2239 << current->job_id << ") to "
2240 << serialize_variables(job_variables[current->job_id], ' ')
2242 job_variables[job_id] = job_variables[current->job_id];
2243 if (!rule.deps.empty())
2245 current = clients.insert(current, client_t());
2246 current->job_id = job_id;
2247 current->pending = rule.deps;
2248 current->delayed = new rule_t(rule);
2251 return run_script(job_id, rule);
2255 * Send a reply to a client then remove it.
2256 * If the client was a dependency client, start the actual script.
2258 static void complete_request(client_t &client, bool success)
2260 DEBUG_open << "Completing request from client of job " << client.job_id << "... ";
2263 assert(client.socket == INVALID_SOCKET);
2266 if (still_need_rebuild(client.delayed->targets.front()))
2267 run_script(client.job_id, *client.delayed);
2268 else complete_job(client.job_id, true);
2270 else complete_job(client.job_id, false);
2271 delete client.delayed;
2273 else if (client.socket != INVALID_SOCKET)
2275 char res = success ? 1 : 0;
2276 send(client.socket, &res, 1, 0);
2278 closesocket(client.socket);
2280 close(client.socket);
2285 if (client.job_id < 0 && !success) build_failure = true;
2289 * Return whether there are slots for starting new jobs.
2291 static bool has_free_slots()
2293 if (max_active_jobs <= 0) return true;
2294 return running_jobs - waiting_jobs < max_active_jobs;
2298 * Handle client requests:
2299 * - check for running targets that have finished,
2300 * - start as many pending targets as allowed,
2301 * - complete the request if there are neither running nor pending targets
2302 * left or if any of them failed.
2304 * @return true if some child processes are still running.
2306 * @post If there are pending requests, at least one child process is running.
2308 static bool handle_clients()
2310 DEBUG_open << "Handling client requests... ";
2313 for (client_list::iterator i = clients.begin(), i_next = i,
2314 i_end = clients.end(); i != i_end && has_free_slots(); i = i_next)
2317 DEBUG_open << "Handling client from job " << i->job_id << "... ";
2321 complete_request(*i, false);
2323 DEBUG_close << "failed\n";
2327 // Remove running targets that have finished.
2328 for (string_set::iterator j = i->running.begin(), j_next = j,
2329 j_end = i->running.end(); j != j_end; j = j_next)
2332 status_map::const_iterator k = status.find(*j);
2333 assert(k != status.end());
2334 switch (k->second.status)
2339 if (!keep_going) goto failed;
2344 i->running.erase(j);
2352 // Start pending targets.
2353 while (!i->pending.empty())
2355 std::string target = i->pending.front();
2356 i->pending.pop_front();
2357 switch (get_status(target).status)
2360 i->running.insert(target);
2364 if (!keep_going) goto failed;
2372 client_list::iterator j = i;
2373 if (!start(target, i)) goto pending_failed;
2374 j->running.insert(target);
2375 if (!has_free_slots()) return true;
2376 // Job start might insert a dependency client.
2383 // Try to complete the request.
2384 // (This might start a new job if it was a dependency client.)
2385 if (i->running.empty())
2387 if (i->failed) goto failed;
2388 complete_request(*i, true);
2390 DEBUG_close << "finished\n";
2395 if (running_jobs != waiting_jobs) return true;
2396 if (running_jobs == 0 && clients.empty()) return false;
2398 // There is a circular dependency.
2399 // Try to break it by completing one of the requests.
2400 assert(!clients.empty());
2401 std::cerr << "Circular dependency detected" << std::endl;
2402 client_list::iterator i = clients.begin();
2403 complete_request(*i, false);
2409 * Create a named unix socket that listens for build requests. Also set
2410 * the REMAKE_SOCKET environment variable that will be inherited by all
2413 static void create_server()
2418 perror("Failed to create server");
2424 DEBUG_open << "Creating server... ";
2427 // Prepare a windows socket.
2428 struct sockaddr_in socket_addr;
2429 socket_addr.sin_family = AF_INET;
2430 socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
2431 socket_addr.sin_port = 0;
2433 // Create and listen to the socket.
2434 socket_fd = socket(AF_INET, SOCK_STREAM, 0);
2435 if (socket_fd == INVALID_SOCKET) goto error;
2436 if (!SetHandleInformation((HANDLE)socket_fd, HANDLE_FLAG_INHERIT, 0))
2438 if (bind(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
2440 int len = sizeof(sockaddr_in);
2441 if (getsockname(socket_fd, (struct sockaddr *)&socket_addr, &len))
2443 std::ostringstream buf;
2444 buf << socket_addr.sin_port;
2445 if (!SetEnvironmentVariable("REMAKE_SOCKET", buf.str().c_str()))
2447 if (listen(socket_fd, 1000)) goto error;
2449 // Set signal handlers for SIGCHLD and SIGINT.
2450 // Block SIGCHLD (unblocked during select).
2452 sigemptyset(&sigmask);
2453 sigaddset(&sigmask, SIGCHLD);
2454 if (sigprocmask(SIG_BLOCK, &sigmask, NULL) == -1) goto error;
2455 struct sigaction sa;
2457 sigemptyset(&sa.sa_mask);
2458 sa.sa_handler = &sigchld_handler;
2459 if (sigaction(SIGCHLD, &sa, NULL) == -1) goto error;
2460 sa.sa_handler = &sigint_handler;
2461 if (sigaction(SIGINT, &sa, NULL) == -1) goto error;
2463 // Prepare a named unix socket in temporary directory.
2464 socket_name = tempnam(NULL, "rmk-");
2465 if (!socket_name) goto error2;
2466 struct sockaddr_un socket_addr;
2467 size_t len = strlen(socket_name);
2468 if (len >= sizeof(socket_addr.sun_path) - 1) goto error2;
2469 socket_addr.sun_family = AF_UNIX;
2470 strcpy(socket_addr.sun_path, socket_name);
2471 len += sizeof(socket_addr.sun_family);
2472 if (setenv("REMAKE_SOCKET", socket_name, 1)) goto error;
2474 // Create and listen to the socket.
2476 socket_fd = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
2477 if (socket_fd == INVALID_SOCKET) goto error;
2479 socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
2480 if (socket_fd == INVALID_SOCKET) goto error;
2481 if (fcntl(socket_fd, F_SETFD, FD_CLOEXEC) < 0) goto error;
2483 if (bind(socket_fd, (struct sockaddr *)&socket_addr, len))
2485 if (listen(socket_fd, 1000)) goto error;
2490 * Accept a connection from a client, get the job it spawned from,
2491 * get the targets, and mark them as dependencies of the job targets.
2493 void accept_client()
2495 DEBUG_open << "Handling client request... ";
2497 // Accept connection.
2499 socket_t fd = accept(socket_fd, NULL, NULL);
2500 if (fd == INVALID_SOCKET) return;
2501 if (!SetHandleInformation((HANDLE)fd, HANDLE_FLAG_INHERIT, 0))
2504 std::cerr << "Unexpected failure while setting connection with client" << std::endl;
2508 // WSAEventSelect puts sockets into nonblocking mode, so disable it here.
2510 if (ioctlsocket(fd, FIONBIO, &nbio)) goto error2;
2511 #elif defined(LINUX)
2512 int fd = accept4(socket_fd, NULL, NULL, SOCK_CLOEXEC);
2515 int fd = accept(socket_fd, NULL, NULL);
2517 if (fcntl(fd, F_SETFD, FD_CLOEXEC) < 0) return;
2519 clients.push_front(client_t());
2520 client_list::iterator proc = clients.begin();
2525 DEBUG_close << "failed\n";
2526 std::cerr << "Received an ill-formed client message" << std::endl;
2532 clients.erase(proc);
2536 // Receive message. Stop when encountering two nuls in a row.
2537 std::vector<char> buf;
2539 while (len < sizeof(int) + 2 || buf[len - 1] || buf[len - 2])
2541 buf.resize(len + 1024);
2542 ssize_t l = recv(fd, &buf[0] + len, 1024, 0);
2543 if (l <= 0) goto error;
2547 // Parse job that spawned the client.
2549 memcpy(&job_id, &buf[0], sizeof(int));
2551 proc->job_id = job_id;
2552 job_targets_map::const_iterator i = job_targets.find(job_id);
2553 if (i == job_targets.end()) goto error;
2554 DEBUG << "receiving request from job " << job_id << std::endl;
2556 // Parse the targets and mark them as dependencies from the job targets.
2557 dependency_t &dep = *dependencies[job_targets[job_id].front()];
2558 // Retrieve the variables of the job, pass them to the client.
2559 //proc->variables = job_variables[job_id];
2561 variable_map &vars = job_variables[job_id];
2563 char const *p = &buf[0] + sizeof(int);
2567 if (len == 1 && p[0] == 1)
2569 //Finished parsing targets.
2574 std::string target(p, p + len);
2575 DEBUG << "adding dependency " << target << " to job\n";
2576 proc->pending.push_back(target);
2577 dep.deps.insert(target);
2584 if (len == 0) return;
2585 std::string line(p, p + len);
2586 std::istringstream in (line);
2587 std::string name = read_word(in);
2588 if (expect_token(in, Equal) == Unexpected) {
2589 std::cerr << '\'' << line << "'" << std::endl;
2592 DEBUG << "adding variable " << line << " to job " << job_id << std::endl;
2594 read_words(in, vars[name]);
2600 * Handle child process exit status.
2602 void finalize_job(pid_t pid, bool res)
2604 pid_job_map::iterator i = job_pids.find(pid);
2605 assert(i != job_pids.end());
2606 int job_id = i->second;
2609 complete_job(job_id, res);
2613 * Loop until all the jobs have finished.
2615 * @post There are no client requests left, not even virtual ones.
2619 while (handle_clients())
2621 DEBUG_open << "Handling events... ";
2623 size_t len = job_pids.size() + 1;
2626 for (pid_job_map::const_iterator i = job_pids.begin(),
2627 i_end = job_pids.end(); i != i_end; ++i, ++num)
2631 WSAEVENT aev = WSACreateEvent();
2633 WSAEventSelect(socket_fd, aev, FD_ACCEPT);
2634 DWORD w = WaitForMultipleObjects(len, h, false, INFINITE);
2635 WSAEventSelect(socket_fd, aev, 0);
2646 bool res = GetExitCodeProcess(pid, &s) && s == 0;
2648 finalize_job(pid, res);
2651 sigemptyset(&emptymask);
2654 FD_SET(socket_fd, &fdset);
2655 int ret = pselect(socket_fd + 1, &fdset, NULL, NULL, NULL, &emptymask);
2656 if (ret > 0 /* && FD_ISSET(socket_fd, &fdset)*/) accept_client();
2657 if (!got_SIGCHLD) continue;
2661 while ((pid = waitpid(-1, &status, WNOHANG)) > 0)
2663 bool res = WIFEXITED(status) && WEXITSTATUS(status) == 0;
2664 finalize_job(pid, res);
2669 assert(clients.empty());
2673 * Load dependencies and rules, listen to client requests, and loop until
2674 * all the requests have completed.
2675 * If Remakefile is obsolete, perform a first run with it only, then reload
2676 * the rules, and perform a second with the original clients.
2678 void server_mode(std::string const &remakefile, string_list const &targets)
2680 load_dependencies();
2681 load_rules(remakefile);
2683 if (get_status(remakefile).status != Uptodate)
2685 clients.push_back(client_t());
2686 clients.back().pending.push_back(remakefile);
2688 if (build_failure) goto early_exit;
2690 specific_rules.clear();
2691 generic_rules.clear();
2692 first_target.clear();
2693 load_rules(remakefile);
2695 clients.push_back(client_t());
2696 if (!targets.empty()) clients.back().pending = targets;
2697 else if (!first_target.empty())
2698 clients.back().pending.push_back(first_target);
2703 remove(socket_name);
2706 save_dependencies();
2707 exit(build_failure ? EXIT_FAILURE : EXIT_SUCCESS);
2713 * @defgroup client Client
2719 * Connect to the server @a socket_name, send a build request for @a targets,
2720 * and exit with the status returned by the server.
2722 void client_mode(char *socket_name, string_list const &targets)
2727 perror("Failed to send targets to server");
2730 if (targets.empty()) exit(EXIT_SUCCESS);
2731 DEBUG_open << "Connecting to server... ";
2733 // Connect to server.
2735 struct sockaddr_in socket_addr;
2736 socket_fd = socket(AF_INET, SOCK_STREAM, 0);
2737 if (socket_fd == INVALID_SOCKET) goto error;
2738 socket_addr.sin_family = AF_INET;
2739 socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
2740 socket_addr.sin_port = atoi(socket_name);
2741 if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
2744 struct sockaddr_un socket_addr;
2745 size_t len = strlen(socket_name);
2746 if (len >= sizeof(socket_addr.sun_path) - 1) exit(EXIT_FAILURE);
2747 socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
2748 if (socket_fd == INVALID_SOCKET) goto error;
2749 socket_addr.sun_family = AF_UNIX;
2750 strcpy(socket_addr.sun_path, socket_name);
2751 if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(socket_addr.sun_family) + len))
2755 if (setsockopt(socket_fd, SOL_SOCKET, SO_NOSIGPIPE, &set_option, sizeof(set_option)))
2760 // Send current job id.
2761 char *id = getenv("REMAKE_JOB_ID");
2762 int job_id = id ? atoi(id) : -1;
2763 if (send(socket_fd, (char *)&job_id, sizeof(job_id), MSG_NOSIGNAL) != sizeof(job_id))
2767 for (string_list::const_iterator i = targets.begin(),
2768 i_end = targets.end(); i != i_end; ++i)
2770 DEBUG_open << "Sending " << *i << "... ";
2771 ssize_t len = i->length() + 1;
2772 if (send(socket_fd, i->c_str(), len, MSG_NOSIGNAL) != len)
2775 // send 10 as as separator between targets and variables.
2777 if (send(socket_fd, &result, 1, MSG_NOSIGNAL) != 1) goto error;
2779 if (send(socket_fd, &result, 1, MSG_NOSIGNAL) != 1) goto error;
2781 // (maybe split vars in small chunks and send... seems overkil)
2782 std::string vars = serialize_variables(variables, 0);
2783 ssize_t sent = vars.size();
2784 DEBUG << "Sending variables: '" << vars << "' to the server" << std::endl;
2785 if (send(socket_fd, vars.data(), sent, MSG_NOSIGNAL) != sent)
2788 // Send terminating nul and wait for reply.
2790 if (send(socket_fd, &result, 1, MSG_NOSIGNAL) != 1) goto error;
2791 if (recv(socket_fd, &result, 1, 0) != 1) exit(EXIT_FAILURE);
2792 exit(result ? EXIT_SUCCESS : EXIT_FAILURE);
2798 * @defgroup ui User interface
2804 * Display usage and exit with @a exit_status.
2806 void usage(int exit_status)
2808 std::cerr << "Usage: remake [options] [target] ...\n"
2810 " -d Echo script commands.\n"
2811 " -d -d Print lots of debugging information.\n"
2812 " -f FILE Read FILE as Remakefile.\n"
2813 " -h, --help Print this message and exit.\n"
2814 " -j[N], --jobs=[N] Allow N jobs at once; infinite jobs with no arg.\n"
2815 " -k Keep going when some targets cannot be made.\n"
2816 " -v V=X, --var V=X Initialize variable V with X.\n"
2817 " -r Look up targets from the dependencies on stdin.\n"
2818 " -s, --silent, --quiet Do not echo targets.\n";
2823 * This program behaves in two different ways.
2825 * - If the environment contains the REMAKE_SOCKET variable, the client
2826 * connects to this socket and sends to the server its build targets.
2827 * It exits once it receives the server reply.
2829 * - Otherwise, it creates a server that waits for build requests. It
2830 * also creates a pseudo-client that requests the targets passed on the
2833 int main(int argc, char *argv[])
2835 char *sn = getenv("REMAKE_SOCKET");
2836 init_working_dir(argv[0], sn);
2838 std::string remakefile = "Remakefile";
2839 string_list targets;
2840 bool indirect_targets = false;
2842 // Parse command-line arguments.
2843 for (int i = 1; i < argc; ++i)
2845 std::string arg = argv[i];
2846 if (arg.empty()) usage(EXIT_FAILURE);
2847 if (arg == "-h" || arg == "--help") usage(EXIT_SUCCESS);
2849 if (echo_scripts) debug.active = true;
2850 else echo_scripts = true;
2851 else if (arg == "-k" || arg =="--keep-going")
2853 else if (arg == "-s" || arg == "--silent" || arg == "--quiet")
2854 show_targets = false;
2855 else if (arg == "-r")
2856 indirect_targets = true;
2857 else if (arg == "-f")
2859 if (++i == argc) usage(EXIT_FAILURE);
2860 remakefile = argv[i];
2862 else if (arg.compare(0, 2, "-j") == 0)
2863 max_active_jobs = atoi(arg.c_str() + 2);
2864 else if (arg.compare(0, 7, "--jobs=") == 0)
2865 max_active_jobs = atoi(arg.c_str() + 7);
2866 else if (arg == "-v" || arg == "--var")
2869 if (i == argc) usage(EXIT_FAILURE);
2871 std::istringstream in (arg);
2872 std::string name = read_word(in);
2873 if (expect_token(in, Equal) == Unexpected) {
2874 std::cerr << "Invalid variable '" << arg << "'" << std::endl;
2877 read_words(in, variables[name]);
2881 if (arg[0] == '-') usage(EXIT_FAILURE);
2882 targets.push_back(normalize(arg));
2883 DEBUG << "New target: " << arg << '\n';
2887 if (indirect_targets)
2889 load_dependencies(std::cin);
2892 if (l.empty() && !dependencies.empty())
2894 l.push_back(dependencies.begin()->second->targets.front());
2896 for (string_list::const_iterator i = l.begin(),
2897 i_end = l.end(); i != i_end; ++i)
2899 dependency_map::const_iterator j = dependencies.find(*i);
2900 if (j == dependencies.end()) continue;
2901 dependency_t const &dep = *j->second;
2902 for (string_set::const_iterator k = dep.deps.begin(),
2903 k_end = dep.deps.end(); k != k_end; ++k)
2905 targets.push_back(normalize(*k));
2908 dependencies.clear();
2913 if (WSAStartup(MAKEWORD(2,2), &wsaData))
2915 std::cerr << "Unexpected failure while initializing Windows Socket" << std::endl;
2920 // Run as client if REMAKE_SOCKET is present in the environment.
2921 if (sn) client_mode(sn, targets);
2923 // Otherwise run as server.
2924 server_mode(remakefile, targets);