2 @mainpage Remake, a build system that bridges the gap between make and redo.
4 As with <b>make</b>, <b>remake</b> uses a centralized rule file, which is
5 named <b>Remakefile</b>. It contains rules with a <em>make</em>-like
9 target1 target2 ... : prerequisite1 prerequisite2 ...
15 A target is known to be up-to-date if all its prerequisites are. If it
16 has no known prerequisites yet the file already exits, it is assumed to
17 be up-to-date. Obsolete targets are rebuilt thanks to the shell script
20 As with <b>redo</b>, <b>remake</b> supports dynamic dependencies in
21 addition to these static dependencies. Whenever a script executes
22 <tt>remake prerequisite4 prerequisite5 ...</tt>, these prerequisites are
23 rebuilt if they are obsolete. (So <b>remake</b> acts like
24 <b>redo-ifchange</b>.) Moreover, all the dependencies are stored in file
25 <b>.remake</b> so that they are remembered in subsequent runs. Note that
26 dynamic dependencies from previous runs are only used to decide whether a
27 target is obsolete; they are not automatically rebuilt when they are
28 obsolete yet a target depends on them. They will only be rebuilt once the
29 dynamic call to <b>remake</b> is executed.
31 In other words, the following two rules have almost the same behavior.
34 target1 target2 ... : prerequisite1 prerequisite2 ...
38 remake prerequisite1 prerequisite2 ...
42 (There is a difference if the targets already exist, have never been
43 built before, and the prerequisites are either younger or obsolete, since
44 the targets will not be rebuilt in the second case.)
46 The above usage of dynamic dependencies is hardly useful. Their strength
47 lies in the fact that they can be computed on the fly:
51 gcc -MMD -MF $@.d -o $@ -c $<
56 ocamldep $< | remake -r $@
59 after.xml: before.xml rules.xsl
60 xsltproc --load-trace -o after.xml rules.xsl before.xml 2> deps
61 remake `sed -n -e "\\,//,! s,^.*URL=\"\\([^\"]*\\).*\$,\\1,p" deps`
65 Note that the first rule fails if any of the header files included by
66 a C source file has to be automatically generated. In that case, one
67 should perform a first call to <b>remake</b> them before calling the
68 compiler. (Dependencies from several calls to <b>remake</b> are
69 cumulative, so they will all be remembered the next time.)
71 \section sec-usage Usage
73 Usage: <tt>remake <i>options</i> <i>targets</i></tt>
77 - <tt>-d</tt>: Echo script commands.
78 - <tt>-f FILE</tt>: Read <tt>FILE</tt> as <b>Remakefile</b>.
79 - <tt>-j[N]</tt>, <tt>--jobs=[N]</tt>: Allow <tt>N</tt> jobs at once;
80 infinite jobs with no argument.
81 - <tt>-k</tt>, <tt>--keep-going</tt>: Keep going when some targets cannot be made.
82 - <tt>-r</tt>: Look up targets from the dependencies on standard input.
83 - <tt>-s</tt>, <tt>--silent</tt>, <tt>--quiet</tt>: Do not echo targets.
85 \section sec-syntax Syntax
87 Lines starting with a space character or a tabulation are assumed to be rule
88 scripts. They are only allowed after a rule header.
90 Lines starting with <tt>#</tt> are considered to be comments and are ignored.
91 They do interrupt rule scripts though.
93 Any other line is either a rule header or a variable definition. If such a
94 line ends with a backslash, the following line break is ignored and the line
95 extends to the next one.
97 Rule headers are a nonempty list of names, followed by a colon, followed by
98 another list of names, possibly empty. Variable definitions are a single
99 name followed by equal followed by a list of names, possibly empty. Basically,
100 the syntax of a rule is as follows:
103 targets : prerequisites
107 List of names are space-separated sequences of names. If a name contains a
108 space character, it should be put into double quotes. Names can not be any
109 of the following special characters <tt>:$(),="</tt>. Again, quotation
110 should be used. Quotation marks can be escaped by a backslash inside
113 \subsection sec-variables Variables
115 Variables can be used to factor lists of targets or prerequisites. They are
116 expanded as they are encountered during <b>Remakefile</b> parsing.
125 Variable assignments can appear instead of prerequisites inside non-generic
126 rules with no script. They are then expanded inside the corresponding
130 foo.o: CFLAGS += -DBAR
133 gcc $(CFLAGS) -MMD -MF $@.d -o $@ -c $<
138 Note: contrarily to <b>make</b>, variable names have to be enclosed in
139 parentheses. For instance, <tt>$y</tt> is not a shorthand for <tt>\$(y)</tt> and
142 \subsection sec-autovars Automatic variables
144 The following special symbols can appear inside scripts:
146 - <tt>$<</tt> expands to the first static prerequisite of the rule.
147 - <tt>$^</tt> expands to all the static prerequisites of the rule, including
149 - <tt>$\@</tt> expands to the first target of the rule.
150 - <tt>$*</tt> expands to the string that matched <tt>%</tt> in a generic rule.
151 - <tt>$$</tt> expands to a single dollar symbol.
153 Note: contrarily to <b>make</b>, there are no corresponding variables. For
154 instance, <tt>$^</tt> is not a shorthand for <tt>$(^)</tt>. Another
155 difference is that <tt>$\@</tt> is always the first target, not the one that
158 \subsection sec-functions Built-in functions
160 <b>remake</b> also supports a few built-in functions inspired from <b>make</b>.
162 - <tt>$(addprefix <i>prefix</i>, <i>list</i>)</tt> returns the list obtained
163 by prepending its first argument to each element of its second argument.
164 - <tt>$(addsuffix <i>suffix</i>, <i>list</i>)</tt> returns the list obtained
165 by appending its first argument to each element of its second argument.
167 \section sec-semantics Semantics
169 \subsection src-obsolete When are targets obsolete?
171 A target is obsolete:
173 - if there is no file corresponding to the target, or to one of its siblings
174 in a multi-target rule,
175 - if any of its dynamic prerequisites from a previous run or any of its static
176 prerequisites is obsolete,
177 - if the latest file corresponding to its siblings or itself is older than any
178 of its dynamic prerequisites or static prerequisites.
180 In all the other cases, it is assumed to be up-to-date (and so are all its
181 siblings). Note that the last rule above says "latest" and not "earliest". While
182 it might cause some obsolete targets to go unnoticed in corner cases, it allows
183 for the following kind of rules:
186 config.h stamp-config_h: config.h.in config.status
187 ./config.status config.h
191 A <tt>config.status</tt> file generally does not update header files (here
192 <tt>config.h</tt>) if they would not change. As a consequence, if not for the
193 <tt>stamp-config_h</tt> file above, a header would always be considered obsolete
194 once one of its prerequisites is modified. Note that touching <tt>config.h</tt>
195 rather than <tt>stamp-config_h</tt> would defeat the point of not updating it
196 in the first place, since the program files would need to be rebuilt.
198 Once all the static prerequisites of a target have been rebuilt, <b>remake</b>
199 checks whether the target still needs to be built. If it was obsolete only
200 because its prerequisites needed to be rebuilt and none of them changed, the
201 target is assumed to be up-to-date.
203 \subsection sec-rules How are targets (re)built?
205 There are two kinds of rules. If any of the targets or prerequisites contains
206 a <tt>%</tt> character, the rule is said to be <em>generic</em>. All the
207 targets of the rule shall then contain a single <tt>%</tt> character. All the
208 other rules are said to be <em>specific</em>.
210 A rule is said to <em>match</em> a given target:
212 - if it is specific and the target appears inside its target list,
213 - if it is generic and there is a way to replace the <tt>%</tt> character
214 from one of its targets so that it matches the given target.
216 When <b>remake</b> tries to build a given target, it looks for a specific rule
217 that matches it. If there is one and its script is nonempty, it uses it to
220 Otherwise, it looks for a generic rule that match the target. If there are
221 several matching rules, it chooses the one with the shortest pattern (and if
222 there are several ones, the earliest one). <b>remake</b> then looks for
223 specific rules that match each target of the generic rule. All the
224 prerequisites of these specific rules are added to those of the generic rule.
225 The script of the generic rule is used to build the target.
231 commands building t%1 and t2%
234 commands building t2z
238 # t2x is built by the first rule (which also builds tx1) and its prerequisites are p1, px2
239 # t2y is built by the first rule (which also builds ty1) and its prerequisites are p1, py2, p3
240 # t2z is built by the second rule and its prerequisite is p4
243 The set of rules from <b>Remakefile</b> is ill-formed:
245 - if any specific rule matching a target of the generic rule has a nonempty script,
246 - if any target of the generic rule is matched by a generic rule with a shorter pattern.
248 \section sec-compilation Compilation
250 - On Linux, MacOSX, and BSD: <tt>g++ -o remake remake.cpp</tt>
251 - On Windows: <tt>g++ -o remake.exe remake.cpp -lws2_32</tt>
253 Installing <b>remake</b> is needed only if <b>Remakefile</b> does not
254 specify the path to the executable for its recursive calls. Thanks to its
255 single source file, <b>remake</b> can be shipped inside other packages and
256 built at configuration time.
258 \section sec-differences Differences with other build systems
260 Differences with <b>make</b>:
262 - Dynamic dependencies are supported.
263 - For rules with multiple targets, the shell script is executed only once
264 and is assumed to build all the targets. There is no need for
265 convoluted rules that are robust enough for parallel builds. For generic
266 rules, this is similar to the behavior of pattern rules from <b>gmake</b>.
267 - As with <b>redo</b>, only one shell is run when executing a script,
268 rather than one per script line. Note that the shells are run with
269 option <tt>-e</tt>, thus causing them to exit as soon as an error is
271 - The prerequisites of generic rules (known as implicit rules in make lingo)
272 are not used to decide between several of them. <b>remake</b> does not
273 select one for which it could satisfy the dependencies.
274 - Variables and built-in functions are expanded as they are encountered
275 during <b>Remakefile</b> parsing.
277 Differences with <b>redo</b>:
279 - As with <b>make</b>, it is possible to write the following kind of rules
282 Remakefile: Remakefile.in ./config.status
283 ./config.status Remakefile
285 - If a target is already built the first time <b>remake</b> runs, it still
286 uses the static prerequisites of rules mentioning it to check whether it
287 needs to be rebuilt. It does not assume it to be up-to-date. As with
288 <b>redo</b> though, if its obsolete status would be due to a dynamic
289 prerequisite, it will go unnoticed; it should be removed beforehand.
290 - Multiple targets are supported.
291 - <b>remake</b> has almost no features: no checksum-based dependencies, no
292 compatibility with job servers, etc.
294 \section sec-limitations Limitations
296 - When the user calls <b>remake</b>, the current working directory should be
297 the one containing <b>.remake</b>. Rules are understood relatively to this
298 directory. If a rule script calls <b>remake</b>, the current working
299 directory should be the same as the one from the original <b>remake</b>.
300 - Some cases of ill-formed rules are not caught by <b>remake</b> and can
301 thus lead to unpredictable behaviors.
303 \section sec-links Links
305 @see http://cr.yp.to/redo.html for the philosophy of <b>redo</b> and
306 https://github.com/apenwarr/redo for an implementation and some comprehensive documentation.
308 \section sec-licensing Licensing
310 @author Guillaume Melquiond
314 This program is free software: you can redistribute it and/or modify
315 it under the terms of the GNU General Public License as published by
316 the Free Software Foundation, either version 3 of the License, or
317 (at your option) any later version.
319 This program is distributed in the hope that it will be useful,
320 but WITHOUT ANY WARRANTY; without even the implied warranty of
321 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
322 GNU General Public License for more details.
324 \section sec-internals Internals
326 The parent <b>remake</b> process acts as a server. The other ones have a
327 REMAKE_SOCKET environment variable that tells them how to contact the
328 server. They send the content of the REMAKE_JOB_ID environment variable,
329 so that the server can associate the child targets to the jobs that
330 spawned them. They then wait for completion and exit with the status
331 returned by the server. This is handled by #client_mode.
333 The server calls #load_dependencies and #save_dependencies to serialize
334 dynamic dependencies from <b>.remake</b>. It loads <b>Remakefile</b> with
335 #load_rules. It then runs #server_mode, which calls #server_loop.
337 When building a target, the following sequence of events happens:
339 - #start calls #find_rule (and #find_generic_rule) to get the rule.
340 - It then creates a pseudo-client if the rule has static dependencies, or
341 calls #run_script otherwise. In both cases, a new job is created and its
342 targets are put into #job_targets.
343 - #run_script creates a shell process and stores it in #job_pids. It
344 increases #running_jobs.
345 - The child process possibly calls <b>remake</b> with a list of targets.
346 - #accept_client receives a build request from a child process and adds
347 it to #clients. It also records the new dependencies of the job into
348 #dependencies. It increases #waiting_jobs.
349 - #handle_clients uses #get_status to look up the obsoleteness of the
351 - Once the targets of a request have been built or one of them has failed,
352 #handle_clients calls #complete_request and removes the request from
354 - If the build targets come from a pseudo-client, #complete_request calls
355 #run_script. Otherwise it sends the reply to the corresponding child
356 process and decreases #waiting_jobs.
357 - When a child process ends, #server_loop calls #finalize_job, which
358 removes the process from #job_pids, decreases #running_jobs, and calls
360 - #complete_job removes the job from #job_targets and calls #update_status
361 to change the status of the targets. It also removes the target files in
366 #define WIN32_LEAN_AND_MEAN
385 #include <sys/stat.h>
386 #include <sys/types.h>
399 #include <winsock2.h>
401 typedef SOCKET socket_t;
403 #include <sys/socket.h>
405 #include <sys/wait.h>
406 typedef int socket_t;
407 enum { INVALID_SOCKET = -1 };
410 #if defined(WINDOWS) || defined(MACOSX)
411 enum { MSG_NOSIGNAL = 0 };
414 typedef std::list<std::string> string_list;
416 typedef std::set<std::string> string_set;
419 * Reference-counted shared object.
420 * @note The default constructor delays the creation of the object until it
421 * is first dereferenced.
431 content(T const &t): cnt(1), val(t) {}
433 mutable content *ptr;
434 ref_ptr(): ptr(NULL) {}
435 ref_ptr(T const &t): ptr(new content(t)) {}
436 ref_ptr(ref_ptr const &p): ptr(p.ptr) { if (ptr) ++ptr->cnt; }
437 ~ref_ptr() { if (ptr && --ptr->cnt == 0) delete ptr; }
438 ref_ptr &operator=(ref_ptr const &p)
440 if (ptr == p.ptr) return *this;
441 if (ptr && --ptr->cnt == 0) delete ptr;
448 if (!ptr) ptr = new content;
451 T *operator->() const { return &**this; }
460 typedef std::map<std::string, ref_ptr<dependency_t> > dependency_map;
462 typedef std::map<std::string, string_list> variable_map;
465 * Build status of a target.
469 Uptodate, ///< Target is up-to-date.
470 Todo, ///< Target is missing or obsolete.
471 Recheck, ///< Target has an obsolete dependency.
472 Running, ///< Target is being rebuilt.
473 Remade, ///< Target was successfully rebuilt.
474 Failed ///< Build failed for target.
478 * Build status of a target.
482 status_e status; ///< Actual status.
483 time_t last; ///< Last-modified date.
486 typedef std::map<std::string, status_t> status_map;
489 * Delayed assignment to a variable.
498 typedef std::list<assign_t> assign_list;
501 * A rule loaded from Remakefile.
505 string_list targets; ///< Files produced by this rule.
506 string_list deps; ///< Files used for an implicit call to remake at the start of the script.
507 assign_list vars; ///< Values of variables.
508 std::string script; ///< Shell script for building the targets.
509 std::string stem; ///< String used to instantiate a generic rule.
512 typedef std::list<rule_t> rule_list;
514 typedef std::map<std::string, ref_ptr<rule_t> > rule_map;
516 typedef std::map<int, string_list> job_targets_map;
518 typedef std::map<int, variable_map> job_variables_map;
520 typedef std::map<pid_t, int> pid_job_map;
523 * Client waiting for a request to complete.
525 * There are two kinds of clients:
526 * - real clients, which are instances of remake created by built scripts,
527 * - pseudo clients, which are created by the server to build specific targets.
529 * Among pseudo clients, there are two categories:
530 * - original clients, which are created for the targets passed on the
531 * command line by the user or for the initial regeneration of the rule file,
532 * - dependency clients, which are created to handle rules that have
533 * explicit dependencies and thus to emulate a call to remake.
537 socket_t socket; ///< Socket used to reply to the client (invalid for pseudo clients).
538 int job_id; ///< Job for which the built script called remake and spawned the client (negative for original clients).
539 bool failed; ///< Whether some targets failed in mode -k.
540 string_list pending; ///< Targets not yet started.
541 string_set running; ///< Targets being built.
542 rule_t *delayed; ///< Rule that implicitly created a dependency client, and which script has to be started on request completion.
543 client_t(): socket(INVALID_SOCKET), job_id(-1), failed(false), delayed(NULL) {}
546 typedef std::list<client_t> client_list;
549 * Map from variable names to their content.
551 static variable_map variables;
554 * Map from targets to their known dependencies.
556 static dependency_map dependencies;
559 * Map from targets to their build status.
561 static status_map status;
564 * Set of generic rules loaded from Remakefile.
566 static rule_list generic_rules;
569 * Map from targets to specific rules loaded from Remakefile.
571 static rule_map specific_rules;
574 * Map from jobs to targets being built.
576 static job_targets_map job_targets;
579 * Map from jobs to job specific variables
581 static job_variables_map job_variables;
584 * Map from jobs to shell pids.
586 static pid_job_map job_pids;
589 * List of clients waiting for a request to complete.
590 * New clients are put to front, so that the build process is depth-first.
592 static client_list clients;
595 * Maximum number of parallel jobs (non-positive if unbounded).
596 * Can be modified by the -j option.
598 static int max_active_jobs = 1;
601 * Whether to keep building targets in case of failure.
602 * Can be modified by the -k option.
604 static bool keep_going = false;
607 * Number of jobs currently running:
608 * - it increases when a process is created in #run_script,
609 * - it decreases when a completion message is received in #finalize_job.
611 * @note There might be some jobs running while #clients is empty.
612 * Indeed, if a client requested two targets to be rebuilt, if they
613 * are running concurrently, if one of them fails, the client will
614 * get a failure notice and might terminate before the other target
617 static int running_jobs = 0;
620 * Number of jobs currently waiting for a build request to finish:
621 * - it increases when a build request is received in #accept_client
622 * (since the client is presumably waiting for the reply),
623 * - it decreases when a reply is sent in #complete_request.
625 static int waiting_jobs = 0;
628 * Global counter used to produce increasing job numbers.
631 static int job_counter = 0;
634 * Socket on which the server listens for client request.
636 static socket_t socket_fd;
639 * Whether the request of an original client failed.
641 static bool build_failure;
645 * Name of the server socket in the file system.
647 static char *socket_name;
651 * Name of the first target of the first specific rule, used for default run.
653 static std::string first_target;
656 * Whether a short message should be displayed for each target.
658 static bool show_targets = true;
661 * Whether script commands are echoed.
663 static bool echo_scripts = false;
665 static time_t now = time(NULL);
667 static std::string working_dir;
670 static volatile sig_atomic_t got_SIGCHLD = 0;
672 static void sigchld_handler(int)
677 static void sigint_handler(int)
679 // Child processes will receive the signal too, so just prevent
680 // new jobs from starting and wait for the running jobs to fail.
689 log(): active(false), open(false), depth(0)
692 std::ostream &operator()()
694 if (open) std::cerr << std::endl;
696 std::cerr << std::string(depth * 2, ' ');
700 std::ostream &operator()(bool o)
702 if (o && open) std::cerr << std::endl;
705 if (o || !open) std::cerr << std::string(depth * 2, ' ');
714 struct log_auto_close
717 log_auto_close(): still_open(true)
722 if (debug.active && still_open) debug(false) << "done\n";
726 #define DEBUG if (debug.active) debug()
727 #define DEBUG_open log_auto_close auto_close; if (debug.active) debug(true)
728 #define DEBUG_close if ((auto_close.still_open = false), debug.active) debug(false)
731 * Strong typedef for strings that need escaping.
732 * @note The string is stored as a reference, so the constructed object is
733 * meant to be immediately consumed.
737 std::string const &input;
738 escape_string(std::string const &s): input(s) {}
742 * Write the string in @a se to @a out if it does not contain any special
743 * characters, a quoted and escaped string otherwise.
745 static std::ostream &operator<<(std::ostream &out, escape_string const &se)
747 std::string const &s = se.input;
748 char const *quoted_char = ",: '";
749 char const *escaped_char = "\"\\$!";
750 bool need_quotes = false;
752 size_t len = s.length(), last = 0, j = 0;
753 for (size_t i = 0; i < len; ++i)
755 if (strchr(escaped_char, s[i]))
758 if (!buf) buf = new char[len * 2];
759 memcpy(&buf[j], &s[last], i - last);
765 if (!need_quotes && strchr(quoted_char, s[i]))
768 if (!need_quotes) return out << s;
770 if (!buf) return out << s << '"';
772 out.write(&s[last], len - last);
778 * @defgroup paths Path helpers
784 * Initialize #working_dir.
786 void init_working_dir()
789 char *res = getcwd(buf, sizeof(buf));
792 perror("Failed to get working directory");
799 * Normalize an absolute path with respect to the working directory.
800 * Paths outside the working subtree are left unchanged.
802 static std::string normalize_abs(std::string const &s)
804 size_t l = working_dir.length();
805 if (s.compare(0, l, working_dir)) return s;
806 size_t ll = s.length();
807 if (ll == l) return ".";
810 size_t pos = s.rfind('/', l);
811 assert(pos != std::string::npos);
812 return s.substr(pos + 1);
814 if (ll == l + 1) return ".";
815 return s.substr(l + 1);
819 * Normalize a target name.
821 static std::string normalize(std::string const &s)
824 char const *delim = "/\\";
828 size_t prev = 0, len = s.length();
829 size_t pos = s.find_first_of(delim);
830 if (pos == std::string::npos) return s;
831 bool absolute = pos == 0;
837 std::string n = s.substr(prev, pos - prev);
840 if (!l.empty()) l.pop_back();
842 return normalize(working_dir + '/' + s);
848 if (pos >= len) break;
850 pos = s.find_first_of(delim, prev);
851 if (pos == std::string::npos) pos = len;
853 string_list::const_iterator i = l.begin(), i_end = l.end();
854 if (i == i_end) return absolute ? "/" : ".";
856 if (absolute) n.push_back('/');
858 for (++i; i != i_end; ++i)
863 if (absolute) return normalize_abs(n);
868 * Normalize the content of a list of targets.
870 static void normalize_list(string_list &l)
872 for (string_list::iterator i = l.begin(),
873 i_end = l.end(); i != i_end; ++i)
882 * @defgroup lexer Lexer
890 static void skip_spaces(std::istream &in)
893 while (strchr(" \t", (c = in.get()))) {}
894 if (in.good()) in.putback(c);
900 static void skip_empty(std::istream &in)
903 while (strchr("\r\n", (c = in.get()))) {}
904 if (in.good()) in.putback(c);
908 * Skip end of line. If @a multi is true, skip the following empty lines too.
909 * @return true if there was a line to end.
911 static bool skip_eol(std::istream &in, bool multi = false)
914 if (c == '\r') c = in.get();
915 if (c != '\n' && in.good()) in.putback(c);
916 if (c != '\n' && !in.eof()) return false;
917 if (multi) skip_empty(in);
934 * Skip spaces and peek at the next token.
935 * If it is one of @a mask, skip it (if it is not Word) and return it.
936 * @note For composite tokens allowed by @a mask, input characters might
937 * have been eaten even for an Unexpected result.
939 static int expect_token(std::istream &in, int mask)
945 if (!in.good()) return Unexpected;
950 case '\n': return Unexpected;
951 case ':': tok = Colon; break;
952 case ',': tok = Comma; break;
953 case '=': tok = Equal; break;
954 case ')': tok = Rightpar; break;
956 if (!(mask & Dollarpar)) return Unexpected;
959 if (in.peek() != '(') return Unexpected;
962 if (!(mask & Plusequal)) return Unexpected;
965 if (in.peek() != '=') return Unexpected;
969 if (skip_eol(in)) continue;
971 return mask & Word ? Word : Unexpected;
973 return mask & Word ? Word : Unexpected;
975 if (!(tok & mask)) return Unexpected;
982 * Read a (possibly quoted) word.
984 static std::string read_word(std::istream &in)
988 if (!in.good()) return res;
989 char const *separators = " \t\r\n:$(),=+\"";
990 bool quoted = c == '"';
993 if (strchr(separators, c))
1003 if (!in.good()) return res;
1015 if (strchr(separators, c))
1028 * @defgroup stream Token streams
1034 * Possible results from word producers.
1044 * Interface for word producers.
1048 virtual ~generator() {}
1049 virtual input_status next(std::string &) = 0;
1053 * Generator for the words of a variable.
1055 struct variable_generator: generator
1058 string_list::const_iterator cur1, end1;
1059 assign_list::const_iterator cur2, end2;
1060 variable_generator(std::string const &, assign_list const *);
1061 input_status next(std::string &);
1064 variable_generator::variable_generator(std::string const &n,
1065 assign_list const *local_variables): name(n)
1068 if (local_variables)
1070 // Set cur2 to the last variable overwriter, if any.
1071 cur2 = local_variables->begin();
1072 end2 = local_variables->end();
1073 for (assign_list::const_iterator i = cur2; i != end2; ++i)
1075 if (i->name == name && !i->append)
1084 static assign_list dummy;
1085 cur2 = dummy.begin();
1088 static string_list dummy;
1089 cur1 = dummy.begin();
1093 variable_map::const_iterator i = variables.find(name);
1094 if (i == variables.end()) return;
1095 cur1 = i->second.begin();
1096 end1 = i->second.end();
1100 input_status variable_generator::next(std::string &res)
1109 while (cur2 != end2)
1111 if (cur2->name == name)
1113 cur1 = cur2->value.begin();
1114 end1 = cur2->value.end();
1124 * Generator for the words of an input stream.
1126 struct input_generator
1130 assign_list const *local_variables;
1131 bool earliest_exit, done;
1132 input_generator(std::istream &i, assign_list const *lv, bool e = false)
1133 : in(i), nested(NULL), local_variables(lv), earliest_exit(e), done(false) {}
1134 input_status next(std::string &);
1135 ~input_generator() { assert(!nested); }
1138 static generator *get_function(input_generator const &, std::string const &);
1140 input_status input_generator::next(std::string &res)
1145 input_status s = nested->next(res);
1146 if (s == Success) return Success;
1149 if (s == SyntaxError) return SyntaxError;
1151 if (done) return Eof;
1152 if (earliest_exit) done = true;
1153 switch (expect_token(in, Word | Dollarpar))
1156 res = read_word(in);
1160 std::string name = read_word(in);
1161 if (name.empty()) return SyntaxError;
1162 if (expect_token(in, Rightpar))
1163 nested = new variable_generator(name, local_variables);
1166 nested = get_function(*this, name);
1167 if (!nested) return SyntaxError;
1177 * Read a list of words from an input generator.
1178 * @return false if a syntax error was encountered.
1180 static bool read_words(input_generator &in, string_list &res)
1184 res.push_back(std::string());
1185 input_status s = in.next(res.back());
1186 if (s == Success) continue;
1192 static bool read_words(std::istream &in, string_list &res)
1194 input_generator gen(in, NULL);
1195 return read_words(gen, res);
1200 * Serialize a variable map.
1202 std::string serialize_variables(variable_map &vars, char sep = ' ')
1204 std::ostringstream buf;
1205 for(variable_map::const_iterator i = vars.begin(),
1206 i_end = vars.end(); i != i_end; ++i)
1208 buf << i->first << '=';
1211 for (string_list::const_iterator j = i->second.begin(),
1212 j_end = i->second.end(); j != j_end; ++j)
1214 if (first) first = false;
1218 buf << escape_string(val) << sep;
1220 std::string s = buf.str();
1225 * Generator for the result of function addprefix.
1227 struct addprefix_generator: generator
1229 input_generator gen;
1231 string_list::const_iterator prei;
1234 addprefix_generator(input_generator const &, bool &);
1235 input_status next(std::string &);
1238 addprefix_generator::addprefix_generator(input_generator const &top, bool &ok)
1239 : gen(top.in, top.local_variables)
1241 if (!read_words(gen, pre)) return;
1242 if (!expect_token(gen.in, Comma)) return;
1248 input_status addprefix_generator::next(std::string &res)
1265 switch (gen.next(res))
1268 if (!prel) return Success;
1274 return expect_token(gen.in, Rightpar) ? Eof : SyntaxError;
1281 * Generator for the result of function addsuffix.
1283 struct addsuffix_generator: generator
1285 input_generator gen;
1287 string_list::const_iterator sufi;
1290 addsuffix_generator(input_generator const &, bool &);
1291 input_status next(std::string &);
1294 addsuffix_generator::addsuffix_generator(input_generator const &top, bool &ok)
1295 : gen(top.in, top.local_variables)
1297 if (!read_words(gen, suf)) return;
1298 if (!expect_token(gen.in, Comma)) return;
1304 input_status addsuffix_generator::next(std::string &res)
1316 switch (gen.next(res))
1319 if (!sufl) return Success;
1325 return expect_token(gen.in, Rightpar) ? Eof : SyntaxError;
1332 * Return a generator for function @a name.
1334 generator *get_function(input_generator const &in, std::string const &name)
1337 generator *g = NULL;
1339 if (name == "addprefix") g = new addprefix_generator(in, ok);
1340 else if (name == "addsuffix") g = new addsuffix_generator(in, ok);
1341 if (!g || ok) return g;
1349 * @defgroup database Dependency database
1355 * Load dependencies from @a in.
1357 static void load_dependencies(std::istream &in)
1362 std::cerr << "Failed to load database" << std::endl;
1368 string_list targets;
1369 if (!read_words(in, targets)) goto error;
1370 if (in.eof()) return;
1371 if (targets.empty()) goto error;
1372 DEBUG << "reading dependencies of target " << targets.front() << std::endl;
1373 if (in.get() != ':') goto error;
1374 ref_ptr<dependency_t> dep;
1375 dep->targets = targets;
1377 if (!read_words(in, deps)) goto error;
1378 dep->deps.insert(deps.begin(), deps.end());
1379 for (string_list::const_iterator i = targets.begin(),
1380 i_end = targets.end(); i != i_end; ++i)
1382 dependencies[*i] = dep;
1389 * Load known dependencies from file <tt>.remake</tt>.
1391 static void load_dependencies()
1393 DEBUG_open << "Loading database... ";
1394 std::ifstream in(".remake");
1397 DEBUG_close << "not found\n";
1400 load_dependencies(in);
1405 * Save all the dependencies in file <tt>.remake</tt>.
1407 static void save_dependencies()
1409 DEBUG_open << "Saving database... ";
1410 std::ofstream db(".remake");
1411 while (!dependencies.empty())
1413 ref_ptr<dependency_t> dep = dependencies.begin()->second;
1414 for (string_list::const_iterator i = dep->targets.begin(),
1415 i_end = dep->targets.end(); i != i_end; ++i)
1417 db << escape_string(*i) << ' ';
1418 dependencies.erase(*i);
1421 for (string_set::const_iterator i = dep->deps.begin(),
1422 i_end = dep->deps.end(); i != i_end; ++i)
1424 db << ' ' << escape_string(*i);
1433 * @defgroup parser Rule parser
1439 * Register a specific rule with an empty script:
1441 * - Check that none of the targets already has an associated rule with a
1443 * - Create a new rule with a single target for each target, if needed.
1444 * - Add the prerequisites of @a rule to all these associated rules.
1446 static void register_transparent_rule(rule_t const &rule, string_list const &targets)
1448 assert(rule.script.empty());
1449 for (string_list::const_iterator i = targets.begin(),
1450 i_end = targets.end(); i != i_end; ++i)
1452 std::pair<rule_map::iterator, bool> j =
1453 specific_rules.insert(std::make_pair(*i, ref_ptr<rule_t>()));
1454 ref_ptr<rule_t> &r = j.first->second;
1457 r = ref_ptr<rule_t>(rule);
1458 r->targets = string_list(1, *i);
1461 if (!r->script.empty())
1463 std::cerr << "Failed to load rules: " << *i
1464 << " cannot be the target of several rules" << std::endl;
1467 assert(r->targets.size() == 1 && r->targets.front() == *i);
1468 r->deps.insert(r->deps.end(), rule.deps.begin(), rule.deps.end());
1469 r->vars.insert(r->vars.end(), rule.vars.begin(), rule.vars.end());
1472 for (string_list::const_iterator i = targets.begin(),
1473 i_end = targets.end(); i != i_end; ++i)
1475 ref_ptr<dependency_t> &dep = dependencies[*i];
1476 if (dep->targets.empty()) dep->targets.push_back(*i);
1477 dep->deps.insert(rule.deps.begin(), rule.deps.end());
1482 * Register a specific rule with a nonempty script:
1484 * - Check that none of the targets already has an associated rule.
1485 * - Create a single shared rule and associate it to all the targets.
1486 * - Merge the prerequisites of all the targets into a single set and
1487 * add the prerequisites of the rule to it. (The preexisting
1488 * prerequisites, if any, come from a previous run.)
1490 static void register_scripted_rule(rule_t const &rule)
1492 ref_ptr<rule_t> r(rule);
1493 for (string_list::const_iterator i = rule.targets.begin(),
1494 i_end = rule.targets.end(); i != i_end; ++i)
1496 std::pair<rule_map::iterator, bool> j =
1497 specific_rules.insert(std::make_pair(*i, r));
1498 if (j.second) continue;
1499 std::cerr << "Failed to load rules: " << *i
1500 << " cannot be the target of several rules" << std::endl;
1504 ref_ptr<dependency_t> dep;
1505 dep->targets = rule.targets;
1506 dep->deps.insert(rule.deps.begin(), rule.deps.end());
1507 for (string_list::const_iterator i = rule.targets.begin(),
1508 i_end = rule.targets.end(); i != i_end; ++i)
1510 ref_ptr<dependency_t> &d = dependencies[*i];
1511 dep->deps.insert(d->deps.begin(), d->deps.end());
1517 * Read a rule starting with target @a first, if nonempty.
1518 * Store into #generic_rules or #specific_rules depending on its genericity.
1520 static void load_rule(std::istream &in, std::string const &first)
1522 DEBUG_open << "Reading rule for target " << first << "... ";
1526 DEBUG_close << "failed\n";
1527 std::cerr << "Failed to load rules: syntax error" << std::endl;
1532 // Read targets and check genericity.
1533 string_list targets;
1534 if (!read_words(in, targets)) goto error;
1535 if (!first.empty()) targets.push_front(first);
1536 else if (targets.empty()) goto error;
1537 else DEBUG << "actual target: " << targets.front() << std::endl;
1538 bool generic = false;
1539 normalize_list(targets);
1540 for (string_list::const_iterator i = targets.begin(),
1541 i_end = targets.end(); i != i_end; ++i)
1543 if (i->empty()) goto error;
1544 if ((i->find('%') != std::string::npos) != generic)
1546 if (i == targets.begin()) generic = true;
1550 std::swap(rule.targets, targets);
1552 if (in.get() != ':') goto error;
1554 bool assignment = false;
1556 // Read dependencies.
1557 if (expect_token(in, Word))
1559 std::string d = read_word(in);
1560 if (int tok = expect_token(in, Equal | Plusequal))
1562 rule.vars.push_back(assign_t());
1564 if (!read_words(in, v)) goto error;
1565 assign_t &a = rule.vars.back();
1567 a.append = tok == Plusequal;
1574 if (!read_words(in, v)) goto error;
1583 if (!read_words(in, v)) goto error;
1588 if (!skip_eol(in, true)) goto error;
1591 std::ostringstream buf;
1595 if (!in.good()) break;
1596 if (c == '\t' || c == ' ')
1598 in.get(*buf.rdbuf());
1599 if (in.fail() && !in.eof()) in.clear();
1601 else if (c == '\r' || c == '\n')
1609 rule.script = buf.str();
1611 // Add generic rules to the correct set.
1614 if (assignment) goto error;
1615 generic_rules.push_back(rule);
1619 if (!rule.script.empty())
1621 if (assignment) goto error;
1622 register_scripted_rule(rule);
1626 // Swap away the targets to avoid costly copies when registering.
1627 string_list targets;
1628 std::swap(rule.targets, targets);
1629 register_transparent_rule(rule, targets);
1630 std::swap(rule.targets, targets);
1633 // If there is no default target yet, mark it as such.
1634 if (first_target.empty())
1635 first_target = rule.targets.front();
1639 * Load rules from @a remakefile.
1640 * If some rules have dependencies and non-generic targets, add these
1641 * dependencies to the targets.
1643 static void load_rules(std::string const &remakefile)
1645 DEBUG_open << "Loading rules... ";
1649 std::cerr << "Failed to load rules: syntax error" << std::endl;
1652 std::ifstream in(remakefile.c_str());
1655 std::cerr << "Failed to load rules: no Remakefile found" << std::endl;
1666 while (in.get() != '\n') {}
1670 if (c == ' ' || c == '\t') goto error;
1671 if (expect_token(in, Word))
1673 std::string name = read_word(in);
1674 if (name.empty()) goto error;
1675 if (int tok = expect_token(in, Equal | Plusequal))
1677 DEBUG << "Assignment to variable " << name << std::endl;
1679 if (!read_words(in, value)) goto error;
1680 string_list &dest = variables[name];
1681 if (tok == Equal) dest.swap(value);
1682 else dest.splice(dest.end(), value);
1683 if (!skip_eol(in, true)) goto error;
1685 else load_rule(in, name);
1687 else load_rule(in, std::string());
1694 * @defgroup rules Rule resolution
1700 * Substitute a pattern into a list of strings.
1702 static void substitute_pattern(std::string const &pat, string_list const &src, string_list &dst)
1704 for (string_list::const_iterator i = src.begin(),
1705 i_end = src.end(); i != i_end; ++i)
1707 size_t pos = i->find('%');
1708 if (pos == std::string::npos)dst.push_back(*i);
1709 else dst.push_back(i->substr(0, pos) + pat + i->substr(pos + 1));
1714 * Find a generic rule matching @a target:
1715 * - the one leading to shorter matches has priority,
1716 * - among equivalent rules, the earliest one has priority.
1718 static rule_t find_generic_rule(std::string const &target)
1720 size_t tlen = target.length(), plen = tlen + 1;
1722 for (rule_list::const_iterator i = generic_rules.begin(),
1723 i_end = generic_rules.end(); i != i_end; ++i)
1725 for (string_list::const_iterator j = i->targets.begin(),
1726 j_end = i->targets.end(); j != j_end; ++j)
1728 size_t len = j->length();
1729 if (tlen < len) continue;
1730 if (plen <= tlen - (len - 1)) continue;
1731 size_t pos = j->find('%');
1732 if (pos == std::string::npos) continue;
1733 size_t len2 = len - (pos + 1);
1734 if (j->compare(0, pos, target, 0, pos) ||
1735 j->compare(pos + 1, len2, target, tlen - len2, len2))
1737 plen = tlen - (len - 1);
1739 rule.stem = target.substr(pos, plen);
1740 rule.script = i->script;
1741 substitute_pattern(rule.stem, i->targets, rule.targets);
1742 substitute_pattern(rule.stem, i->deps, rule.deps);
1750 * Find a specific rule matching @a target. Return a generic one otherwise.
1751 * If there is both a specific rule with an empty script and a generic rule, the
1752 * generic one is returned after adding the dependencies of the specific one.
1754 static rule_t find_rule(std::string const &target)
1756 rule_map::const_iterator i = specific_rules.find(target),
1757 i_end = specific_rules.end();
1758 // If there is a specific rule with a script, return it.
1759 if (i != i_end && !i->second->script.empty()) return *i->second;
1760 rule_t grule = find_generic_rule(target);
1761 // If there is no generic rule, return the specific rule (no script), if any.
1762 if (grule.targets.empty())
1764 if (i != i_end) return *i->second;
1767 // Optimize the lookup when there is only one target (already looked up).
1768 if (grule.targets.size() == 1)
1770 if (i == i_end) return grule;
1771 grule.deps.insert(grule.deps.end(),
1772 i->second->deps.begin(), i->second->deps.end());
1773 grule.vars.insert(grule.vars.end(),
1774 i->second->vars.begin(), i->second->vars.end());
1777 // Add the dependencies of the specific rules of every target to the
1778 // generic rule. If any of those rules has a nonempty script, error out.
1779 for (string_list::const_iterator j = grule.targets.begin(),
1780 j_end = grule.targets.end(); j != j_end; ++j)
1782 i = specific_rules.find(*j);
1783 if (i == i_end) continue;
1784 if (!i->second->script.empty()) return rule_t();
1785 grule.deps.insert(grule.deps.end(),
1786 i->second->deps.begin(), i->second->deps.end());
1787 grule.vars.insert(grule.vars.end(),
1788 i->second->vars.begin(), i->second->vars.end());
1796 * @defgroup status Target status
1802 * Compute and memoize the status of @a target:
1803 * - if the file does not exist, the target is obsolete,
1804 * - if any dependency is obsolete or younger than the file, it is obsolete,
1805 * - otherwise it is up-to-date.
1807 * @note For rules with multiple targets, all the targets share the same
1808 * status. (If one is obsolete, they all are.) The second rule above
1809 * is modified in that case: the latest target is chosen, not the oldest!
1811 static status_t const &get_status(std::string const &target)
1813 std::pair<status_map::iterator,bool> i =
1814 status.insert(std::make_pair(target, status_t()));
1815 status_t &ts = i.first->second;
1816 if (!i.second) return ts;
1817 DEBUG_open << "Checking status of " << target << "... ";
1818 dependency_map::const_iterator j = dependencies.find(target);
1819 if (j == dependencies.end())
1822 if (stat(target.c_str(), &s) != 0)
1824 DEBUG_close << "missing\n";
1829 DEBUG_close << "up-to-date\n";
1830 ts.status = Uptodate;
1831 ts.last = s.st_mtime;
1834 dependency_t const &dep = *j->second;
1835 status_e st = Uptodate;
1837 for (string_list::const_iterator k = dep.targets.begin(),
1838 k_end = dep.targets.end(); k != k_end; ++k)
1841 if (stat(k->c_str(), &s) != 0)
1843 if (st == Uptodate) DEBUG_close << *k << " missing\n";
1847 status[*k].last = s.st_mtime;
1848 if (s.st_mtime > latest) latest = s.st_mtime;
1850 if (st == Todo) goto update;
1851 for (string_set::const_iterator k = dep.deps.begin(),
1852 k_end = dep.deps.end(); k != k_end; ++k)
1854 status_t const &ts_ = get_status(*k);
1855 if (latest < ts_.last)
1857 DEBUG_close << "older than " << *k << std::endl;
1861 if (ts_.status == Uptodate) continue;
1863 DEBUG << "obsolete dependency " << *k << std::endl;
1866 if (st == Uptodate) DEBUG_close << "all siblings up-to-date\n";
1868 for (string_list::const_iterator k = dep.targets.begin(),
1869 k_end = dep.targets.end(); k != k_end; ++k)
1871 status[*k].status = st;
1877 * Change the status of @a target to #Remade or #Uptodate depending on whether
1878 * its modification time changed.
1880 static void update_status(std::string const &target)
1882 DEBUG_open << "Rechecking status of " << target << "... ";
1883 status_map::iterator i = status.find(target);
1884 assert(i != status.end());
1885 status_t &ts = i->second;
1889 DEBUG_close << "possibly remade\n";
1893 if (stat(target.c_str(), &s) != 0)
1895 DEBUG_close << "missing\n";
1898 else if (s.st_mtime != ts.last)
1900 DEBUG_close << "remade\n";
1901 ts.last = s.st_mtime;
1905 DEBUG_close << "unchanged\n";
1906 ts.status = Uptodate;
1911 * Check whether all the prerequisites of @a target ended being up-to-date.
1913 static bool still_need_rebuild(std::string const &target)
1915 DEBUG_open << "Rechecking obsoleteness of " << target << "... ";
1916 status_map::const_iterator i = status.find(target);
1917 assert(i != status.end());
1918 if (i->second.status != Recheck) return true;
1919 dependency_map::const_iterator j = dependencies.find(target);
1920 assert(j != dependencies.end());
1921 dependency_t const &dep = *j->second;
1922 for (string_set::const_iterator k = dep.deps.begin(),
1923 k_end = dep.deps.end(); k != k_end; ++k)
1925 if (status[*k].status != Uptodate) return true;
1927 for (string_list::const_iterator k = dep.targets.begin(),
1928 k_end = dep.targets.end(); k != k_end; ++k)
1930 status[*k].status = Uptodate;
1932 DEBUG_close << "no longer obsolete\n";
1939 * @defgroup server Server
1945 * Handle job completion.
1947 static void complete_job(int job_id, bool success)
1949 DEBUG_open << "Completing job " << job_id << "... ";
1950 job_targets_map::iterator i = job_targets.find(job_id);
1951 assert(i != job_targets.end());
1952 string_list const &targets = i->second;
1955 for (string_list::const_iterator j = targets.begin(),
1956 j_end = targets.end(); j != j_end; ++j)
1963 DEBUG_close << "failed\n";
1964 std::cerr << "Failed to build";
1965 for (string_list::const_iterator j = targets.begin(),
1966 j_end = targets.end(); j != j_end; ++j)
1968 status[*j].status = Failed;
1969 std::cerr << ' ' << *j;
1972 std::cerr << std::endl;
1974 job_targets.erase(i);
1975 job_variables.erase(job_variables.find(job_id));
1979 * Return the script obtained by substituting variables.
1981 static std::string prepare_script(rule_t const &rule)
1983 std::string const &s = rule.script;
1984 std::istringstream in(s);
1985 std::ostringstream out;
1986 size_t len = s.size();
1990 size_t pos = in.tellg(), p = s.find('$', pos);
1991 if (p == std::string::npos || p == len - 1) p = len;
1992 out.write(&s[pos], p - pos);
1993 if (p == len) break;
2002 if (!rule.deps.empty())
2003 out << rule.deps.front();
2009 for (string_list::const_iterator i = rule.deps.begin(),
2010 i_end = rule.deps.end(); i != i_end; ++i)
2012 if (first) first = false;
2020 assert(!rule.targets.empty());
2021 out << rule.targets.front();
2032 input_generator gen(in, &rule.vars, true);
2036 input_status s = gen.next(w);
2037 if (s == SyntaxError)
2042 if (s == Eof) break;
2043 if (first) first = false;
2050 // Let dollars followed by an unrecognized character
2051 // go through. This differs from Make, which would
2052 // use a one-letter variable.
2062 * Execute the script from @a rule.
2064 static bool run_script(int job_id, rule_t /*const */ &rule)
2068 std::cout << "Building";
2069 for (string_list::const_iterator i = rule.targets.begin(),
2070 i_end = rule.targets.end(); i != i_end; ++i)
2072 std::cout << ' ' << *i;
2074 std::cout << std::endl;
2077 ref_ptr<dependency_t> dep;
2078 dep->targets = rule.targets;
2079 dep->deps.insert(rule.deps.begin(), rule.deps.end());
2080 for (string_list::const_iterator i = rule.targets.begin(),
2081 i_end = rule.targets.end(); i != i_end; ++i)
2083 dependencies[*i] = dep;
2085 variable_map job_vars = job_variables[job_id];
2086 for(variable_map::const_iterator i = job_vars.begin(),
2087 i_end = job_vars.end(); i != i_end; ++i)
2089 assign_t a = assign_t();
2092 a.value = i->second;
2093 rule.vars.push_back(a);
2096 std::string script = prepare_script(rule);
2098 std::ostringstream job_id_buf;
2099 job_id_buf << job_id;
2100 std::string job_id_ = job_id_buf.str();
2102 DEBUG_open << "Starting script for job " << job_id << "... ";
2106 DEBUG_close << "failed\n";
2107 complete_job(job_id, false);
2116 CloseHandle(pfd[0]);
2117 CloseHandle(pfd[1]);
2120 if (!CreatePipe(&pfd[0], &pfd[1], NULL, 0))
2122 if (!SetHandleInformation(pfd[0], HANDLE_FLAG_INHERIT, HANDLE_FLAG_INHERIT))
2125 ZeroMemory(&si, sizeof(STARTUPINFO));
2126 si.cb = sizeof(STARTUPINFO);
2127 si.hStdError = GetStdHandle(STD_ERROR_HANDLE);
2128 si.hStdOutput = GetStdHandle(STD_OUTPUT_HANDLE);
2129 si.hStdInput = pfd[0];
2130 si.dwFlags |= STARTF_USESTDHANDLES;
2131 PROCESS_INFORMATION pi;
2132 ZeroMemory(&pi, sizeof(PROCESS_INFORMATION));
2133 if (!SetEnvironmentVariable("REMAKE_JOB_ID", job_id_.c_str()))
2135 char const *argv = echo_scripts ? "SH.EXE -e -s -v" : "SH.EXE -e -s";
2136 if (!CreateProcess(NULL, (char *)argv, NULL, NULL,
2137 true, 0, NULL, NULL, &si, &pi))
2141 CloseHandle(pi.hThread);
2142 DWORD len = script.length(), wlen;
2143 if (!WriteFile(pfd[1], script.c_str(), len, &wlen, NULL) || wlen < len)
2144 std::cerr << "Unexpected failure while sending script to shell" << std::endl;
2145 CloseHandle(pfd[0]);
2146 CloseHandle(pfd[1]);
2148 job_pids[pi.hProcess] = job_id;
2159 if (pipe(pfd) == -1)
2161 if (setenv("REMAKE_JOB_ID", job_id_.c_str(), 1))
2163 if (pid_t pid = vfork())
2165 if (pid == -1) goto error2;
2166 ssize_t len = script.length();
2167 if (write(pfd[1], script.c_str(), len) < len)
2168 std::cerr << "Unexpected failure while sending script to shell" << std::endl;
2172 job_pids[pid] = job_id;
2175 // Child process starts here. Notice the use of vfork above.
2176 char const *argv[5] = { "sh", "-e", "-s", NULL, NULL };
2177 if (echo_scripts) argv[3] = "-v";
2184 execve("/bin/sh", (char **)argv, environ);
2185 _exit(EXIT_FAILURE);
2190 * Create a job for @a target according to the loaded rules.
2191 * Mark all the targets from the rule as running and reset their dependencies.
2192 * If the rule has dependencies, create a new client to build them just
2193 * before @a current, and change @a current so that it points to it.
2195 static bool start(std::string const &target, client_list::iterator ¤t)
2197 DEBUG_open << "Starting job " << job_counter << " for " << target << "... ";
2198 rule_t rule = find_rule(target);
2199 if (rule.targets.empty())
2201 status[target].status = Failed;
2202 DEBUG_close << "failed\n";
2203 std::cerr << "No rule for building " << target << std::endl;
2206 for (string_list::const_iterator i = rule.targets.begin(),
2207 i_end = rule.targets.end(); i != i_end; ++i)
2209 status[*i].status = Running;
2211 int job_id = job_counter++;
2212 job_targets[job_id] = rule.targets;
2213 DEBUG << "Setting variables of job: " << job_id << " (spawn by: "
2214 << current->job_id << ") to "
2215 << serialize_variables(job_variables[current->job_id], ' ')
2217 job_variables[job_id] = job_variables[current->job_id];
2218 if (!rule.deps.empty())
2220 current = clients.insert(current, client_t());
2221 current->job_id = job_id;
2222 current->pending = rule.deps;
2223 current->delayed = new rule_t(rule);
2226 return run_script(job_id, rule);
2230 * Send a reply to a client then remove it.
2231 * If the client was a dependency client, start the actual script.
2233 static void complete_request(client_t &client, bool success)
2235 DEBUG_open << "Completing request from client of job " << client.job_id << "... ";
2238 assert(client.socket == INVALID_SOCKET);
2241 if (still_need_rebuild(client.delayed->targets.front()))
2242 run_script(client.job_id, *client.delayed);
2243 else complete_job(client.job_id, true);
2245 else complete_job(client.job_id, false);
2246 delete client.delayed;
2248 else if (client.socket != INVALID_SOCKET)
2250 char res = success ? 1 : 0;
2251 send(client.socket, &res, 1, 0);
2253 closesocket(client.socket);
2255 close(client.socket);
2260 if (client.job_id < 0 && !success) build_failure = true;
2264 * Return whether there are slots for starting new jobs.
2266 static bool has_free_slots()
2268 if (max_active_jobs <= 0) return true;
2269 return running_jobs - waiting_jobs < max_active_jobs;
2273 * Handle client requests:
2274 * - check for running targets that have finished,
2275 * - start as many pending targets as allowed,
2276 * - complete the request if there are neither running nor pending targets
2277 * left or if any of them failed.
2279 * @return true if some child processes are still running.
2281 * @post If there are pending requests, at least one child process is running.
2283 static bool handle_clients()
2285 DEBUG_open << "Handling client requests... ";
2288 for (client_list::iterator i = clients.begin(), i_next = i,
2289 i_end = clients.end(); i != i_end && has_free_slots(); i = i_next)
2292 DEBUG_open << "Handling client from job " << i->job_id << "... ";
2296 complete_request(*i, false);
2298 DEBUG_close << "failed\n";
2302 // Remove running targets that have finished.
2303 for (string_set::iterator j = i->running.begin(), j_next = j,
2304 j_end = i->running.end(); j != j_end; j = j_next)
2307 status_map::const_iterator k = status.find(*j);
2308 assert(k != status.end());
2309 switch (k->second.status)
2314 if (!keep_going) goto failed;
2319 i->running.erase(j);
2327 // Start pending targets.
2328 while (!i->pending.empty())
2330 std::string target = i->pending.front();
2331 i->pending.pop_front();
2332 switch (get_status(target).status)
2335 i->running.insert(target);
2339 if (!keep_going) goto failed;
2347 client_list::iterator j = i;
2348 if (!start(target, i)) goto pending_failed;
2349 j->running.insert(target);
2350 if (!has_free_slots()) return true;
2351 // Job start might insert a dependency client.
2358 // Try to complete the request.
2359 // (This might start a new job if it was a dependency client.)
2360 if (i->running.empty())
2362 if (i->failed) goto failed;
2363 complete_request(*i, true);
2365 DEBUG_close << "finished\n";
2370 if (running_jobs != waiting_jobs) return true;
2371 if (running_jobs == 0 && clients.empty()) return false;
2373 // There is a circular dependency.
2374 // Try to break it by completing one of the requests.
2375 assert(!clients.empty());
2376 std::cerr << "Circular dependency detected" << std::endl;
2377 client_list::iterator i = clients.begin();
2378 complete_request(*i, false);
2384 * Create a named unix socket that listens for build requests. Also set
2385 * the REMAKE_SOCKET environment variable that will be inherited by all
2388 static void create_server()
2393 perror("Failed to create server");
2399 DEBUG_open << "Creating server... ";
2402 // Prepare a windows socket.
2403 struct sockaddr_in socket_addr;
2404 socket_addr.sin_family = AF_INET;
2405 socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
2406 socket_addr.sin_port = 0;
2408 // Create and listen to the socket.
2409 socket_fd = socket(AF_INET, SOCK_STREAM, 0);
2410 if (socket_fd == INVALID_SOCKET) goto error;
2411 if (!SetHandleInformation((HANDLE)socket_fd, HANDLE_FLAG_INHERIT, 0))
2413 if (bind(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
2415 int len = sizeof(sockaddr_in);
2416 if (getsockname(socket_fd, (struct sockaddr *)&socket_addr, &len))
2418 std::ostringstream buf;
2419 buf << socket_addr.sin_port;
2420 if (!SetEnvironmentVariable("REMAKE_SOCKET", buf.str().c_str()))
2422 if (listen(socket_fd, 1000)) goto error;
2424 // Set signal handlers for SIGCHLD and SIGINT.
2425 // Block SIGCHLD (unblocked during select).
2427 sigemptyset(&sigmask);
2428 sigaddset(&sigmask, SIGCHLD);
2429 if (sigprocmask(SIG_BLOCK, &sigmask, NULL) == -1) goto error;
2430 struct sigaction sa;
2432 sigemptyset(&sa.sa_mask);
2433 sa.sa_handler = &sigchld_handler;
2434 if (sigaction(SIGCHLD, &sa, NULL) == -1) goto error;
2435 sa.sa_handler = &sigint_handler;
2436 if (sigaction(SIGINT, &sa, NULL) == -1) goto error;
2438 // Prepare a named unix socket in temporary directory.
2439 socket_name = tempnam(NULL, "rmk-");
2440 if (!socket_name) goto error2;
2441 struct sockaddr_un socket_addr;
2442 size_t len = strlen(socket_name);
2443 if (len >= sizeof(socket_addr.sun_path) - 1) goto error2;
2444 socket_addr.sun_family = AF_UNIX;
2445 strcpy(socket_addr.sun_path, socket_name);
2446 len += sizeof(socket_addr.sun_family);
2447 if (setenv("REMAKE_SOCKET", socket_name, 1)) goto error;
2449 // Create and listen to the socket.
2451 socket_fd = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
2452 if (socket_fd == INVALID_SOCKET) goto error;
2454 socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
2455 if (socket_fd == INVALID_SOCKET) goto error;
2456 if (fcntl(socket_fd, F_SETFD, FD_CLOEXEC) < 0) goto error;
2458 if (bind(socket_fd, (struct sockaddr *)&socket_addr, len))
2460 if (listen(socket_fd, 1000)) goto error;
2465 * Accept a connection from a client, get the job it spawned from,
2466 * get the targets, and mark them as dependencies of the job targets.
2468 void accept_client()
2470 DEBUG_open << "Handling client request... ";
2472 // Accept connection.
2474 socket_t fd = accept(socket_fd, NULL, NULL);
2475 if (fd == INVALID_SOCKET) return;
2476 if (!SetHandleInformation((HANDLE)fd, HANDLE_FLAG_INHERIT, 0))
2479 std::cerr << "Unexpected failure while setting connection with client" << std::endl;
2483 // WSAEventSelect puts sockets into nonblocking mode, so disable it here.
2485 if (ioctlsocket(fd, FIONBIO, &nbio)) goto error2;
2486 #elif defined(LINUX)
2487 int fd = accept4(socket_fd, NULL, NULL, SOCK_CLOEXEC);
2490 int fd = accept(socket_fd, NULL, NULL);
2492 if (fcntl(fd, F_SETFD, FD_CLOEXEC) < 0) return;
2494 clients.push_front(client_t());
2495 client_list::iterator proc = clients.begin();
2500 DEBUG_close << "failed\n";
2501 std::cerr << "Received an ill-formed client message" << std::endl;
2507 clients.erase(proc);
2511 // Receive message. Stop when encountering two nuls in a row.
2512 std::vector<char> buf;
2514 while (len < sizeof(int) + 2 || buf[len - 1] || buf[len - 2])
2516 buf.resize(len + 1024);
2517 ssize_t l = recv(fd, &buf[0] + len, 1024, 0);
2518 if (l <= 0) goto error;
2522 // Parse job that spawned the client.
2524 memcpy(&job_id, &buf[0], sizeof(int));
2526 proc->job_id = job_id;
2527 job_targets_map::const_iterator i = job_targets.find(job_id);
2528 if (i == job_targets.end()) goto error;
2529 DEBUG << "receiving request from job " << job_id << std::endl;
2531 // Parse the targets and mark them as dependencies from the job targets.
2532 dependency_t &dep = *dependencies[job_targets[job_id].front()];
2533 // Retrieve the variables of the job, pass them to the client.
2534 //proc->variables = job_variables[job_id];
2536 variable_map &vars = job_variables[job_id];
2538 char const *p = &buf[0] + sizeof(int);
2542 if (len == 1 && p[0] == 1)
2544 //Finished parsing targets.
2549 std::string target(p, p + len);
2550 DEBUG << "adding dependency " << target << " to job\n";
2551 proc->pending.push_back(target);
2552 dep.deps.insert(target);
2559 if (len == 0) return;
2560 std::string line(p, p + len);
2561 std::istringstream in (line);
2562 std::string name = read_word(in);
2563 if (expect_token(in, Equal) == Unexpected) {
2564 std::cerr << '\'' << line << "'" << std::endl;
2567 DEBUG << "adding variable " << line << " to job " << job_id << std::endl;
2568 read_words(in, vars[name]);
2574 * Handle child process exit status.
2576 void finalize_job(pid_t pid, bool res)
2578 pid_job_map::iterator i = job_pids.find(pid);
2579 assert(i != job_pids.end());
2580 int job_id = i->second;
2583 complete_job(job_id, res);
2587 * Loop until all the jobs have finished.
2589 * @post There are no client requests left, not even virtual ones.
2593 while (handle_clients())
2595 DEBUG_open << "Handling events... ";
2597 size_t len = job_pids.size() + 1;
2600 for (pid_job_map::const_iterator i = job_pids.begin(),
2601 i_end = job_pids.end(); i != i_end; ++i, ++num)
2605 WSAEVENT aev = WSACreateEvent();
2607 WSAEventSelect(socket_fd, aev, FD_ACCEPT);
2608 DWORD w = WaitForMultipleObjects(len, h, false, INFINITE);
2609 WSAEventSelect(socket_fd, aev, 0);
2620 bool res = GetExitCodeProcess(pid, &s) && s == 0;
2622 finalize_job(pid, res);
2625 sigemptyset(&emptymask);
2628 FD_SET(socket_fd, &fdset);
2629 int ret = pselect(socket_fd + 1, &fdset, NULL, NULL, NULL, &emptymask);
2630 if (ret > 0 /* && FD_ISSET(socket_fd, &fdset)*/) accept_client();
2631 if (!got_SIGCHLD) continue;
2635 while ((pid = waitpid(-1, &status, WNOHANG)) > 0)
2637 bool res = WIFEXITED(status) && WEXITSTATUS(status) == 0;
2638 finalize_job(pid, res);
2643 assert(clients.empty());
2647 * Load dependencies and rules, listen to client requests, and loop until
2648 * all the requests have completed.
2649 * If Remakefile is obsolete, perform a first run with it only, then reload
2650 * the rules, and perform a second with the original clients.
2652 void server_mode(std::string const &remakefile, string_list const &targets)
2654 load_dependencies();
2655 load_rules(remakefile);
2657 if (get_status(remakefile).status != Uptodate)
2659 clients.push_back(client_t());
2660 clients.back().pending.push_back(remakefile);
2662 if (build_failure) goto early_exit;
2664 specific_rules.clear();
2665 generic_rules.clear();
2666 first_target.clear();
2667 load_rules(remakefile);
2669 clients.push_back(client_t());
2670 if (!targets.empty()) clients.back().pending = targets;
2671 else if (!first_target.empty())
2672 clients.back().pending.push_back(first_target);
2677 remove(socket_name);
2680 save_dependencies();
2681 exit(build_failure ? EXIT_FAILURE : EXIT_SUCCESS);
2687 * @defgroup client Client
2693 * Connect to the server @a socket_name, send a build request for @a targets,
2694 * and exit with the status returned by the server.
2696 void client_mode(char *socket_name, string_list const &targets)
2701 perror("Failed to send targets to server");
2704 if (targets.empty()) exit(EXIT_SUCCESS);
2705 DEBUG_open << "Connecting to server... ";
2707 // Connect to server.
2709 struct sockaddr_in socket_addr;
2710 socket_fd = socket(AF_INET, SOCK_STREAM, 0);
2711 if (socket_fd == INVALID_SOCKET) goto error;
2712 socket_addr.sin_family = AF_INET;
2713 socket_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
2714 socket_addr.sin_port = atoi(socket_name);
2715 if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(sockaddr_in)))
2718 struct sockaddr_un socket_addr;
2719 size_t len = strlen(socket_name);
2720 if (len >= sizeof(socket_addr.sun_path) - 1) exit(EXIT_FAILURE);
2721 socket_fd = socket(AF_UNIX, SOCK_STREAM, 0);
2722 if (socket_fd == INVALID_SOCKET) goto error;
2723 socket_addr.sun_family = AF_UNIX;
2724 strcpy(socket_addr.sun_path, socket_name);
2725 if (connect(socket_fd, (struct sockaddr *)&socket_addr, sizeof(socket_addr.sun_family) + len))
2729 if (setsockopt(socket_fd, SOL_SOCKET, SO_NOSIGPIPE, &set_option, sizeof(set_option)))
2734 // Send current job id.
2735 char *id = getenv("REMAKE_JOB_ID");
2736 int job_id = id ? atoi(id) : -1;
2737 if (send(socket_fd, (char *)&job_id, sizeof(job_id), MSG_NOSIGNAL) != sizeof(job_id))
2741 for (string_list::const_iterator i = targets.begin(),
2742 i_end = targets.end(); i != i_end; ++i)
2744 DEBUG_open << "Sending " << *i << "... ";
2745 ssize_t len = i->length() + 1;
2746 if (send(socket_fd, i->c_str(), len, MSG_NOSIGNAL) != len)
2749 // send 10 as as separator between targets and variables.
2751 if (send(socket_fd, &result, 1, MSG_NOSIGNAL) != 1) goto error;
2753 if (send(socket_fd, &result, 1, MSG_NOSIGNAL) != 1) goto error;
2755 // (maybe split vars in small chunks and send... seems overkil)
2756 std::string vars = serialize_variables(variables, 0);
2757 ssize_t sent = vars.size();
2758 DEBUG << "Sending variables: '" << vars << "' to the server" << std::endl;
2759 if (send(socket_fd, vars.data(), sent, MSG_NOSIGNAL) != sent)
2762 // Send terminating nul and wait for reply.
2764 if (send(socket_fd, &result, 1, MSG_NOSIGNAL) != 1) goto error;
2765 if (recv(socket_fd, &result, 1, 0) != 1) exit(EXIT_FAILURE);
2766 exit(result ? EXIT_SUCCESS : EXIT_FAILURE);
2772 * @defgroup ui User interface
2778 * Display usage and exit with @a exit_status.
2780 void usage(int exit_status)
2782 std::cerr << "Usage: remake [options] [target] ...\n"
2784 " -d Echo script commands.\n"
2785 " -d -d Print lots of debugging information.\n"
2786 " -f FILE Read FILE as Remakefile.\n"
2787 " -h, --help Print this message and exit.\n"
2788 " -j[N], --jobs=[N] Allow N jobs at once; infinite jobs with no arg.\n"
2789 " -k Keep going when some targets cannot be made.\n"
2790 " -v V=X, --var V=X Initialize variable V with X.\n"
2791 " -r Look up targets from the dependencies on stdin.\n"
2792 " -s, --silent, --quiet Do not echo targets.\n";
2797 * This program behaves in two different ways.
2799 * - If the environment contains the REMAKE_SOCKET variable, the client
2800 * connects to this socket and sends to the server its build targets.
2801 * It exits once it receives the server reply.
2803 * - Otherwise, it creates a server that waits for build requests. It
2804 * also creates a pseudo-client that requests the targets passed on the
2807 int main(int argc, char *argv[])
2811 std::string remakefile = "Remakefile";
2812 string_list targets;
2813 bool indirect_targets = false;
2815 // Parse command-line arguments.
2816 for (int i = 1; i < argc; ++i)
2818 std::string arg = argv[i];
2819 if (arg.empty()) usage(EXIT_FAILURE);
2820 if (arg == "-h" || arg == "--help") usage(EXIT_SUCCESS);
2822 if (echo_scripts) debug.active = true;
2823 else echo_scripts = true;
2824 else if (arg == "-k" || arg =="--keep-going")
2826 else if (arg == "-s" || arg == "--silent" || arg == "--quiet")
2827 show_targets = false;
2828 else if (arg == "-r")
2829 indirect_targets = true;
2830 else if (arg == "-f")
2832 if (++i == argc) usage(EXIT_FAILURE);
2833 remakefile = argv[i];
2835 else if (arg.compare(0, 2, "-j") == 0)
2836 max_active_jobs = atoi(arg.c_str() + 2);
2837 else if (arg.compare(0, 7, "--jobs=") == 0)
2838 max_active_jobs = atoi(arg.c_str() + 7);
2839 else if (arg == "-v" || arg == "--var")
2842 if (i == argc) usage(EXIT_FAILURE);
2844 std::istringstream in (arg);
2845 std::string name = read_word(in);
2846 if (expect_token(in, Equal) == Unexpected) {
2847 std::cerr << "Invalid variable '" << arg << "'" << std::endl;
2850 read_words(in, variables[name]);
2854 if (arg[0] == '-') usage(EXIT_FAILURE);
2855 targets.push_back(normalize(arg));
2856 DEBUG << "New target: " << arg << '\n';
2860 if (indirect_targets)
2862 load_dependencies(std::cin);
2865 if (l.empty() && !dependencies.empty())
2867 l.push_back(dependencies.begin()->second->targets.front());
2869 for (string_list::const_iterator i = l.begin(),
2870 i_end = l.end(); i != i_end; ++i)
2872 dependency_map::const_iterator j = dependencies.find(*i);
2873 if (j == dependencies.end()) continue;
2874 dependency_t const &dep = *j->second;
2875 for (string_set::const_iterator k = dep.deps.begin(),
2876 k_end = dep.deps.end(); k != k_end; ++k)
2878 targets.push_back(normalize(*k));
2881 dependencies.clear();
2886 if (WSAStartup(MAKEWORD(2,2), &wsaData))
2888 std::cerr << "Unexpected failure while initializing Windows Socket" << std::endl;
2893 // Run as client if REMAKE_SOCKET is present in the environment.
2894 if (char *sn = getenv("REMAKE_SOCKET")) client_mode(sn, targets);
2896 // Otherwise run as server.
2897 server_mode(remakefile, targets);