;;; -*- Mode:gate; Fonts:(HL12 HL12I HL12B CPTFONTB HL12BI HL12B HL12I ) -*-

=Node: Introduction

=Text:
3INTRODUCTION*

     When a program gets large, it is often desirable to split it up into several files.  One reason for
this is to help keep the parts of the program organized, to make things easier to find.  It's also useful
to have the program broken into small pieces that are more convenient to edit and compile.  It is
particularly important to avoid the need to recompile all of a large program every time any piece of it
changes; if the program is broken up into many files, only the files that have changes in them need
to be recompiled.

     The apparent drawback to splitting up a program is that more commands are needed to
manipulate it.  To load the program, you now have to load several files separately, instead of just
loading one file.  To compile it, you have to figure out which files need compilation, by seeing which
have been edited since they were last compiled, and then you have to compile those files.

     What's even more complicated is that files can have interdependencies.  You might have a file
called 2DEFS* that contains some macro definitions (or flavor or structure definitions), and functions in
other files might use those macros.  This means that in order to compile any of those other files,
you must first load the file 2DEFS* into the Lisp environment so that the macros will be defined and
can be expanded at compile time.  You have to remember this whenever you compile any of those
files.  Furthermore, if 2DEFS* has changed, other files of the program may need to be recompiled
because the macros may have changed and need to be re-expanded.

     This chapter describes the 1system* facility, which takes care of all these things for you.  The
way it works is that you define a set of files to be a 1system*, using the 2defsystem* special form,
described below.  This system definition says which files make up the system, which ones depend
on the presence of others, and so on.  You put this system definition into its own little file, and then
all you have to do is load that file and the Lisp environment will know about your system and what
files are in it.  You can then use the 2make-system* function (see 4(SYSTEMS-1)Making a System*) 
to load in all the files of the system, recompile all the files that need compiling, and so on.

     The system facility is very general and extensible.  This chapter explains how to use it and how
to extend it.  This chapter also explains the 1patch* facility, which lets you conveniently update a large
program with incremental changes.

=Node: 4Defining a System*

=Text:
3DEFINING A SYSTEM

defsystem* 1name* 1(keyword* 1args...)...*                                                            1Macro*        
        Defines a system named 1name*.  The options selected by the keywords are explained in detail
        later.  In general, they fall into two categories:  properties of the system and 1transformations*.
        A transformation is an operation such as compiling or loading that takes one or more files and
        does something to them.  The simplest system is a set of files and a transformation to be
        performed on them.  

Here are a few examples.  

        3(defsystem mysys*
        3  (:compile-load ("OZ:<GEORGE>PROG1.LISP" "OZ:<GEORGE2>PROG2.LISP")))*

        3(defsystem zmail*
        3  (:name "ZMail")*
        3  (:pathname-default "SYS: ZMAIL;")*
        3  (:package zwei)*
        3  (:module defs "DEFS")*
        3  (:module mult "MULT" :package tv)*
        3  (:module main ("TOP" "COMNDS" "MAIL" "USER" "WINDOW"*
        3                 "FILTER" mult "COMETH"))*
        3  (:compile-load defs)*
        3  (:compile-load main (:fasload defs)))*

        3(defsystem bar*
        3  (:module reader-macros "BAR:BAR;RDMAC")*
        3  (:module other-macros "BAR:BAR;MACROS")*
        3  (:module main-program "BAR:BAR;MAIN")*
        3  (:compile-load reader-macros)*
        3  (:compile-load other-macros (:fasload reader-macros))*
        3  (:compile-load main-program (:fasload reader-macros*
        3                                        other-macros)))*

     The first example defines a new 1system* called 2mysys*, which consists of two files, stored on a
Tops-20 host names OZ, both of which are to be compiled and loaded.  The second example is
somewhat more complicated.  What all the options mean is described below, but the primary
difference is that there is a file 2DEFS* which must be loaded before the rest of the files (2main*) can be
compiled.  Also, the files are stored on logical host SYS and directory ZMAIL.

     The last example has two levels of dependency.  2reader-macros* must be compiled and loaded
before 2other-macros* can be compiled.  Both 2reader-macros* and 2other-macros* must then be loaded
before 2main-program* can be compiled.  All the source files are stored on host BAR, presumably a
logical host defined specifically for this system.  It is desirable to use a logical host for the files of a
system if there is a chance that people at more than one site will be using it; the logical host allows
the identical 2defsystem* to be valid at all sites.  See 4(FILENAMES-3)Logical Pathnames* for more on
logical hosts and logical pathnames.

     Note that The 2defsystem* options other than transformations are:  

2:name*        Specifies a ``pretty'' version of the name for the system, for use in printing.

2:short-name*  
              Specified an abbreviated name used in constructing disk label comments and in patch file
              names for some file systems.

2:component-systems*  
              Specifies the names of other systems used to make up this system.  Performing an
              operation on a system with component systems is equivalent to performing the same
              operation on all the individual systems.  The format is 2(:component-systems 1names*...)*.

2:package*    Specifies the package in which transformations are performed.  A package specified here
              overrides one in the 3-*-* line of the file in question.

2:pathname-default*  
              Gives a local default within the definition of the system for strings to be parsed into
              pathnames.  Typically this specifies the directory, when all the files of a system are on
              the same directory.

2:warnings-pathname-default*  
              Gives a default for the file to use to store compiler warnings in, when 2make-system* is
              used with the 2:batch* option.

2:patchable*   Makes the system be a patchable system (see 4(SYSTEMS-2)The Patch Facility*).  
	      An optional argument specifies the directory to put patch files in.  The default is the
              2:pathname-default* of the system.

2:initial-status*  
              Specifies what the status of the system should be when 2make-system* is used to create
              a new major version.  The default is 2:experimental*.  See 4(SYSTEMS-2)System Status* for
              further details.

2:not-in-disk-label*  
              Make a patchable system not appear in the disk label comment.  This should probably
              never be specified for a user system.  It is used by patchable systems internal to the
              main Lisp system, to avoid cluttering up the label.

2:default-binary-file-type*  
              Specifies the file type to use for compiled Lisp files.  The value you specify should be a
              string.  If you do not specify this, the standard file type 2:qfasl* is used.

2:module*      Allows assigning a name to a set of files within the system.  This name can then be
              used instead of repeating the filenames.  The format is 2(:module 1name* 1files* 1options*...)*.
              1files* is usually a list of filenames (strings).  In general, it is a 1module-specification*, which
              can be any of the following:  

              2a string    *  
                            This is a file name.

              2a symbol*    This is a module name.  It stands for all of the files which are in that module
                            of this system.

              2an 1external module component**  
                            This is a list of the form 2(1system-name* 1module-names*...)*, to specify
                            modules in another system.  It stands for all of the files which are in all of
                            those modules.

              2a list of 1module components**  
                            A module component is any of the above, or the following:

              2a list of file names*  
                            This is used in the case where the names of the input and output files of a
                            transformation are not related according to the standard naming conventions,
                            for example when a QFASL file has a different name or resides on a
                            different directory than the source file.  The file names in the list are used
                            from left to right, thus the first name is the source file.  Each file name after
                            the first in the list is defaulted from the previous one in the list.

                            To avoid syntactic ambiguity, this is allowed as a module component but not
                            as a module specification.  

              The currently defined options for the 2:module* clause are 

              2:package*    Overrides any package specified for the whole system for transformations
                            performed on just this module.  

              In the second 2defsystem* example above, there are three modules.  Each of the first two
              has only one file, and the third one (2main*) is made up both of files and another module.
              To take examples of the other possibilities, 

                      3(:module prog (("SYS: GEORGE; PROG" "SYS: GEORG2; PROG")))*
                      3(:module foo (defs (zmail defs)))*

              The 2prog* module consists of one file, but it lives in two directories, 2GEORGE* and
              2GEORG2*.  If this were a Lisp program, that would mean that the file 2SYS:  GEORGE;*
              2PROG LISP* would be compiled into 2SYS:  GEORG2; PROG QFASL*.  The 2foo* module
              consists of two other modules the 2defs* module in the same system, and the 2defs* module
              in the 2zmail* system.  It is not generally useful to compile files that belong to other
              systems; thus this 2foo* module would not normally be the subject of a transformation.
              However, 1dependencies* (defined below) use modules and need to be able to refer to
              (depend on) modules of other systems.  

3si:set-system-source-file* 1system-name* 1filename*                                                     
        This function specifies which file contains the 2defsystem* for the system 1system-name*.
        1filename* can be a pathname object or a string.

        Sometimes it is useful to say where the definition of a system can be found without taking
        time to load that file.  If 2make-system*, or 2require* (4(SYSTEMS-2)Common Lisp Modules*), is
        ever used on that system, the file whose name has been specified will be loaded
        automatically.  

=Node: 4Transformations*

=Text:
3TRANSFORMATIONS*

     Transformations are of two types, simple and complex.  A simple transformation is a single
operation on a file, such as compiling it or loading it.  A complex transformation takes the output
from one transformation and performs another transformation on it, such as loading the results of
compilation.

     The general format of a simple transformation is 2(1name* 1input* 1dependencies* 1condition*)*.  1input* is
usually a module specification or another transformation whose output is used.  The transformation
1name* is to be performed on all the files in the module, or all the output files of the other
transformation.

     1dependencies* and 1condition* are optional.

     1dependencies* is a 1transformation specification*, either a list 2(1transformation-name
module-names*...)* or a list of such lists.  A 1module-name* is either a symbol that is the name of a
module in the current system, or a list 2(1system-name* 1module-names*...)*.  A dependency declares that
all of the indicated transformations must be performed on the indicated modules before the current
transformation itself can take place.  Thus in the zmail example above, the 2defs* module must have
the 2:fasload* transformation performed on it before the 2:compile* transformation can be performed on
2main*.

     The dependency has to be a tranformation that is explicitly specified as a transformation in the
system definition, not just an action that might be performed by anything.  That is, if you have a
dependency 2(:fasload foo)*, it means that 2(fasload foo)* is a tranformation of your system and you
depend on that tranformation; it does not simply mean that you depend on 2foo*'s being loaded.
Furthermore, it doesn't work if 2(:fasload foo)* is an implicit piece of another tranformation.  For
example, the following works:  

        3(defsystem foo*
        3  (:module foo "FOO")*
        3  (:module bar "BAR")*
        3  (:compile-load (foo bar)))*

     but this doesn't work:  

        3(defsystem foo*
        3  (:module foo "FOO")*
        3  (:module bar "BAR")*
        3  (:module blort "BLORT")*
        3  (:compile-load (foo bar))*
        3  (:compile-load blort (:fasload foo)))*

because 2foo*'s 2:fasload* is not mentioned explicitly (i.e.  at top level) but is only implicit in the
2(:compile-load (foo bar))*.  One must instead write:  

        3(defsystem foo*
        3  (:module foo "FOO")*
        3  (:module bar "BAR")*
        3  (:module blort "BLORT")*
        3  (:compile-load foo)*
        3  (:compile-load bar)*
        3  (:compile-load blort (:fasload foo)))*

     1condition* is a predicate which specifies when the transformation should take place.  Generally it
defaults according to the type of the transformation.  Conditions are discussed further on
4(SYSTEMS-2)More Esoteric Transformations*.

     The defined simple transformations are:  

2:fasload*      Calls the 2fasload* function to load the indicated files, which must be QFASL files whose
              pathnames have canonical type 2:qfasl* (see 4(FILENAMES-1)Canonical Types*).  The
              1condition* defaults to 2si:file-newer-than-installed-p*, which is 2t* if a newer version of the
              file exists on the file computer than was read into the current environment.

2:readfile*     Calls the 2readfile* function to read in the indicated files, whose names must have
              canonical type 2:lisp*.  Use this for files that are not to be compiled.  1condition* defaults to
              2si:file-newer-than-installed-p*.

2:compile*     Calls the 2compile-file* function to compile the indicated files, whose names must have
              canonical type 2:lisp*.  1condition* defaults to 2si:file-newer-than-file-p*, which returns 2t* if
              the source file has been written more recently than the binary file.  

     A special simple transformation is 

2:do-components*  
              2(:do-components 1dependencies*)* inside a system with component systems causes the
              1dependencies* to be done before anything in the component systems.  This is useful
              when you have a module of macro files used by all of the component systems.  

     The defined complex transformations are 

2:compile-load*  
              2(:compile-load 1input* 1compile-dependencies* 1load-dependencies* 1compile-condition**
              1load-condition2)** is the same as 2(:fasload (:compile 1input* 1compile-dependencies**
              1compile-condition2) *load-dependencies2 *load-condition2)**.  This is the most commonly-used
              transformation.  Everything after 1input* is optional.

2:compile-load-init*  
              See 4(SYSTEMS-2)More Esoteric Transformations*.  

     As was explained above, each filename in an input specification can in fact be a list of strings
when the source file of a program differs from the binary file in more than just the file type.  In fact,
every filename is treated as if it were an infinite list of filenames with the last filename, or in the
case of a single string the only filename, repeated forever at the end.  Each simple transformation
takes some number of input filename arguments and some number of output filename arguments.  As
transformations are performed, these arguments are taken from the front of the filename list.  The
input arguments are actually removed and the output arguments left as input arguments to the next
higher transformation.  To make this clearer, consider the 2prog* module above having the
2:compile-load* transformation performed on it.  This means that 2prog* is given as the input to the
2:compile* transformation and the output from this transformation is given as the input to the 2:fasload*
transformation.  The 2:compile* transformation takes one input filename argument, the name of a Lisp
source file, and one output filename argument, the name of the QFASL file.  The 2:fasload*
transformation takes one input filename argument, the name of a QFASL file, and no output filename
arguments.  So, for the first and only file in the 2prog* module, the filename argument list looks like
2("SYS:  GEORGE; PROG" "SYS:  GEORG2; PROG" "SYS:  GEORG2; PROG" ...)*.  The 2:compile*
transformation is given arguments of 2"SYS:  GEORGE; PROG"* and 2"SYS:  GEORG2; PROG"* and the
filename argument list which it outputs as the input to the 2:fasload* transformation is 2("SYS:  GEORG2;
PROG" "SYS:  GEORG2; PROG" ...)*.  The 2:fasload* transformation then is given its one argument of
2"SYS:  GEORG2; PROG"*.

     Note that dependencies are not transitive or inherited.  For example, if module 2a* depends on
macros defined in module 2b*, and therefore needs 2b* to be loaded in order to compile, and 2b* has a
similar dependency on 2c*, 2c* need not be loaded for compilation of 2a*.  Transformations with these
dependencies would be written 

        3(:compile-load a (:fasload b))*
        3(:compile-load b (:fasload c))*

     To say that compilation of 2a* depends on both 2b* and 2c*, you would instead write 

        3(:compile-load a (:fasload b c))*
        3(:compile-load b (:fasload c))*

     If in addition 2a* depended on 2c* (but not 2b*) during loading (perhaps 2a* contains 2defvar*s whose
initial values depend on functions or special variables defined in 2c*) you would write the
transformations 

        3(:compile-load a (:fasload b c) (:fasload c))*
        3(:compile-load b (:fasload c))*

=Node: 4Making a System*

=Text:
3MAKING A SYSTEM

make-system* 1name* &rest 1keywords*                                                                           
        The 2make-system* function does the actual work of compiling and loading.  In the example
        above, if 2PROG1* and 2PROG2* have both been compiled recently, then 

                3(make-system 'mysys)*

        loads them as necessary.  If either one might also need to be compiled, then 

                3(make-system 'mysys :compile)*

        does that first as necessary.

        The very first thing 2make-system* does is check whether the file which contains the
        2defsystem* for the specified system has changed since it was loaded.  If so, it offers to load
        the latest version, so that the remainder of the 2make-system* can be done using the latest
        system definition.  (This only happens if the filetype of that file is 2LISP*.) After loading this file
        or not, 2make-system* goes on to process the files that compose the system.

        If the system name is not recognized, 2make-system* attempts to load the file 2SYS:  SITE;*
        1system-name2 SYSTEM**, in the hope that that contains a system definition or a call to
        2si:set-system-source-file*.

        2make-system* lists what transformations it is going to perform on what files, then asks the
        user for confirmation.  If the user types S when confirmation is requested, then 2make-system*
        asks about each file individually so that the user can decide selectively which transformations
        should be performed; then collective reconfirmation is requested.  This is like what happens if
        the 2:selective* keyword is specified.  If the user types Y, the transformations are performed.
        Before each transformation a message is printed listing the transformation being performed,
        the file it is being done to, and the package.  This behavior can be altered by 1keywords*.

        If the system being made is patchable, and if loading has not been inhibited, then the system's
        patches are loaded afterward.  Loading of patches is silent if the 2make-system* is, and
        requires confirmation if the 2make-system* does.  

     These are the keywords recognized by the 2make-system* function and what they do.  

2:noconfirm*  Assumes a yes answer for all questions that would otherwise be asked of the user.

2:selective*    Asks the user whether or not to perform each transformation that appears to be needed
              for each file.

2:silent*        Avoids printing out each transformation as it is performed.

2:reload*       Bypasses the specified conditions for performing a transformation.  Thus files are
              compiled even if they haven't changed and loaded even if they aren't newer than the
              installed version.

2:noload*      Does not load any files except those required by dependencies.  For use in conjunction
              with the 2:compile* option.

2:compile*     Compiles files also if need be.  The default is to load but not compile.

2:recompile*   This is equivalent to a combination of 2:compile* and 2:reload*:  it specifies compilation of all
              files, even those whose sources have not changed since last compiled.

2:no-increment-patch*  
              When given along with the 2:compile* option, disables the automatic incrementing of the
              major system version that would otherwise take place.  See
              4(SYSTEMS-2)The Patch Facility*.

2:increment-patch*  
              Increments a patchable system's major version without doing any compilations.  See
              4(SYSTEMS-2)The Patch Facility*.

2:no-reload-system-declaration*  
              Turns off the check for whether the file containing the 2defsystem* has been changed.
              Then the file is loaded only if it has never been loaded before.

2:batch*       Allows a large compilation to be done unattended.  It acts like 2:noconfirm* with regard to
              questions, turns off more-processing and fdefine-warnings (see 2inhibit-fdefine-warnings*,
              4(FUNCTIONS-2)How Programs Manipulate Function Specs*), and saves the compiler
              warnings in an editor buffer and a file (it asks you for the name).

2:defaulted-batch*  
              This is like 2:batch* except that it uses the default for the pathname to store warnings in
              and does not ask the user to type a pathname.

2:print-only*  Just prints out what transformations would be performed; does not actually do any
              compiling or loading.

2:noop*        Is ignored.  This is useful mainly for programs that call 2make-system*, so that such
              programs can include forms like 

                      3(make-system 'mysys (if compile-p :compile :noop))*

=Node: 4Adding New Keywords to make-system*

=Text:
3ADDING NEW KEYWORDS TO MAKE-SYSTEM*

     2make-system* keywords are defined as functions on the 2si:make-system-keyword* property of
the keyword.  The functions are called with no arguments.  Some of the relevant variables they can
use are

3si:*system-being-made**                                                                  1Variable*        
        The internal data structure that represents the system being made.  

3si:*make-system-forms-to-be-evaled-before**                                    1Variable*        
        A list of forms that are evaluated before the transformations are performed.  

3si:*make-system-forms-to-be-evaled-after**                                      1Variable*        
        A list of forms that are evaluated after the transformations have been performed.
        Transformations can push entries here too.  

3si:*make-system-forms-to-be-evaled-finally**                                   1Variable*        
        A list of forms that are evaluated by an 2unwind-protect* when the body of 2make-system* is
        exited, whether it is completed or not.  Closing the batch warnings file is done here.  Unlike
        the 2si:*make-system-forms-to-be-evaled-after** forms, these forms are evaluated outside of
        the ``compiler warnings context''.  

3si:*query-type**                                                                             1Variable*        
        Controls how questions are asked.  Its normal value is 2:normal*.  2:noconfirm* means ask no
        questions and 2:selective* means asks a question for each individual file transformation.  

3si:*silent-p**                                                                                1Variable*        
        If 2t*, no messages are printed out.  

3si:*batch-mode-p**                                                                          1Variable*        
        If 2t*, 2:batch* was specified.  

3si:*redo-all**                                                                                1Variable*        
        If 2t*, all transformations are performed, regardless of the condition functions.  

3si:*top-level-transformations**                                                      1Variable*        
        A list of the types of transformations that should be performed, such as 2(:fasload :readfile)*.
        The contents of this list are controlled by the keywords given to 2make-system*.  This list
        then controls which transformations are actually performed.  

3si:*file-transformation-function**                                                  1Variable*        
        The actual function that gets called with the list of transformations that need to be performed.
        The default is 2si:do-file-transformations*.  

3si:define-make-system-special-variable* 1variable* 1value* 1[defvar-p]*              1Macro*        
        Causes 1variable* to be bound to 1value* during the body of the call to 2make-system*.  This
        allows you to define new variables similar to those listed above.  1value* is evaluated on entry
        to 2make-system*.  If 1defvar-p* is specified as (or defaulted to) 2t*, 1variable* is defined with
        2defvar*.  It is not given an initial value.  If 1defvar-p* is specified as 1nil*, 1variable* belongs to
        some other program and is not 2defvar*'ed here.  

     The following simple example adds a new keyword to 2make-system* called 2:just-warn*, which
means that 2fdefine* warnings (see 4(FUNCTIONS-2)How Programs Manipulate Function Specs*)
regarding functions being overwritten should be printed out, but the user should not be queried.  

        3(si:define-make-system-special-variable*
        3   inhibit-fdefine-warnings inhibit-fdefine-warnings nil)*

        3(defun (:just-warn si:make-system-keyword) ()*
        3  (setq inhibit-fdefine-warnings :just-warn))*

     (See the description of the 2inhibit-fdefine-warnings* variable, on
4(FUNCTIONS-2)How Programs Manipulate Function Specs*.)

     2make-system* keywords can do something directly when called, or they can have their effect by
pushing a form to be evaluated onto 2si:*make-system-forms-to-be-evaled-after** or one of the
other two similar lists.  In general, the only useful thing to do is to set some special variable defined
by 2si:define-make-system-special-variable*.  In addition to the ones mentioned above, user-defined
transformations may have their behavior controlled by new special variables, which can be set by
new keywords.  If you want to get at the list of transformations to be performed, for example, the
right way is to set 2si:*file-transformation-function** to a new function, which then can call
2si:do-file-transformations* with a possibly modified list.  That is how the 2:print-only* keyword works.