;;; -*- Mode:gate; Fonts:(CPTFONT CPTFONTB) -*- ;;; -*- Mode:gate; Fonts:(HL12 HL12I HL12B CPTFONTB HL12BI HL12B HL12I ) -*- =Node: Hints to Macro Writers =Text: 3HINTS* 3TO MACRO WRITERS* There are many useful techniques for writing macros. Over the years, Lisp programmers have discovered techniques that most programmers find useful, and have identified pitfalls that must be avoided. This section discusses some of these techniques and illustrates them with examples. The most important thing to keep in mind as you learn to write macros is that the first thing you should do is figure out what the macro form is supposed to expand into, and only then should you start to actually write the code of the macro. If you have a firm grasp of what the generated Lisp program is supposed to look like, you will find the macro much easier to write. In general any macro that can be written as a substitutable function (see 4(MACROS-1)Substitutable Functions*) should be written as one, not as a macro, for several reasons: substitutable functions are easier to write and to read; they can be passed as functional arguments (for example, you can pass them to 2mapcar*); and there are some subtleties that can occur in macro definitions that need not be worried about in substitutable functions. A macro can be a substitutable function only if it has exactly the semantics of a function, rather than of a special form. The macros we will see in this section are not semantically like functions; they must be written as macros. ;;; -*- Mode:gate; Fonts:(HL12 HL12I HL12B CPTFONTB HL12BI HL12B HL12I ) -*- =Node: Name Conflicts =Text: 3NAME CONFLICTS* One of the most common errors in writing macros is best illustrated by example. Suppose we wanted to write 2dolist* (see 4(FLOWCTL-2)Iteration*) as a macro that expanded into a 2do* (see 4(FLOWCTL-2)Iteration*). The first step, as always, is to figure out what the expansion should look like. Let's pick a representative example form, and figure out what its expansion should be. Here is a typical 2dolist* form. 3(dolist (element (append a b))* 3 (push element *big-list*)* 3 (foo element 3))* We want to create a 2do* form that does the thing that the above 2dolist* form says to do. That is the basic goal of the macro: it must expand into code that does the same thing that the original code says to do, but it should be in terms of existing Lisp constructs. The 2do* form might look like this: 3(do ((list (append a b) (cdr list))* 3 (element))* 3 ((null list))* 3 (setq element (car list))* 3 (push element *big-list*)* 3 (foo element 3))* Now we could start writing the macro that would generate this code, and in general convert any 2dolist* into a 2do*, in an analogous way. However, there is a problem with the above scheme for expanding the 2dolist*. The above example's expansion works fine. But what if the input form had been the following: 3(dolist (list (append a b))* 3 (push list *big-list*)* 3 (foo list 3))* This is just like the form we saw above, except that the programmer happened to decide to name the looping variable 2list* rather than 2element*. The corresponding expansion would be: 3(do ((list (append a b) (cdr list))* 3 (list))* 3 ((null list))* 3 (setq list (car list))* 3 (push list *big-list*)* 3 (foo list 3))* This doesn't work at all! In fact, this is not even a valid program, since it contains a 2do* that uses the same variable in two different iteration clauses. Here's another example that causes trouble: 3(let ((list nil))* 3 (dolist (element (append a b))* 3 (push element list)* 3 (foo list 3)))* If you work out the expansion of this form, you will see that there are two variables named 2list*, and that the programmer meant to refer to the outer one but the generated code for the 2push* actually uses the inner one. The problem here is an accidental name conflict. This can happen in any macro that has to create a new variable. If that variable ever appears in a context in which user code might access it, then you have to worry that it might conflict with some other name that the user is using for his own program. One way to avoid this problem is to choose a name that is very unlikely to be picked by the user, simply by choosing an unusual name, in a package which only you will write code in. This will probably work, but it is inelegant since there is no guarantee that the user won't just happen to choose the same name. The way to avoid the name conflict reliably is to use an uninterned symbol as the variable in the generated code. The function 2gensym* (see 4(SYMBOLS-1)Creating Symbols*) is useful for creating such symbols. Here is the expansion of the original form, using an uninterned symbol created by 2gensym*. 3(do ((#:g0005 (append a b) (cdr #:g0005))* 3 (element))* 3 ((null #:g0005))* 3 (setq element (car #:g0005))* 3 (push element *big-list*)* 3 (foo element 3))* This is the right kind of thing to expand into. (This is how the expression would print; this text would not read in properly because a new uninterned symbol would be created by each use of 3#:*.) Now that we understand how the expansion works, we are ready to actually write the macro. Here it is: 3(defmacro dolist ((var form) . body)* 3 (let ((dummy (gensym)))* 3 `(do ((,dummy ,form (cdr ,dummy))* 3 (,var))* 3 ((null ,dummy))* 3 (setq ,var (car ,dummy))* 3 . ,body)))* Many system macros do not use 2gensym* for the internal variables in their expansions. Instead they use symbols whose print names begin and end with a dot. This provides meaningful names for these variables when looking at the generated code and when looking at the state of a computation in the error-handler. These symbols are in the 2si* package; as a result, a name conflict is possible only in code which uses variables in the 2si* package. This would not normally happen in user code, which resides in other packages. ;;; -*- Mode:gate; Fonts:(HL12 HL12I HL12B CPTFONTB HL12BI HL12B HL12I ) -*- =Node: Block-Name Conflicts =Text: 3BLOCK-NAME CONFLICTS* A related problem occurs when you write a macro that expands into a 2prog* or 2do* (or anything equivalent) behind the user's back (unlike 2dolist*, which is documented to be like 2do*). Consider the 2error-restart* special form (see 4(DEBUGGING-3)Nonlocal Proceed Types*). Suppose we wanted to implement it as a macro that expands into a 2do-forever*, which becomes a 2prog*. Then the following (contrived) Lisp program would not behave correctly: 3(dolist (a list)* 3 (error-restart ((sys:abort error) "Return from FOO.")* 3 (cond ((> a 10)* 3 (return 5))* 3 ((> a 4)* 3 (ferror 'lose "You lose.")))))* The problem is that the 2return* would return from the 2error-restart* instead of the 2prog*. There are two possible ways to avoid this. The best is to make the expanded code use only explicit 2block*'s with obscure or gensymmed block names, and never a 2prog* or 2do*. The other is to give any 2prog* or 2do* the name 2t*. 2t* as a 2prog* name is special; it causes the 2prog* to generate only a 2block* named 2t*, omitting the usual 2block* named 2nil* which is normally generated as well. Because only 2block*s named 2nil* affect 2return*, the problem is avoided. When 2error-restart*'s expansion is supposed to return from the 2prog* named 2t*, it uses 2return-from* 2t*. Macros like 2dolist* specifically should expand into an ordinary 2do*, because the user expects to be able to exit them with 2return*. ;;; -*- Mode:gate; Fonts:(HL12 HL12I HL12B CPTFONTB HL12BI HL12B HL12I ) -*- =Node: Macros Expanding into Many Forms =Text: 3MACROS EXPANDING INTO MANY FORMS* Sometimes a macro wants to do several different things when its expansion is evaluated. Another way to say this is that sometimes a macro wants to expand into several things, all of which should happen sequentially at run time (not macro-expand time). For example, suppose you wanted to implement 2defconst* (see 4(EVAL-1)Defining Global Variables*) as a macro. 2defconst* must do two things, declare the variable to be special and set the variable to its initial value. (Here we implement a simplified 2defconst* that does only these two things, and doesn't have any options.) What should a 2defconst* form expand into? Well, what we would like is for an appearance of 3(defconst a (+ 4 b))* in a file to be the same thing as the appearance of the following two forms: 3(proclaim '(special a))* 3(setq a (+ 4 b))* However, because of the way that macros work, they only expand into one form, not two. So we need to have a 2defconst* form expand into one form that is just like having two forms in the file. There is such a form. It looks like this: 3(progn (proclaim '(special a))* 3 (setq a (+ 4 b)))* In interpreted Lisp, it is easy to see what happens here. This is a 2progn* special form, and so all its subforms are evaluated, in turn. The 2proclaim* form and the 2setq* form are evaluated. The compiler recognizes 2progn* specially and treats each argument of the 2progn* form as if it had been encountered at top level. Here is the macro definition: 3(defmacro defconst (variable init-form)* 3 `(progn (proclaim '(special ,variable))* 3 (setq ,variable ,init-form)))* Here is another example of a form that wants to expand into several things. We implement a special form called 2define-command*, which is intended to be used in order to define commands in some interactive user subsystem. For each command, there are two things provided by the 2define-command* form: a function that executes the command, and a character that should invoke the function in this subsystem. Suppose that in this subsystem, commands are always functions of no arguments, and characters are used to index a vector called 2dispatch-table* to find the function to use. A typical call to 2define-command* would look like: 3(define-command move-to-top #\meta-<* 3 (do () ((at-the-top-p))* 3 (move-up-one)))* Expanding into: 3(progn (setf (aref dispatch-table #\meta-<)* 3 'move-to-top)* 3 (push 'move-to-top *command-name-list*)* 3 (defun move-to-top ()* 3 (do ()* 3 ((at-the-top-p))* 3 (move-up-one)))* 3 )* The 2define-command* expands into three forms. The first one sets up the specified character to invoke this command. The second one puts the command name onto the list of all command names. The third one is the 2defun* that actually defines the function itself. Note that the 2setf* and 2push* happen at load-time (when the file is loaded); the function, of course, also gets defined at load time. (See the description of 2eval-when* (4(COMPILER-1)Input to the Compiler*) for more discussion of the differences between compile time, load time, and eval time.) This technique makes Lisp a powerful language in which to implement your own language. When you write a large system in Lisp, frequently you can make things much more convenient and clear by using macros to extend Lisp into a customized language for your application. In the above example, we have created a little language extension: a new special form that defines commands for our system. It lets the writer of the system attach the code for a command character to the character itself. Macro expansion allows the function definitions and the command dispatch table to be made from the same source code. ;;; -*- Mode:gate; Fonts:(HL12 HL12I HL12B CPTFONTB HL12BI HL12B HL12I ) -*- =Node: Macros that Surround Code =Text: 3MACROS THAT SURROUND CODE* There is a particular kind of macro that is very useful for many applications. This is a macro that you place ``around'' some Lisp code, in order to make the evaluation of that code happen in a modified context. For a very simple example, we could define a macro called 2with-output-in-base*, that executes the forms within its body with any output of numbers that is done defaulting to a specified base. 3(defmacro with-output-in-base ((base-form) &body body)* 3 `(let ((*print-base* ,base-form))* 3 . ,body))* A typical use of this macro might look like: 3(with-output-in-base (*default-base*)* 3 (print x) (print y))* which would expand into 3(let ((*print-base* *default-base*))* 3 (print x) (print y))* This example is too trivial to be very useful; it is intended to demonstrate some stylistic issues. There are standard Zetalisp constructs that are similar to this macro; see 2with-open-file* (4(FILEACCESS-1)Opening and Closing File Streams*) and 2with-input-from-string* (4(IOSYSTEM-2)String I/O Streams*), for example. The really interesting thing, of course, is that you can define your own such constructs for your applications. One very powerful application of this technique was used in a system that manipulates and solves the Rubik's cube puzzle. The system heavily uses a construct called 2with-front-and-top*, whose meaning is ``evaluate this code in a context in which this specified face of the cube is considered the front face, and this other specified face is considered the top face''. The first thing to keep in mind when you write this sort of macro is that you can make your macro much clearer to people who might read your program if you conform to a set of loose standards of syntactic style. By convention, the names of such constructs start with ``2with-*''. This seems to be a clear way of expressing the concept that we are setting up a context; the meaning of the construct is ``do this stuff 1with* the following things true''. Another convention is that any ``parameters'' to the construct should appear in a list that is the first subform of the construct, and that the rest of the elements should make up a body of forms that are evaluated sequentially with the last one returned. All of the examples cited above work this way. In our 2with-output-in-base* example, there was one parameter (the base), which appears as the first (and only) element of a list that is the first subform of the construct. The extra level of parentheses in the printed representation serves to separate the ``parameter'' forms from the ``body'' forms so that it is textually apparent which is which; it also provides a convenient way to provide default parameters (a good example is the 2with-input-from-string* construct (4(IOSYSTEM-2)String I/O Streams*), which takes two required and two optional parameters). Another convention/technique is to use the 2&body* keyword in the 2defmacro* to tell the editor how to indent the elements of the body (see 4(MACROS-1)Defmacro*). The other thing to keep in mind is that control can leave the construct either by the last form's returning, or by a non-local exit (2go*, 2return* or 2throw*). You should write the definition in such a way that everything is cleaned up appropriately no matter how control exits. In our 2with-output-in-base* example, there is no problem, because non-local exits undo lambda-bindings. However, in even slightly more complicated cases, an 2unwind-protect* form (see 4(FLOWCTL-2)Dynamic Non-Local Exits*) is needed: the macro must expand into an 2unwind-protect* that surrounds the body, with ``cleanup'' forms that undo the context-setting-up that the macro did. For example, 2using-resource* (see 4(MANLISTSTR-3)Allocating Resource Objects*) expands 3(using-resource (window menu-resource) 1body*...)* into 3(let ((window nil))* 3 (unwind-protect* 3 (progn (setq window* 3 (allocate-resource 'menu-resource))* 3 1body*...)* 3 (and window* 3 (deallocate-resource 'menu-resource window))))* This way the allocated resource item is deallocated whenever control leaves the 2using-resource* special form. ;;; -*- Mode:gate; Fonts:(HL12 HL12I HL12B CPTFONTB HL12BI HL12B HL12I ) -*- =Node: Multiple and Out-of-Order Evaluation =Text: 3MULTIPLE AND OUT-OF-ORDER EVALUATION* In any macro, you should always pay attention to the problem of multiple or out-of-order evaluation of user subforms. Here is an example of a macro with such a problem. This macro defines a special form with two subforms. The first is a reference, and the second is a form. The special form is defined to create a cons whose car and cdr are both the value of the second subform, and then to set the reference to be that cons. Here is a possible definition: 3(defmacro test (reference form)* 3 `(setf ,reference (cons ,form ,form)))* Simple cases work all right: 3(test foo 3) ==>* 3 (setf foo (cons 3 3))* But a more complex example, in which the subform has side effects, can produce surprising results: 3(test foo (setq x (1+ x))) ==>* 3 (setf foo (cons (setq x (1+ x))* 3 (setq x (1+ x))))* The resulting code evaluates the 2setq* form twice, and so 2x* is increased by two instead of by one. A better definition of 2test* that avoids this problem is: 3(defmacro test (reference form)* 3 (let ((value (gensym)))* 3 `(let ((,value ,form))* 3 (setf ,reference (cons ,value ,value)))))* With this definition, the expansion works as follows: 3(test foo (setq x (1+ x))) ==>* 3 (let ((#:g0005 (setq x (1+ x))))* 3 (setf foo (cons #:g0005 #:g0005)))* Once again, the expansion would print this way, but this text would not read in as a valid expression due to the inevitable problems of 3#:*. In general, when you define a new construct which contains one or more argument forms, you must be careful that the expansion evaluates the argument forms the proper number of times and in the proper order. There's nothing fundamentally wrong with multiple or out-of-order evalation if that is really what you want and if it is what you document your special form to do. But if this happens unexpectedly, it can make invocations fail to work as they appear they should. 2once-only* is a macro that can be used to avoid multiple evaluation. It is most easily explained by example. You would write 2test* using 2once-only* as follows: 3(defmacro test (reference form)* 3 (once-only (form)* 3 `(setf ,reference (cons ,form ,form))))* This defines 2test* in such a way that the 2form* is only evaluated once, and references to 2form* inside the macro body refer to that value. 2once-only* automatically introduces a lambda-binding of a generated symbol to hold the value of the form. Actually, it is more clever than that; it avoids introducing the lambda-binding for forms whose evaluation is trivial and may be repeated without harm or cost, such as numbers, symbols, and quoted structure. This is just an optimization that helps produce more efficient code. The 2once-only* macro makes it easier to follow the principle, but it does not completely or automatically solve the problems of multiple and out-of-order evaluation. It is just a tool that can solve some of the problems some of the time; it is not a panacea. The following description attempts to explain what 2once-only* does, but it is a lot easier to use 2once-only* by imitating the example above than by trying to understand 2once-only*'s rather tricky definition. 3once-only* 1var-list* 1body...* 1Macro* 1var-list* is a list of variables. The 1body* is a Lisp program that presumably uses the values of those variables. When the form resulting from the expansion of the 2once-only* is evaluated, the first thing it does is to inspect the values of each of the variables in 1var-list*; these values are assumed to be Lisp forms. For each of the variables, it binds that variable either to its current value, if the current value is a trivial form, or to a generated symbol. Next, 2once-only* evaluates the 1body* in this new binding environment and, when they have been evaluated, it undoes the bindings. The result of the evaluation of the last form in 1body* is presumed to be a Lisp form, typically the expansion of a macro. If all of the variables have been bound to trivial forms, then 1once-only* just returns that result. Otherwise, 2once-only* returns the result wrapped in a lambda-combination that binds the generated symbols to the result of evaluating the respective non-trivial forms. The effect is that the program produced by evaluating the 2once-only* form is coded in such a way that, each of the forms which was the value of one of the variables in 1var-list* is evaluated only once, unless the form is such as to have no side effects. At the same time, no unnecessary temporary variables appear in the generated code, but the body of the 2once-only* is not cluttered up with extraneous code to decide whether temporary variables are needed. ;;; -*- Mode:gate; Fonts:(HL12 HL12I HL12B CPTFONTB HL12BI HL12B HL12I ) -*- =Node: Nesting Macros =Text: 3NESTING MACROS* A useful technique for building language extensions is to define programming constructs that employ two special forms, one of which is used inside the body of the other. Here is a simple example. There are two special forms. The outer one is called 2with-collection*, and the inner one is called 2collect*. 2collect* takes one subform, which it evaluates; 2with-collection* just has a body, whose forms it evaluates sequentially. 2with-collection* returns a list of all of the values that were given to 2collect* during the evaluation of the 2with-collection*'s body. For example, 3(with-collection (dotimes (i 5) (collect i)))* 3 => (1 2 3 4 5)* Remembering the first piece of advice we gave about macros, the next thing to do is to figure out what the expansion looks like. Here is how the above example could expand: 3(let ((#:g0005 nil))* 3 (dotimes (i 5)* 3 (push i #:g0005))* 3 (nreverse #:g0005))* Now, how do we write the definition of the macros? Well, 2with-collection* is pretty easy: 3(defmacro with-collection (&body body)* 3 (let ((var (gensym)))* 3 `(let ((,var nil))* 3 ,@body* 3 (nreverse ,var))))* The hard part is writing 2collect*. Let's try it: 3(defmacro collect (argument) `(push ,argument ,var))* Note that something unusual is going on here: 2collect* is using the variable 2var* freely. It is depending on the binding that takes place in the body of 2with-collection* in order to get access to the value of 2var*. Unfortunately, that binding took place when 2with-collection* got expanded; 2with-collection*'s expander function bound 2var*, and the binding of 2var* was unmade when the expander function was done. By the time the 2collect* form gets expanded, the binding is long gone. The macro definitions above do not work. Somehow the expander function of 2with-collection* has to communicate with the expander function of 2collect* to pass over the generated symbol. The only way for 2with-collection* to convey information to the expander function of 2collect* is for it to expand into something that passes that information. One way to write these macros is using 2macrolet*: 3(defmacro with-collection (&body body)* 3 (let ((var (gensym)))* 3 `(macrolet ((collect (argument)* 3 `(push ,argument ,',var)))* 3(let ((,var nil))* 3 ,@body* 3 (nreverse ,var)))))* Here 2with-collection* expands into code which defines 2collect* specially to know about which variable to collect into. 2,',* causes 2var*'s value to be substituted when the outer backquote, the one around the 2macrolet*, is executed. 2argument*, however, is substituted in when the inner backquote is executed, which happens when 2collect* is expanded. This technique has the interesting consequence that 2collect* is defined only within the body of a 2with-collection*. It would simply not be recognized elsewhere; or it could have another definition, for some other purpose, globally. This has both advantages and disadvantages. Another technique is to communicate through local declarations. The code generated by 2with-collection* can contain a 2local-declare*. The expansion of 2collect* can examine the declararion with 2getdecl* to decide what to do. Here is the code: 3(defmacro with-collection (&body body)* 3 (let ((var (gensym)))* 3 `(let ((,var nil))* 3(local-declare ((collection-var nil ,var))* 3 ,@body* 3 (nreverse ,var)))))* 3(defmacro collect (argument)* 3 (let ((var ,(getdecl nil 'collection-var)))* 3 (unless var * 3 (ferror nil "COLLECT not within a WITH-COLLECTION"))* 3 `(push ,argument var)))* Another way, used before 2getdecl* existed, was with 2compiler-let* (see 4(COMPILER-2)Maclisp Compatibility*). 2compiler-let* is identical to 2let* as far as the interpreter is concerned, so the macro continues to work in the interpreter with this change. When the compiler encounters a 2compiler-let*, however, it actually performs the bindings that the 2compiler-let* specifies and proceeds to compile the body of the 2compiler-let* with all of those bindings in effect. In other words, it acts as the interpreter would. Here's the right way to write these macros in this fashion: 3(defvar *collect-variable*)* 3(defmacro with-collection (&body body)* 3 (let ((var (gensym)))* 3 `(let ((,var nil))* 3 (compiler-let ((*collect-variable* ',var))* 3 . ,body)* 3 (nreverse ,var))))* 3(defmacro collect (argument)* 3 `(push ,argument ,*collect-variable*))* ;;; -*- Mode:gate; Fonts:(HL12 HL12I HL12B CPTFONTB HL12BI HL12B HL12I ) -*- =Node: Functions Used During Expansion =Text: 3FUNCTIONS USED DURING EXPANSION* The technique of defining functions to be used during macro expansion deserves explicit mention here. It may not occur to you, but a macro expander function is a Lisp program like any other Lisp program, and it can benefit in all the usual ways by being broken down into a collection of functions that do various parts of its work. Usually macro expander functions are pretty simple Lisp programs that take things apart and put them together slightly differently, but some macros are quite complex and do a lot of work. Several features of Zetalisp, including flavors, 2loop*, and 2defstruct*, are implemented using very complex macros, which, like any complex well-written Lisp program, are broken down into modular functions. You should keep this in mind if you ever invent an advanced language extension or ever find yourself writing a five-page expander function. A particular thing to note is that any functions used by macro-expander functions must be available at compile-time. You can make a function available at compile time by surrounding its defining form with an 2(eval-when (compile load eval) ...)*; see 4(COMPILER-1)Input to the Compiler* for more details. Doing this means that at compile time the definition of the function is interpreted, not compiled, and hence runs more slowly. Another approach is to separate macro definitions and the functions they call during expansion into a separate file, often called a ``defs'' (definitions) file. This file defines all the macros, and also all functions that the macros call. It can be separately compiled and loaded up before compiling the main part of the program, which uses the macros. The 1system* facility (see 4(SYSTEMS-0)Maintaining Large Systems*) helps keep these various files straight, compiling and loading things in the right order.