POP-11 OBJECT-MODULE COMPILATION
John Gibson Dec 1994
COPYRIGHT University of Sussex 1994. All Rights Reserved.This file is a reference manual for the Pop-11 object-module compilation and linking system. It describes in detail Popc, the object-module compiler for Pop-11 programs, and also discusses the associated utilities Poplink and Poplibr. It should be read in conjunction with HELP * POPC, which is a user-guide for the three programs (listing all their available options, etc).
Contents
Select headings to return to index
In conventional object-module (or 'batch') compilation, each source file of program code is compiled to a corresponding object file, which contains a translation of the program into host-machine instructions; the set of object files is then linked by the operating system (O/S) linker to produce an executable image file. This usually also results in the incorporation of one or library object modules, which are extracted automatically by the linker on demand.
With Popc the situation is similar, but with one major difference: in order to handle the linking of global data structures within a Pop program, a second file is produced for each source file compiled. These extra files (called w-files, from their .w extension) are in binary format, and describe the usage by their corresponding object files of such things as Pop identifiers, quoted words, and other data.
Thus a w-file together with its associated object file can be thought of as a composite 'Pop object file'. (Ideally, the two would actually be combined into a single file, but O/S considerations make this impossible.) Similarly, just as an O/S library archive can be built from a set of object files (e.g. with the ar utility in Unix, or library in VMS), so Poplibr can be used to create and maintain 'Pop object libraries' from sets of w- and object- files, where each library consists of a w-library file plus an ordinary O/S object library.
All the normal Poplog system libraries are compiled to corresponding w+object libraries; these reside in the directory $popobjlib.
Another respect in which Popc differs from conventional object compilers is that library archives are used not only at link-time, but also at compile-time, to extract declarations for library identifiers used by the files being compiled. This makes it unnecessary for programs to have to provide declarations for system identifiers. For each file compiled, the libraries so used are remembered on the output w-file, and automatically supplied to the linking process.
Linking of a program is performed by the Pop linker, Poplink. This takes as input the set of all w-files for the program, automatically extracts any Pop system library modules used, and performs the necessary global-structure linking. The linked global data structures are generated as some additional object files, which Poplink then passes, together with all object files/libraries corresponding to the w-files/libraries used, as input to the O/S linker.
The diagram below illustrates the compilation and linking process for a program consisting of two source files 'foo.p' and 'baz.p'. In this, the dotted lines represent the usage of w-libraries by Popc for extracting identifier declarations, and add.o represents the additional object files produced by Poplink:
Popc Poplink O/S linker
---------
--------- | foo.w | -----
| foo.p | --O--> |-------| |
--------- | | foo.o | ----+--------------------
| --------- | |
| --------- | |
--------- | | baz.w | ----| |
| baz.p | --O--> |-------| | |
--------- | | baz.o | ----+-------------------|
| --------- | |
| | --------- | ----------
| O---> | add.o | ----O---> | image |
| | --------- | ----------
| --------- | |
w-library --<- | lib.w | ----- |
(.wlb) | ... | |
|-------| |
o-library | lib.o | -------------------------
(.olb) | ... |
---------
As the description above implies, Poplink runs the O/S linker automatically. In addition (as with Unix C compilers), Popc runs Poplink automatically for you (unless you specify otherwise with the -c option). Thus Popc is used for both compilation and linking.
In compiling Pop-11 programs with Popc, you are strongly advised to follow Poplog system and library source code by using
compile_mode :pop11 +strict;
in all Pop-11 source files (see HELP * COMPILE_MODE).
First, this disallows the use of 'old-style' Pop-11 where the formal argument and result identifiers of procedures default to dynamically-local vars, and other procedure locals are declared with vars. It forces all procedure formals to be declared explicitly (usually as lvars).
For example, without +strict a procedure such as
define old_style(x, y, z);
vars u, v, w;
...
enddefine;
is allowed; all of its locals (x, y, z, u, v, w) are dynamically-local permanent variables, and it is thus the same as
vars x, y, z, u, v, w;
define old_style(x, y, z);
dlocal x, y, z, u, v, w;
...
enddefine;
Compared to a similar procedure using lvars, the above is (a) highly non-modular, (b) generates much less efficient code, and (c) in a batch-compiled environment will lead to much slower linking times (since every permanent identifier corresponds to a global external symbol).
Second, +strict establishes the default declaration for top-level defines as 'constant procedure' (note that without +strict, the form
define fred();
...
enddefine;
declares fred as an untyped variable). With Popc, it is important to have as many procedures as possible declared as procedure-type constants, both for efficiency reasons (a procedure identifier generates much more efficient code), and because of restrictions on execute-level cross-file references to variables (described below).
+strict also disallows the vars default for nested defines, forcing them to be declared explicitly. (Except for the occasional dlocal or lvars, all nested procedures should be lconstant.)
Finally, +strict causes all permanent identifiers to be automatically declared global.
As described by Creating Layered Systems in REF * SYSTEM, 'non-specialised' data structures such as pairs, lists, strings, full vectors, etc, can have a general writeable or nonwriteable default specified for them when creating a saved image with sys_lock_system. There is also a separate default for (user-created) closures.
The corresponding behaviour for Popc is controlled by the compile_mode :popc options wrdflt and wrclos (see HELP * COMPILE_MODE). Since the composite +strict mode includes both -wrdflt and -wrclos for Popc, an additional effect of using
compile_mode :pop11 +strict;
in programs is to force a nonwriteable default for all such structures, including closures. That is, Popc will compile execute-level instances of these structures into nonwriteable areas of memory.
For batch-compiled programs, this has similar advantages to those described in Creating Layered Systems: reduced garbage collection time, greater shareability of the executable image, etc. But it does mean you will need to apply writeable to any structures of this type that your program requires to update.
Alternatively, you can add another compile_mode statement such as
compile_mode :popc +wrdflt +wrclos;
after the one for +strict (note that the two statements cannot be directly combined, since they specify contradictory values for wrdflt and wrclos).
Popc compiles a file by calling the ordinary Pop-11 compiler, but after setting the system variable pop_pas_mode to the word "popc". This variable having a true value causes the Poplog Virtual Machine (VM) to divert VM code-planting and permanent identifier declarations to various hook procedures, for which Popc provides appropriate definitions.
This is 'Popc compile mode': all compilation in this mode results in procedures containing Popc's own representation of the code, which will be interpreted if necessary, i.e. when code is actually executed at compile-time.
In particular, Popc-compiled code traps all assignments to permanent identifiers, and stores them on a global list. This list is the basic target of Popc's operation, and is the only source for generation of output object code.
However, the 'real' identifiers in the underlying system are not actually assigned to. This is necessary because Popc can be used to compile source code that makes assignments to identifiers actually being used by the underlying Pop-11 system (in which Popc is running); if the real identifiers were assigned to, the operation of the underlying system would be changed or corrupted.
The point is that the purpose of a Popc-compiled assignment is to set the value of an identifier in the output object file (and hence in the linked executable image), not to set its value in the Pop-11 system in which Popc is running.
On the other hand, a Popc-compiled access to an identifier returns any Popc value assigned to that identifier (if it has one -- otherwise the current value of the real identifier is returned).
In summary, Popc maintains shadow values for permanent identifiers that have been assigned to. The Popc shadow value of an identifier is also accessed and updated by sys_current_val when Popc mode is in force.
Not all code compiled during Popc's operation will necessarily be compiled in Popc mode. When compilation of library files is invoked, either by autoloading or with uses-now, Popc mode is turned off. (Note that an ordinary uses does not compile libraries -- see Use of uses-now Instead of uses below.) Popc mode can also be explicitly turned off with the syntax construct normal_compile ... end_normal_compile.
The principal problem that arises with ordinary-mode compilation of libraries is that Popc runs in a Pop-11 system containing neither the X nor Ved parts of Poplog. Attempting to compile libraries that use these facilities will generally lead to multiple 'DECLARING VARIABLE ...' messages, or mishaps. It is thus essential to restrict loading of libraries to those situations where the code is actually required at compile-time.
On the one hand, if a library is explicitly specified with uses-now, Popc has no choice but to compile it (it is up to you to ensure that only suitable libraries are so specified).
On the other hand, Popc will not attempt to autoload a library identifier unless absolutely necessary. Moreover, in certain cases Popc limits the library directories that are searched for autoloadable files, as follows:
As in the normal interactive system, autoloading will happen only for an identifier that is strongly referenced but not declared (or only weakly declared). Unlike the normal system, however, autoloading will not generally happen if a declaration for the identifier can be extracted from a w-library. The exceptions to this are where
(a) the identifier is a syntax word or macro (and therefore needs to
be executed immediately);
(b) the value of the identifier is actually accessed (by
execute-level code).
In case (a), the assumption is made that (being syntax or macro) the library will not contain any 'hard' references to Ved or X facilities (the section Syntax, Macro & Other Compile-Time Executed Procedures below discusses this in more detail). Therefore the full set of popautolist directories are searched, as normal. The case where the identifier is completely undeclared has to be treated similarly (since it might be syntax or macro).
However, in case (b) (that is, where a non-syntax/macro declaration is available), it is in many circumstances good enough to simply allow the undef value of the identifier to be accessed, without attempting to autoload it. (The reason for this is explained more fully under Value Generation Phase below; basically, providing the undef value is not executed or otherwise used in some computation, it will be correctly interpreted during generation of output code.)
Hence in this case, only a restricted set of directories ($popautolib + $poplocalauto) are searched, instead of the full popautolist. While this prevents many potentially problematic (i.e. Ved and X) identifiers being autoloaded unnecessarily, it does means that uses-now will be needed for a small number of library identifiers which are not syntax/macro, but which nevertheless can be successfully used at execute-level with Popc. (The X procedures XptSpecifyResourceInfo and XptSpecifyPopValueTypes are examples of this.)
The definition
lconstant macro POPC_COMPILING = true;
is automatically added to the beginning of every source file. This enables a file to use
#_IF DEF POPC_COMPILING
...
#_ENDIF
etc, to test for being compiled with Popc (either in or out of Popc mode).
Occasionally, some syntax and macro code may need to behave differently when being run in Popc compile mode. In these cases the test
if pop_pas_mode == "popc" then
...
endif
is appropriate.
After compiling a file, Popc works through the list of assignments to permanent identifiers, and generates object (assembler) code for each value assigned. The output assembler file is then translated to an object file with the operating-system assembler.
Permanent assignments are the only source for code generation. Assignments to lexical identifiers will be ignored unless they are in some way referenced by structure(s) assigned to permanent identifier(s).
In a Pop-11 program, certain data structures must be unique across the system as a whole: for example, the dictionary word "abcd" must always be the same object, wherever referred to. Generation of such structures cannot be done by individual object modules, since there is no way of deciding that one module rather than another should do the job.
Thus (as described in Overview above), a linking process is required, to generate these structures centrally: this is the purpose of w-files and Poplink. The structures which are always generated by Poplink are:
(Poplink also generates other structures when these are the values of incremental identifiers. See Incremental Identifiers below.)
A constant identifier can only be assigned a value in one file, and a (strongly-referenced) constant must always have a value. Poplink treats a violation of either of these conditions as an error.
On the other hand, Poplink allows some leeway in assigning values to vars. For example, an assignment for a variable in an object file is allowed to override one in a library module; also, a variable may be unassigned (and default to an undef value).
These (and other) considerations mean that, while the Popc/Poplink system supports inter-file values for constants, it does not support inter-file values for vars. In practice this means the following:
Except for the Poplink-generated structures listed above, the only compile-time cross-file references allowed are to the values of constants. Thus the values of constants are compile-time accessible between files, but the values of variables are not.
Note that this restriction does not concern run-time accessing of variable values; such accesses (i.e. in procedure code), will always be to variable identifiers (Poplink-generated structures). It concerns only Popc's generation of object code from the values in the permanent assignment list -- that is, 'static' references between files resulting from compile-time executed code.
To use Popc successfully, you need to know how the above restriction operates in practice, and how it relates to errors of the form
NON-GENERATABLE VALUE (optional-qualifying-message)
INVOLVING: <object>
which may occur during Popc's value generation phase.
When presented with a ("full") location containing a value to generate, Popc does one of three things:
(a) If the value is a simple object (integer or decimal), it
generates the appropriate bit pattern in the location;
(b) If the value is a Poplink-generated structure, OR if there is a
constant identifier in the dictionary whose idval is the object,
it assumes that the structure is generated elsewhere, and merely
outputs a reference to it (that is, outputs its corresponding
external symbol);
(c) Otherwise, it actually generates the structure -- that is,
allocates memory for it in the current output file (and
recursively generates its sub-structures, etc).
Note that (b) includes the case where the object is a named undef record contained in some constant identifier in the dictionary. Such a value may have resulted by accessing the value of a constant which has been declared but not assigned to: this is the way in which constants from other files are usually dealt with (hence Popc does not need to know the actual value of the constant).
For example, if baz is a constant procedure defined in another file, then the execute-level code
constant procedure baz;
define foo = baz(% 1, 2, 3 %) enddefine;
will result in the Popc shadow-value for foo being a closure whose pdpart is <undef baz>; this will be mapped correctly to the external symbol for the constant baz.
On the other hand (because of the restriction on variables described above), Popc will refuse to generate a named undef record that is the idval of any variable in the dictionary. Thus if baz were instead declared
vars procedure baz;
the error
NON-GENERATABLE VALUE (undef value of perm variable)
INVOLVING: <undef baz>
would result.
For other structures (and where no suitable constant can be found), case (c) comes into play, i.e. Popc actually generates the structure. The following kinds of structure cannot be generated, however:
(1) Proper procedures that have not been compiled in Popc mode;
(2) Currently, instances of shadowclass structures (but note that
shadowkeys are OK);
(3) Currently, XptDescriptor structures;
(4) System devices (user devices are OK);
(5) Processes.
For example, (1) will give an error like
NON-GENERATABLE VALUE (procedure not compiled in popc mode)
INVOLVING: <procedure>
In general, an error of the form
NON-GENERATABLE VALUE (optional-qualifying-message)
INVOLVING: <object>
means that the object is not a generatable structure, and moreover, is not the value of any constant in the dictionary. Note that a value does not have to be a generatable structure, providing it can be mapped to a constant; this enables a structure such as
vars list = [% hd, tl, lmember %];
to be generated, even though the accesses to hd, tl and lmember will result in the list containing the actual Pop-11 system procedures (which, of course, are not Popc-mode compiled). Compare, however
vars list = [% hd, tl, member %];
This will give
NON-GENERATABLE VALUE (procedure not compiled in popc mode)
INVOLVING: <procedure member>
because the procedure member is not a constant (unfortunately).
Another restriction not covered by the last section is the following:
You cannot assign a constant from another file to be the direct value of a constant in the current file.
For example, if baz is defined in a file other than the current, the following is invalid:
constant procedure baz;
constant procedure foo = baz;
and will result in the error
INVALID INITIALISATION FOR PERM CONSTANT foo
This only concerns setting foo directly equal to baz; there is no problem when baz is some sub-element of a structure being assigned to foo (that is, when the reference to baz is in a memory location). There is also no problem when setting a variable equal to a constant, e.g.
vars procedure foo = baz;
is correct. (Once again, the reference to baz will be in a memory location.)
You can, however, define a constant to have any Poplink-generated structure as its value. E.g.
constant
XXXX = true,
a_word = "abcd",
baz_id = ident baz,
a_list = [];
are all valid.
(The above restriction arises because of the inability of O/S object code to set the value of one external symbol equal to another, where the other is not defined in the current file. Note that it does not apply to lconstants, because these have scope only in the current file and hence do not need to define external symbols.)
Many Pop-11 programs that compile and work correctly in the normal interactive Poplog system will require changes to compile under Popc. However, these changes can always be made in such a way that the end result will work with either system.
The main respects in which programs will require changing can be summarised as:
Permanent Identifier Declarations
All permanent identifiers must be declared before being
referenced. All declarations must be consistent, both within and
across files.
Use of uses-now instead of uses
uses for libraries that are required at compile-time must employ
uses-now instead.
Compile-Time Executed Code
There are certain restrictions on the code that be executed at
compile-time.
Keeping Compile-Time and Run-Time Code Separate.
Permanent syntax words and macros should not be mixed with
run-time code in the same file.
These areas are dealt with in the following sections.
In a file compiled with Popc, all user permanent identifiers must be declared before being referenced. An undeclared identifier will result in the error
UNDECLARED IDENTIFIER xxxx
Moreover, any re-declaration of an identifier must be identical to its existing declaration (in terms of constant/vars and identprops), otherwise the error
REDECLARING STATUS OF PERM IDENTIFIER xxxx
results.
If an identifier is referenced with weakref in a file, it must be referenced everywhere in that file with weakref, i.e. all references in a single file must be either weak or strong. Referring to an identifier both weakly and strongly in the same file will produce the error
MIXED WEAK/STRONG REFERENCES TO IDENTIFIER xxxx
(Note, however, that Popc does not distinguish between strong and weak declarations: you may declare an identifier as either -- although it may generally be clearer to use weak for a weakly-referenced identifiers, etc.)
Only identifiers actually assigned to at execute-level or referenced in procedure code are recorded on the output w-file; identifiers which are declared but not used anywhere are simply ignored.
Both Poplink and Poplibr also enforce the consistency of identifier declarations across files: all permanent identifiers must have identical declarations in all files. You cannot declare an identifier as constant in one file, and vars in another, or as procedure-type in one file, but untyped in another, etc. (An identifier may, of course, be referenced weakly in one file and strongly in another.)
You do not need to supply declarations for Poplog system identifiers. As described in Overview above, Popc automatically extracts declarations for these from the system w-libraries, that is, .wlb files in the directory $popobjlib.
However, which of these files are actually inspected is controlled by the current value of * popsyslist (which contains * popuseslist as a sub-list). The names of files in $popobjlib just replicate the directory pathnames in which the corresponding source files reside, e.g.
$usepop/pop/src/ --> src.wlb
$usepop/pop/lib/auto/ --> libauto.wlb
$usepop/pop/x/ved/src/ --> xvedsrc.wlb
Using this name mapping, Popc will search the .wlb files corresponding to the directories in popsyslist. Thus for example, a program that normally requires
uses popxlib;
to extend popautolist and popuseslist with the X library directories, will require the same with Popc. (But note that for this to function correctly, you must have
uses-now popxlib;
instead. See Use of uses-now Instead of uses below.)
If you have your own w-libraries, Popc can be told to use these for declaration-extraction with the -wlib option. (You can also prevent the use of popsyslist in this respect with the -nosys option.) See HELP * POPC.
The problem of declarations will mainly affect multi-file programs, which when loaded in the ordinary Poplog system usually rely on previously-compiled files supplying declarations for later ones. This technique does not work with Popc since each file is compiled individually. (This applies even when compiling a number of source files in a single run, e.g. in
popc -c foo.p baz.p ...
all declarations for baz.p and subsequent files must be supplied as if each were being compiled alone.)
(However, for special uses Popc provides the -g option, which enables a group of files to be compiled to a single object module. With this, the source files are effectively just concatenated, and so identifier declarations in earlier files are valid for references in later ones.)
If you use defclass to define a (permanent) record or vector class in one file, and wish to use one or more of its associated identifiers in other files, then you must provide declarations for those identifiers in the other files.
This can be done using ordinary declarations, e.g. if file 'foo.p' defines a class
defclass fred {fred_f1 :full, fred_f2 :full};
then file 'baz.p' can make use of the associated procedures by declaring them appropriately:
constant fred_key;
constant procedure (consfred, destfred, fred_f1, fred_f2);
(Note that +strict is assumed here, i.e. that defclass will default to constant, with type procedure for all the identifiers except fred_key).
There is, however, one situation in which the above is not sufficient: namely, where a file (other than that defining the class) uses the constructor procedure to make an instance of the class at compile-time. For instance, baz.p might contain something like
lconstant a_fred = consfred(1,2);
In this case, merely having consfred declared is clearly not sufficient; the proper constructor procedure must actually be available.
In order to achieve this, note that you cannot simply replicate the ordinary definition of the fred class in baz.p, because that would cause all the associated identifiers to be multiply-defined at link-time (only one file may actually define the class). Instead, use defclass-declare:
defclass-declare fred {fred_f1 :full, fred_f2 :full};
This is identical to an ordinary defclass, except that all the identifiers -- while declared in Popc mode -- have their values assigned in non-Popc mode. Thus as far as Popc is concerned, they are declared but not initialised; instead, the real identifiers in the underlying system are assigned to (making the constructor procedure available for execution).
(Note also another point concerning this 'instance-constructing' case. As with any other structure, the instance will be generated by Popc as described in Value Generation Phase above. You might think the key field of the structure would somehow be special, but in fact, it follows exactly the same rules as for any other field: Its value is fred_key -- a structure defined in another file -- and so fred_key must be a constant. Thus you cannot construct compile-time instances of a class from another file for which the key identifier is a variable.)
While defclass-declare is essential in the instance-constructing case, whether you use it as a substitute for simple declarations in other cases is a matter of taste.
Popc provides two special declarations for permanent identifiers which you may need to use (usually with multi-file programs): declare_incremental and declare_updater. Both of these declarations are also available in ordinary Pop-11 (where they essentially do nothing).
The first (declare_incremental) is required where a program needs to build up some global structure (such as a list) across several files -- that is, in the normal interactive system the structure is initialised to an empty state in (say) the first file loaded, after which later files add increments to it.
This technique is supported with Popc, but only for lists, procedures, and properties. See Incremental Identifiers below for full details.
The second declaration (declare_updater) may be required when a constant procedure from one file is used (for example) as the pdpart of a closure in another. E.g.
constant procedure baz; ;;; defined in some other file
define foo = baz(% 1, 2, 3 %) enddefine;
When generating code for the closure foo, Popc has no a priori way of knowing whether the pdpart procedure baz has an updater or not, and thus whether foo should have one: the default is to assume not. Hence if baz does have an updater (and you want foo to have one), then
declare_updater baz;
is needed (following the ordinary declaration of baz).
See * declare_updater below.
To make it easy to supply identifier declarations that are not required for ordinary interactive compilation, Popc provides the -include option. This automatically #_INCLUDEs a file of declarations in every file compiled, e.g.
popc -c -include popc_declare.ph foo.p baz.p ...
will #_INCLUDE 'declare.ph' at the beginning of every source file.
This and other initial arguments to Popc can be conveniently specified for all source files in a given directory by means of a 'popc.args' file in that directory (see HELP * POPC).
In Popc compile mode, an ordinary uses statement such as
uses foo, baz, ... ;
does not actually cause any files to be loaded. Rather, it merely records that the identifiers foo, baz, ... have been 'used'. At link time, this will force the extraction of the library modules defining the identifiers even if they have not actually been referred to anywhere in the current (or any other) file.
This behaviour means, however, that an ordinary uses is inappropriate for a library which is actually required at compile-time -- that is, one which (for example) defines syntax words or macros, or which extends popuseslist (such as LIB * POPXLIB). In these cases, uses-now must be employed:
uses-now popxlib, Xm, ... ;
This is effectively the same as surrounding an ordinary uses with normal_compile ... end_normal_compile, i.e. Popc compile mode is temporarily turned off, and the files are compiled as normal. (In normal interactive compilation, uses-now is the same as uses.)
Almost all run-time executed code inside procedures will work unchanged with Popc, providing permanent identifiers are declared correctly. The majority of problems in converting files for Popc will occur with code executed at compile-time; that is, either execute-level code (code outside of procedures, lconstant expressions, code inside #_< ... >_# etc), or syntax and macro procedure code which gets executed when the syntax word or macro is run during compilation.
(Note that while execute-level code will always be compiled in Popc mode, syntax/macro code may be compiled either in Popc mode, or in ordinary mode when from a library.)
The principal problems you will encounter will result from the errors already described in Value Generation Phase above. In summary, they are:
# You cannot reference the values of permanent variables assigned in
other files (including system ones).
Or rather, you can reference them, but the value generation phase will not be able to generate their values. Probably the most frequent problem in this respect is attempting to create closures of variable procedures, e.g.
define is_a_member =
member(% [a list of words] %)
enddefine;
This will fail because the procedure member is not a constant (using lmember_= instead would be valid, because that is). The solution in such cases is to define a proper procedure, i.e.
define is_a_member(item);
lvars item;
member(item, [a list of words])
enddefine;
# You cannot set one permanent constant directly equal to another
(unless the latter's value has already been assigned in the current
file).
For example,
constant procedure foo = hd;
is invalid. In this case, a dummy closure could be created:
define foo = hd(%%) enddefine;
# Popc runs in a Pop-11 that contains neither the Ved nor X parts of
the system.
Thus (with the exception of associated syntax procedures and macros) you cannot execute any Ved or X procedures at compile time.
(Note that there are (at least) two X library procedures, XptSpecifyResourceInfo and XptSpecifyPopValueTypes, which are not syntax or macro, but which can nevertheless be used at compile-time since they do not make any 'hard' references to X facilities (that is, they only refer to them 'by name'). As described in Use of Libraries above, you need uses-now to load these procedures.)
# Certain structures are not generatable.
Currently, this involves instances of shadowclass structures (shadowkeys are OK however), and XptDescriptor structures. To surmount such problems, generate the structures at run-time, either with a runtime action or in some other way.
For example, LIB * XptArgList requires a constant zero-length XptArgList structure (a shadowclass instance). This cannot be defined as
lconstant empty_arglist = consXptArgList(0);
since the structure is not generatable; instead it must use a top-level lvar initialised to false, which then caches the structure the first time it is used:
lvars empty_arglist = false;
define XptArgList(list);
lvars list, size;
if list == [] then
empty_arglist or
(consXptArgList(0) ->> empty_arglist), 0;
...
A similar effect could be achieved with a runtime action:
lvars empty_arglist;
define :runtime_action;
consXptArgList(0) -> empty_arglist;
enddefine;
define XptArgList(list);
lvars list, size;
if list == [] then
empty_arglist, 0
...
In this section, compile-time procedure (CTP) means a syntax or macro procedure, or any other procedure that could be executed in a compile-time context.
The points in the last section apply to the code planted or generated by all CTPs, whether lexical or permanent identifiers.
CTPs of any kind should not make any direct references to Ved or X run-time procedures, or any other facilities that are not present in a basic Pop-11 system (because such code will not load inside Popc). All such references should be 'by name', that is:
In fact, references in CTPs to any run-time facilities should generally use the above -- for example, a particular library run-time procedure may be loadable in Popc, but referring to it by name rather than directly will obviate the need to load the procedure at compile-time.
There are additional factors to bear in mind when writing permanent compile-time procedures (PCTPs).
If you are making a CTP a permanent identifier, you are doing so (presumably) because you want it to be usable in more than one file. Since with Popc, each file is compiled individually, you cannot rely on the PCTP having been loaded from some previous file (as often happens in ordinary Pop-11). Thus
(a) the file containing the PCTP definition must either be
autoloadable, or be loaded explicitly with uses-now by every
file that uses it;
(b) as described in Use of Libraries above, this will load the file
in non-Popc mode; hence the file defining the PCTP must be
capable of being loaded (in non-Popc mode) inside Popc.
As usual, (b) means that the file containing the PCTP definition must not make any direct references to Ved or X run-time facilities, etc. This applies both to the PCTP itself, and anything else that may be in the same file. If the PCTP needs to reference these facilities, the references must always be via quoted words or word-identifiers. (For examples of this, see LIB * vedset, * XptVal; these are two syntax constructs which plant calls to Ved and X procedures respectively, but always using sysCALL on procedure names, never sysCALLQ on the procedures directly.)
Use of quoted words/word-identifiers in this way is not a problem for syntax constructs, macros, etc, because such things usually expect to have to use the names of identifiers, and efficiency issues are not involved. (For instance, a sysCALL on the name of a constant will result in the same code as a sysCALLQ on its value.)
On the other hand, run-time code will generally want to make 'hard' references to identifiers (and not refer to names) for efficiency reasons. Hence including other, run-time, procedures in the same file as a PCTP may compromise the latter's ability to load inside Popc.
The above considerations lead to this general rule:
Do not mix permanent syntax and macro definitions (and other procedures that need to be executed at compile-time) with run-time code in the same file -- keep the two completely separate.
A special case of the above comes where a PCTP (such as a syntax word) plants code for run-time procedure(s) that are specific to itself. Because they are used only by a particular PCTP, such procedures are often defined as lconstants in the same file. Although violating the rule above, this in itself would not necessarily prevent the PCTP loading in Popc (it would depend on what the lconstant procedure(s) contain); however, the practice will not work anyway, for another reason:
As described in Value-Generation Errors above, proper procedures that have not been compiled in Popc mode cannot be generated in the output object file. But (when used in a file other than its own), the PCTP file will be compiled in non-Popc mode; thus all the procedures it contains will be non-generatable.
To avoid this problem, run-time procedures for which calls are planted by PCTPs must always be permanent identifiers. Popc will then merely make the planted calls reference the corresponding identifiers without attempting to generate the procedures they contain. (Note that even if Popc could generate lconstant procedures from the (non-Popc-mode compiled) PCTP file, this would be highly undesirable anyway, since then every file using the PCTP would have its own private copy of those procedures.)
Providing the run-time procedures are permanent identifiers, there is nothing to prevent them being in the same file as the PCTP, assuming they do not reference Ved or X facilities. However, this is always undesirable for yet another reason: being in the same object module, the run-time parts cannot be linked separately from the compile-time parts. (Not only will linking the run-time parts into a program force the compile-time parts to be included as well, but the latter will often use system compiler procedures, which in turn will have to be linked in.) Thus a linked program could end up containing large amounts of code it does not actually require -- yet another reason for adhering to the rule above.
Note that both the example syntax procedures mentioned already (i.e. LIB * vedset, * XptVal) also contain instances of such 'own' run-time procedures that are permanent constants defined in separate files (LIB * vedset_pcompose and * XptValTestTypedArg respectively). For ordinary Pop-11 use, there is no reason why these procedures should not simply be defined as lconstants in their respective syntax files; but they have to be permanent identifiers, and in separate files in order to work with Popc.
By definition, lexical CTPs (e.g. lconstant syntax and macro) can only be used in the files they are defined in, that is, can operate only when Popc is being used to directly compile their files. Hence any lconstant run-time procedures they employ must also be compiled in Popc mode, etc.
Unless specifically referenced in a non syntax- or macro-expanding context, lexical compile-time code will not be output on the object file at all: thus the issue of mixing it with run-time code in the same file does not apply.
A technique commonly used in multi-file programs is to build up global data structures across a number of files. In the normal interactive system, a structure (such as a list) may be initialised to an empty state in (say) the first file loaded; later files then add increments to the structure, as appropriate. A permanent identifier containing a structure created in this way is called an incremental identifier.
In an environment where each file is compiled individually, incremental identifiers necessitate a global linking process. This is so because an individual object module can do no more than record its particular contribution to each structure; the final structures must be assembled from the various parts when the program is linked.
Popc provides support for incremental identifiers containing one of the following types of structure:
The basis of the facility is the declare_incremental declaration, which declares a permanent identifier as being incremental and of one of the three types listed above, viz: incremental list, incremental procedure or incremental property. (Note that the identifier must first be declared in the normal way, i.e. declare_incremental does not subsume the basic constant/vars declaration.)
When a permanent identifier is declared incremental, Popc records a top-level assignment to it as being merely a part of the overall structure in the identifier. At program link time, various object modules containing increments for the identifier may have been incorporated in the program; Poplink then assembles the final structure from all the parts included.
Note that various system identifiers are incremental, e.g. popautolist, popuseslist and pop_runtime_actions (lists), popexit (procedure), and vedgetsysfilepdr (property). When Popc extracts declarations for these identifiers from system w-libraries, it also recovers the corresponding incremental declarations, thus allowing user files to define additional increments to the standard values of these identifiers.
declare_incremental [syntax]
This construct has the form
declare_incremental
list [attributes] (perm_identifier, ... ),
procedure [attributes] (perm_identifier, ... ),
property (perm_identifier = propexpr, ... ),
;
The body may contain repeated clauses, each beginning with one of the keywords list, procedure or property, separated from the next keyword clause by comma, or terminated by semicolon.
For list or procedure, each keyword clause consists of either a single perm_identifier name, or a comma-separated list of them in parentheses. The identifiers may be optionally preceded by a comma-separated sequence of attributes in square brackets [...], where each attribute is one of
prec=integer
sublists
writeable
For prec, integer is a positive or negative integer in the range -16:8000 to 16:7FFF. E.g.
declare_incremental list [sublists,prec= -200] (foo, baz);
(N.B. A negative integer currently requires a space between the = and the minus sign.)
A property clause consists of either a single
perm_identifier = propexpr
spec, or a comma-separated list of them in parentheses (no attributes are allowed). In each case, propexpr is any Pop-11 expression which evaluates to a property. E.g.
declare_incremental property (
xxxx = newassoc([]),
yyyy = newproperty([], 8, false, false),
);
The semantics of declare_incremental are explained in the following sections.
After its (first) declaration with declare_incremental, the Popc shadow value of an incremental identifier is set to its corresponding 'empty' value, namely [] for lists, identfn for procedures, and the evaluated propexpr for properties.
This allows execute-level code in a file to add to the identifier, i.e. to add further elements to its 'existing' value (as if it had one). For example, if incr_list is an incremental list, then the code
[abcd] <> incr_list -> incr_list;
'adds' the word "abcd" to the list. It is important to note however, that (even though the end result may appear the same), Popc distinguishes the above from
[abcd] -> incr_list;
which -- by virtue of the fact that it does not access the existing value of the identifier before assigning to it -- is deemed to provide the basic initialisation for the identifier.
The basic initialisation for an incremental identifier is like an ordinary initialisation for a non-incremental identifier (only one file can make the initialisation, etc). It is 'the part that should always be present', as opposed to additions by other modules, which are 'optional parts' (i.e. will be added only if the modules have been incorporated in the linked program for other reasons).
An incremental list identifier must always be a variable (this applies also to incremental procedures, but not to properties -- see below). After declaration with declare_incremental list ..., the Popc shadow value of the identifier is set to []; thus execute-level code that adds to the identifier, e.g.
declare_incremental list incr_list;
"abcd" :: incr_list -> incr_list;
incr_list <> [abcd] -> incr_list;
will work as expected.
The linked value of an incremental list is a list whose elements are the element(s) from each object module.
You can use the prec (precedence) integer attribute with different values in different files to control the order in which elements from the various object modules will appear in the final list. The increments from different modules are sorted in numerical prec order, with the lowest (most negative value) coming first. Note that if prec is not specified in a particular module then it defaults to 0 (thus negative values will come before any defaults, and positive values after them).
Elements from each individual module will appear in the final list in the same order they appeared in the module. However, if the sublists attribute is specified in a module, then all the elements from that module will appear as a single sub-list of the final list (rather than as separate elements).
The writeable attribute controls whether the final list is put into writeable memory: if writeable is present in any module, the list is made writeable.
As with lists, incremental procedure identifiers must be variables. After declaration with declare_incremental procedure ..., the Popc shadow value of the identifier is set to identfn; thus execute-level code that adds to the identifier, e.g.
declare_incremental procedure incr_pdr;
pdr1 <> incr_pdr -> incr_pdr;
will work as expected.
The linked value of an incremental procedure is a procedure which executes in turn the procedures from each object module. (Currently, Poplink implements this as a closure
appdata(% procedure_vec, fast_apply %)
where procedure_vec is a full vector containing the component procedures.)
As with lists, the prec integer attribute controls the order in which procedures from the various object modules will appear in the final procedure.
Incremental properties are slightly different from the other two types, since the identifier itself contains a value (a property procedure) which is independent of whatever increments are actually supplied to build the property (its entries). For this reason, the identifier may be constant or variable, as desired (it may also be procedure-type).
After declaration with declare_incremental property ..., the Popc shadow value of the identifier is set to the evaluation of the Pop-11 expression supplied (which must, of course, produce a property). Thus execute-level code that adds to the identifier, e.g.
declare_incremental property incr_prop = newassoc([]);
27 -> incr_prop("abcd");
28 -> incr_prop("efgh");
will work as expected. Note that whereas with list and procedures, a file is deemed to initialise an identifier if it assigns to it without accessing it first, in the property case this is done by explicitly assigning a property to the identifier. For example, if the above assignments are replaced by
define incr_prop =
newassoc([[abcd 27] [efgh 28]])
enddefine;
etc, then the entries in incr_prop will be taken as its basic initialisation.
There is no precedence relationship for a property; the increments generated by each object module are simply a set of entries, which are arranged inside the property in the usual way.
The property expression supplied to declare_incremental for each identifier may be any kind of property. However, it must be capable of evaluation in an arbitrary Pop-11 context (since incremental property declarations from library modules must be resurrectable in any environment). This means (for example) that the property's default or equal_p values can only be standard system values/procedures.
The same expression must be used for an identifier in all modules (Poplink treats any discrepancy between modules as an error).
As described in Overview above (and also by Use of W-Libraries in HELP * POPC), each w-file records which w-libraries were used for extracting permanent identifier declarations when it was compiled. Thus every w-file knows which libraries it needs when linked as part of a program.
When compiling a set of source files with which to build a new w-library, you cannot (of course) rely on this mechanism to deal with identifier references between the source files of the new library (for the fairly obvious reason that the w-library cannot be used for extracting declarations until after it is built).
As described in earlier sections, therefore, you must supply explicit declarations for all permanent identifiers referenced across the source files of the library. However, to enable each w-file that references other library parts to record that it 'uses' this library, the syntax construct library_declare_section is provided: this informs Popc that a set of identifiers belong to a particular library (even though it doesn't yet exist).
Essentially, library_declare_section is a 'forward' declaration for w-libraries. It is also required in the more complicated situation of building a whole collection of libraries which use each other recursively (as with many of the Poplog libraries). In this case, the Popc -wlib option can be used to make available for declaration-extraction those libraries that have already been built, while library_declare_section is used for those that have yet to be built (including the one currently being compiled).
library_declare_section [syntax]
This construct has the form
library_declare_section w-library-name-string
statement-sequence
end_library_declare_section;
where statement-sequence is any sequence of execute-level statements. Its effect is to mark all permanent identifiers declared inside statement-sequence as belonging to the w-library given by (the quoted string) w-library-name-string.
w-library-name-string follows the same rules for specifying w-library names to Popc, Poplink and Poplibr, as described in Object & Library File Arguments in HELP * POPC.
Note that if w-library-name-string contains an environment variable/logical name, it will NOT be expanded when the name is stored on w-files. This is important, since it means environment variables will remain unexpanded until the w-libraries are actually used at link-time, allowing their locations to change. (For example, all Poplog libraries are referenced by names starting with (at least) $usepop/, and others as $popautolib/, $popliblib/, etc.)
declare_updater [syntax]
Where a permanent identifier contains an undef procedure defined in some other file, this declaration informs Popc that the procedure has an updater. The format is
declare_updater perm_identifier, perm_identifier, ... ;
It is usually required only when a constant procedure from one file is used as the pdpart of a closure in another. E.g.
constant procedure baz; ;;; defined in some other file
declare_updater baz; ;;; say it has an updater
define foo = baz(% true %) enddefine;
When generating code for the closure foo, Popc will generate an updater for it in the normal way (i.e. to call the updater of baz). Without the declare_updater, baz is assumed not to have an updater, and so the closure foo will not be given one either. (Note that declare_updater declarations are not required for system procedures, since the information is extracted automatically from system w-libraries.)
In ordinary Pop-11, this declaration merely checks that each perm_identifier contains a procedure with an updater.
normal_compile [syntax]
This construct allows code to be executed in non-Popc mode while compiling a file with Popc. Its format is
normal_compile
statement-sequence
end_normal_compile;
It locally sets pop_pas_mode to its usual value of false, and then does
pop11_exec_stmnt_seq_to("end_normal_compile") -> ;
i.e. execute statements until the closer is reached. (In ordinary Pop-11, this is therefore effectively a no-op.)
One of the functions performed by Poplink is to generate word records for all quoted words used in a program, and to arrange them in the Pop-11 dictionary.
In the normal Pop-11 system, when you declare a permanent identifier it automatically becomes associated with the word corresponding to its name (either at top-level or inside a section). This makes the identifier accessible from the dictionary, either for further program compilation, or for use with valof.
When Poplink links a program it assumes that in general, no more compilation will take place. Hence by default, the only permanent identifiers 'exported' to the dictionary are those for which the corresponding quoted word was used somewhere in the program. (This allows valof to be used on such words; note however, that it does not cope with the case of a program constructing words at run-time with consword.)
Exporting of other identifiers must be requested explicitly, in one of two ways:
(1) By section, with the -s option to Poplink (see HELP * POPC);
(2) By individual identifier, either with uses-by_name in program
files, or by Poplink options -ubn and -idexp etc. (See the
description of uses-by_name in HELP * uses.)
A section is generated in the section tree either when an explicit -s option is given for it, or when one or more identifiers in that section are exported.
In addition to using valof on quoted words and word identifiers, some programs and facilities also rely on being able to autoload library identifiers at run-time (i.e. before accessing them with valof). In a standalone linked program this is probably undesirable, since it requires that the corresponding library source directories and files be present in the program's execution environment. It also requires the incorporation into the linked program of sys_autoload, together with the whole Pop-11 compiler and Poplog Virtual Machine.
Apart from those parts directly connected with the Pop-11 compiler, all system and library code that uses valof makes only weak references to sys_autoload (including valof itself). If sys_autoload is only weakly defined, these facilities will not attempt to use it. Thus, if you do require your program to perform run-time autoloading, you must include
uses (sys_autoload, pop11_compile);
somewhere within the code.
Without this, run-time autoloading will not occur. You must then ensure that all library identifiers to which valof may be applied are included at link-time, and exported to the dictionary. The best way to do this is with uses-by_name statements in your program.
For example (as described in Class Apply Of Descriptors in REF * XptDescriptor), the general apply mechanism for the XptDescriptor datatype constructs the name "XpttypeApply" when given an XptDescriptor to apply, and then calls the apply procedure got from valof applied to the name; thus (e.g), applying a Widget will call XptWidgetApply. In the normal system this would be autoloaded, but autoloading cannot happen without sys_autoload being present. Hence to apply a Widget, your program must force the inclusion of the procedure with
uses-by_name (XptWidgetApply);
If at any point in the linking process all the references to a given identifier are weak (made with weakref), Poplink will not search for that identifier in libraries. For all identifiers that remain weak at the end, Poplink 'plugs' references to them by generating appropriate dummy structures.
To avoid generating the same dummy structure repeatedly, only one dummy is produced for each different class of identifier, i.e. one for all untyped ordinary identifiers, one for procedure-type ordinary identifiers, one for active variables of each multiplicity M, and so on.
For constants, the dummies generated are anonymous undef records of the appropriate type -- that is, all constants of a particular identifier class are set to have their idval equal to the same undef record.
For variables, a single dummy identifier is generated for each class. The idval of each of these identifiers is initialised to the appropriate constant undef record (although you should never rely on a weak variable having any particular value, because dummy program assignments to it may alter the initial value).
In the ordinary system (as described in REF * IDENT), the expression
testdef idname
is equivalent to the truthvalue of the expression
isdefined(ident weakref idname)
However, when a program is compiled and linked with Popc/Poplink, this is so only in relation to the state at link-time. In order to make such tests as efficient as possible, Popc compiles them to zero/non-zero tests on the values of special external symbols, which are assigned values by Poplink after the 'strengths' of the identifiers have been determined at link-time.
Hence the truthvalues of testdef are permanently frozen, i.e. if an identifier is weak at link-time, but then becomes defined at run-time, a Popc-compiled testdef for it will remain unaffected.
The -e option to Poplink specifies the name of the entry procedure to be called on image startup; if omitted, it defaults to $-Pop$-Main (as for the -entrymain argument to * MKIMAGE). The entry procedure must be defined as a procedure-type constant.
On startup, arguments in poparglist0 and poparglist are set up as described in General System Startup in REF * SYSTEM.
Saved image arguments (starting with +) are interpreted only if syssave and/or sys_lock_system has been linked into the program (these will be incorporated only if a strong reference occurs for either one -- both bring in sysrestore automatically). You can use syssave or sys_lock_system to save an image on top of your program image; such a saved image will of course only restore into your program, not into normal Poplog. If one or more + arguments are present, the saved images(s) are restored and control is passed to the last one (i.e. the syssave or sys_lock_system that created it will return true).
If no saved images are present, the specified entry procedure is called (from inside setpop). Before doing this, or before handing control to a saved image, special flag arguments in poparglist beginning with % are processed in the normal way.
Popc-compiled code may be linked into and called from a C program. The linking aspect is controlled by the Poplink options -exmain and -exlink, which provide for either ordinary Pop linking with external startup (-exmain), or independent external linking (-exlink). See External Startup & Linking in HELP * POPC for details.
The interface for calling Pop procedures from C is the same as for external callback, described in REF * EXTERNAL. Use the callback function pop_get_ident to access procedures from Pop you wish to call from outside. Note that you will generally need to specify uses-by_name for such procedures, to ensure that they are exported to the dictionary (otherwise pop_get_ident will not be able to access them).
Before any callback functions are called, the Pop system must be initialised by calling pop_sys_init (this function is in a library supplied by Poplink):
#include "callback.h" /* $popexternlib/callback.h */
int pop_sys_init(unsigned plog_nwords, unused);
As described in Callback and Prolog in REF * EXTERNAL, the Poplog Prolog area cannot be expanded during a call from external code. When Prolog routines are incorporated into a standalone-linked C program, similar restrictions apply; moreover, the Prolog area cannot be located in its normal position on the callstack, and must be allocated initially in ordinary memory. The plog_nwords argument enables you to specify the size of the area in words; it effectively assigns the (afterwards fixed) value of pop_prolog_size. If plog_nwords is 0, the normal default size is used.
The unused argument is reserved for future use, and should be passed as 0.
After initialisation, interrupt is set to sysexit; hence by default, any mishaps occurring in Pop will cause image exit. However, a callback function called from outside Pop always sets PEF_CATCH_ABEXIT_NEXT in *pop_external_flags, so if you reset interrupt to setpop in your Pop code, any mishaps or abnormal exits will simply result in the callback function returning a 0 result.
In a system where it has full control, Pop normally assigns handlers for all Unix signals (so as to implement Ctrl-C interrupts, timers, child process control, etc, etc).
A controlling C program may, however, wish to have its own signal handlers, and not want Pop to interfere with them. The C variable
sigset_t * _pop_exclude_sigset;
is therefore provided; if set to a pointer to a signal set (see e.g. UNIX * sigsetops), then * sys_reset_signal will not change the underlying handler for any signal in the given set.
Needless to say, if you use this to prevent Pop setting handlers for things like SIGALRM and SIGVTALRM, you can expect the corresponding Pop facilities such as sys_timer to stop working. In terms of modularity Unix signals are a total disaster, and there is no general solution to the problem.
(No corresponding problem exists in VMS, where handlers for conditions and timer interrupts etc are specified in a modular way.)
When Popc generates object code, it allocates one or more global external symbols to each permanent identifier. An identifier whose full pathname is
$-foo$-baz$-grunge
will have associated external symbol(s) of the form
prefix.foo.baz.grunge
where prefix is 1-3 characters, e.g.
i.foo.baz.grunge (identifier record, when required)
c.foo.baz.grunge (value of a constant)
xc.foo.baz.grunge (execute label of a constant procedure)
uc.foo.baz.grunge (constant procedure updater)
xuc.foo.baz.grunge (execute label of constant procedure updater)
(N.B. On systems where '.' is not allowed in symbol names, '__' is used instead.)
--- C.all/ref/popc --- Copyright University of Sussex 1994. All rights reserved.