REF EXTERNAL_DATA John Gibson Dec 1994
COPYRIGHT University of Sussex 1994. All Rights Reserved.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<< EXTERNAL DATA STRUCTURES >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<< >>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
The term external data structure refers to data maintained in memory
outside of the Poplog system proper by 'external' functions and
procedures , that is, those written in non-Poplog languages (such as C,
FORTRAN, PASCAL etc); This file describes the ways in which such data
can be represented and manipulated inside Poplog. It also deals with
issues concerning the use of native Poplog structures in an external
context (e.g. when passed to external functions).
The actual processes of loading external data/functions, and of
calling external functions, etc, are not dealt with here; see
REF * EXTERNAL
CONTENTS - (Use <ENTER> g to access required sections)
1 Representation of External Data
2 External Pointer-Class Records
3 Predicates on External Pointers, Etc
4 Constructing External Pointers
5 External Pointer Information
6 Standard External Access/Conversion Procedures
6.1 Access
6.2 Conversion
7 External Pointer Vectorclass ("exptrvec")
8 Fixed-Address Poplog Structures for External Use
9 Poplog Structures as External Data
10 External Function Closures
11 Miscellaneous
----------------------------------
1 Representation of External Data
----------------------------------
External data is represented by structure-pointer values. These can
either be directly externally-loaded, or can be returned as results from
external functions. Such pointers can be represented inside Poplog (in
the first place) by external pointer records, which contain the pointer
values, and through which the fields in external data structures may be
accessed.
Access through external pointers is done either with in-line code
generated by the Pop-11 syntax form exacc, or with procedures defined by
the syntax form defexacc (both described in REF * DEFSTRUCT).
Note that external functions/procedures are themselves external data
structures, and are represented by external pointers. Calling an
external function is just a special kind of 'data access'.
Since external structure fields may themselves contain further data
pointers, the access code and procedures applied to external pointers
allow a field type "exptr"; this contains an actual pointer value, and
either returns an external pointer record with that value when accessed,
or assigns the value from an external pointer record into the field when
updated. Moreover, such fields are allowed in Poplog records and vectors
too (making it possible for externally-used Poplog structures to contain
external data pointers).
Thus an external pointer record has the properties that it can be used
with external data access code/procedures, and assigned into "exptr"
fields (another property not mentioned is that an external pointer
record is replaced by its pointer value when passed as an argument to an
external function -- see REF * EXTERNAL).
However, so as not to restrict these properties to one kind of record
(and to allow other information to be attached to pointer values), new
classes of Poplog records can be made 'external pointer-class', that is,
to behave directly as external pointers. This is done by giving the
attribute "external_ptr" when a record class is defined, to qualify for
which it must have an "exptr" field in a fixed place (which is assumed
to hold the pointer value an instance of the class represents). The
above properties then apply to any external pointer-class record.
As mentioned above, the syntax constructs exacc and defexacc (see
REF * DEFSTRUCT) are available for accessing or updating through
external pointer-class records (these constructs respectively use the
Poplog Virtual Machine instruction sysFIELD (REF * VMCODE), and the
procedure cons_access (REF * KEYS)). A small number of standard external
access procedures are given below.
---------------------------------
2 External Pointer-Class Records
---------------------------------
As described above, an external pointer-class record is one that can
have external access code applied to it, and which can be assigned into
"exptr" fields. The basic such class of record in the system is the
external pointer (dataword "external_ptr").
The class_= of an external pointer gives equality with any external
pointer-class record with the same pointer value. The procedure
external_ptr_props can be used to attach an arbitrary item to an
external pointer; this props value defaults to false, and is used by its
class_print, e.g.
<external_ptr>
when false, or
<external_ptr props>
when not, etc.
Other built-in external pointer-class records are the "exptr_mem" class
(see below), and "XptDescriptor" used by the Poplog X facilities.
User classes may be constructed by defclass or conskey by supplying the
special attribute "external_ptr" (see REF * DEFSTRUCT, * KEYS); to have
this attribute, the class must have an "exptr" field at the pointer
position ('pointer position' is explained under Format of Data
Structures in REF * DATA). For example,
defclass xptr [external_ptr]
{ xptr_props :full,
>-> xptr_ptr :exptr
};
gives a record class equivalent to an ordinary external pointer. If in
addition, the class starts with a "full" field (as in the example),
external_ptr_props can be used to access or update that field.
---------------------------------------
3 Predicates on External Pointers, Etc
---------------------------------------
isexternal_ptr(item) -> bool [procedure]
Returns true if item is an (ordinary) external pointer, false
otherwise.
isexternal_ptr_class(item) -> key [procedure]
Returns datakey(item) if item is an external pointer-class
structure (as described above), or false otherwise.
is_valid_external_ptr(exptrclass) -> bool [procedure]
Given any external pointer-class record exptrclass, returns true
if its pointer value is a valid address, or false otherwise.
Note that the main purpose of this procedure is to detect
pointers which have standard error-return values like 0 and -1.
It merely checks the value to be roughly in the range of
allowable addresses (and a true return doesn't guarantee that
the pointer is actually usable).
is_null_external_ptr(exptrclass) -> bool [procedure]
Returns true if exptrclass is an external pointer class object
with an external address field containing zero, and false
otherwise. Mishaps if exptrclass is not an external pointer
class record. Many external routines return a NULL as a pointer
field address as an error-return value. Also, external pointers
can refer to things other than addresses, in which case the
is_valid_external_ptr does not always perform the desired test.
---------------------------------
4 Constructing External Pointers
---------------------------------
consexternal_ptr() -> exptr [procedure]
Constructs and returns a new external pointer, with a null (i.e.
zero) pointer value.
-------------------------------
5 External Pointer Information
-------------------------------
external_ptr_props(exptrclass) -> props [procedure]
props -> external_ptr_props(exptrclass)
For exptrclass an external pointer, returns or updates its
associated props item, which may be anything (and defaults to
false, except that the props of an externally-loaded pointer is
initialised to the external symbol name under which it was
loaded, as a string).
Otherwise, exptrclass must be an external pointer-class record
that starts with a "full" field (i.e. its spec_list argument to
conskey begins
[full exptr ... ]
etc); this field is then accessed or updated. (Note that the
applicability of external_props_props to an item can be tested
for using class_attribute on its key, i.e.
class_attribute(datakey(exptrclass), "external_ptr_props")
returns true if it can be applied and false otherwise.)
(Note that "exptr_mem" structures do NOT have an associated
props field, and cannot be used with external_props_props.)
-------------------------------------------------
6 Standard External Access/Conversion Procedures
-------------------------------------------------
Note that the procedures * move_bytes and * set_bytes may also be used
with external pointer-class structures.
6.1 Access
-----------
exacc_ntstring(exptrclass) -> string [procedure]
string -> exacc_ntstring(exptrclass)
Accesses as a Poplog string, or updates from one, a
Null-Terminated sequence of bytes through (the pointer value of)
an external pointer-class record exptrclass.
That is, the base procedure returns a string consisting of all
bytes upto (but not including) the first 0 byte. (N.B. To allow
a null-pointer-terminated array of strings to accessed easily,
termin is returned for a null pointer.)
The updater writes the bytes of the given string into memory at
the pointer, and adds a 0 onto the end. NEEDLESS TO SAY, the
area pointed to must be large enough to contain all bytes
written including the 0 (otherwise corruption of external memory
and/or the Poplog system will result).
6.2 Conversion
---------------
exval_to_popobj(exptr1) -> exptr2 [procedure]
item1 -> exval_to_popobj() -> item2
This procedure is intended for converting an arbitrary value in
and out of an "exval" field.
The base procedure takes the output from an "exval" field, i.e.
an external pointer exptr1, and just returns a copy of it exptr2
(it must copy it since the field access mechanism will generally
pass it a constant external pointer record).
The updater takes a value item1 to be made the value of an
"exval" field, and returns a result item2 which will actually be
assigned into it. Except in the case where item1 is a structure
requiring a fixed-address copy, it is returned unchanged, i.e.
item2 == item1.
On the other hand, item2 is a fixed-address copy of item1 for
any structure whose direct address will be the value, i.e.
anything which satisfies
class_attribute(datakey(item1), "external_noconv")
and which is not already fixed-address. (This basically means
anything that isn't an external pointer class, hasn't the
"external_deref" attribute, and isn't a (big)integer or a
(d)decimal.)
If a fixed-address copy is made, it is cached in a property
against item1 so that assigning item1 into the field again will
return the cached copy. (This also ensures that providing item1
remains non-garbage, then so will the fixed copy.)
exval_to_string(exptr) -> string_or_false [procedure]
string1_or_false -> exval_to_string() -> string2_or_false
This procedure is intended for converting a string value in and
out of an "exval" field.
The base procedure takes the output from an "exval" field, i.e.
an external pointer exptr, and uses exacc_ntstring to access a
null-terminated string from it, returning the string or false if
the pointer was 0. (Note that Poplog strings are always
null-terminated.)
The updater takes a string or false to be made the value of an
"exval" field, where false means a 0 pointer. If string_or_false
is false, then false is returned (since false assigned into an
"exval" field will convert to 0).
Otherwise, string1 is returned unchanged if already
fixed-address, or a fixed-address copy is made; as with
exval_to_popobj, this is cached in a property against string1 so
that assigning string1 into the field again will return the
cached copy. (And ensures that if string1 remains non-garbage,
so will the fixed copy.)
integer_to_boolean(int) -> bool [procedure]
item -> integer_to_boolean() -> int
Converts between a (simple) integer field value and a boolean,
i.e. any nonzero integer translates to true, and 0 to false. The
updater translates any true item to 1, and false to 0.
--------------------------------------------
7 External Pointer Vectorclass ("exptrvec")
--------------------------------------------
A standard vectorclass with element type "exptr" is provided for. This
has dataword "exptrvec", and is defined simply by
defclass exptrvec :exptr;
which gives all the usual procedures for a vectorclass.
The generic data structure procedures described in REF * DATA
(datalength, appdata, explode, fill, copy, etc) are all applicable to
"exptr" vectors, as are the generic vector procedures (initvectorclass,
move_subvector, sysanyvecons, etc) also described in that file.
isexptrvec(item) -> bool [procedure]
Returns true if item is an exptr vector, false if not.
consexptrvec(exptrclass1, ..., exptrclassN, N) [procedure]
-> exptrvec
Construct and return an exptr vector with its elements taken
from the next N external pointer-class records on the user
stack, where the first item on the stack will be at the highest
subscript value.
initexptrvec(n) -> exptrvec [procedure]
Constructs and returns an exptr vector of length n whose
elements are all initialised to 0 pointers. (See also
initvectorclass in REF * DATA.)
destexptrvec(exptrvec) -> (exptr1, ..., exptrN, N) [procedure]
Destruct the given exptr vector, i.e. puts all its elements on
the stack, together with its length (in other words does the
opposite of consexptrvec, except that what you get out are
always standard external pointer records, even though the input
arguments when constructing may be any external pointer-class
records).
subscrexptrvec(N, exptrvec) -> exptr [procedure]
exptrclass -> subscrexptrvec(N, exptrvec)
Returns or updates the N-th element of the exptr vector
exptrvec. Since subscrexptrvec is the class_apply of exptrvecs,
these can also be called as
exptrvec(N) -> exptr
exptrclass -> exptrvec(N)
(Note that you can assign any external pointer-class record into
an exptrvec element, but what you get out is always a standard
external pointer record.)
exptrvec_key -> key [constant]
Hold the key structures for exptr vectors (see REF * KEYS).
---------------------------------------------------
8 Fixed-Address Poplog Structures for External Use
---------------------------------------------------
The sections above deal with the representation of external data inside
Poplog; this section considers the use of native Poplog data structures
externally, i.e, when passed to external functions etc. The major
problem likely to arise in this context concerns garbage collection, for
the following reasons:
(1) The Poplog garbage collector will discard (and reuse the memory
occupied by) any structure which has no references to it
remaining inside Poplog;
(2) When an (ordinary) structure is retained by the garbage
collector, it may be moved to a new location (i.e. its address
may change);
(3) The garbage collector has no knowledge of references to Poplog
structures stored by external functions, and so cannot take them
into account when deciding to retain structures in (1), or to
correct the addresses they refer to in (2).
However, since garbage collections can only occur while executing Poplog
code, the above present no problem PROVIDING the external use of a
structure always finishes before control is ever returned to Poplog.
The problem occurs only when an external agent in some way or
another retains a structure during a return to Poplog execution (during
which a garbage collection happens), and then later expects to use that
structure again (at which time the externally-stored reference may have
become invalid, either through (1) or (2)). That is, the control
sequence
external code --> Poplog --> external code
(GC caused)
must occur (either by returning to Poplog and then re-calling the
external function, or by calling-back to Poplog from the external
function and then returning from the callback, etc).
While this scenario will not arise in many programs (e.g. because
data is always passed anew to an external function on each call, so it
never relies upon stored references), in other programs it will
difficult or impossible to avoid. Poplog therefore provides a solution
to (2) by making possible the creation of fixed-address structures,
whose memory locations do not change; this is done with the procedures
described below. In all cases, they call an ordinary constructor
procedure, but in such a way that the result is allocated memory in a
separate part of the heap reserved for fixed-address objects -- and
where its address is guaranteed to remain unchanged by garbage
collection.
However, while this solves (2), the potential problem of (1)
remains: that is, external code could have stored references to an
(albeit fixed-address) structure for which there are no remaining
references inside Poplog itself (and so a garbage collection would
destroy the object). The procedures below therefore also provide an
option to automatically hold the structures they create on a global list
(from which, when no longer required, they can be freed with the
procedure free_fixed_hold). A given program only need use this option
for structure(s) for which it does not anyway retain suitable references
itself (e.g. in variables or other data structures).
cons_fixed(arg_1, ..., argN, key, hold) -> fixed_struct [procedure]
cons_fixed(arg_1, ..., argN, key) -> fixed_struct
Given a record or vector class key key, calls the class_cons
procedure of key so that the result fixed_struct will be
fixed-address. arg_1, ..., argN are the arguments required by
the class_cons procedure (see REF * KEYS).
hold is an optional boolean argument which if true specifies
that fixed_struct should be retained on a global list (until
freed with free_fixed_hold); if omitted, it defaults to false
(i.e. don't put on hold).
For example,
cons_fixed(1, [], pair_key, true) -> fixed_pair;
will call conspair (and hold the pair), while
cons_fixed('a', 'b', 'c', 3, vector_key) -> fixed_vector;
will call consvector (and not hold the vector).
init_fixed(N, vec_key, hold) -> fixed_vec [procedure]
init_fixed(N, vec_key) -> fixed_vec
Given a vector class key vec_key, calls the class_init procedure
of vec_key so that the result fixed_vec will be fixed-address; N
is the length of vector required. hold is the same as for
cons_fixed. E.g.
init_fixed(10, string_key) -> fixed_string;
will call inits.
copy_fixed(struct, hold) -> fixed_struct [procedure]
copy_fixed(struct) -> fixed_struct
Calls copy on struct so that the copy fixed_struct will be
fixed-address. hold is the same as for cons_fixed.
is_fixed(struct) -> bool [procedure]
Returns true if the address of the structure struct is fixed, or
false if it can change.
This will be true if either (a) struct is an object in the
system (in which case isinheap(struct) will be false), or (b)
struct was produced by cons_fixed, init_fixed or copy_fixed.
A typical use of this procedure would be in combination with
copy_fixed to ensure a fixed-address version of a structure,
i.e,
unless is_fixed(struct) then
copy_fixed(struct) -> struct
endunless;
Note that structures locked in by sys_lock_system become part of
the system, and are therefore fixed-address, whereas those
locked with sys_lock_heap are not (since they can be unlocked
with sys_unlock_heap).
free_fixed_hold(item) [procedure]
For a item a fixed-address structure produced by one of the
above procedures with a true hold argument, removes the
reference to that structure from the fixed hold list (for any
other item it does nothing).
sys_grbg_fixed(fixed_struct) [procedure]
sys_grbg_fixed(fixed_struct1, ..., fixed_structN, N)
This procedure frees the memory occupied by the N fixed-address
structures given as arguments, where the first form is the just
same as
sys_grbg_fixed(fixed_struct, 1)
in the second form. free_fixed_hold is also called automatically
for each structure.
Freeing the memory occupied by a fixed-address structure makes
it available for re-use in any later fixed-address allocation.
USE THIS PROCEDURE WITH EXTREME CARE: as soon as you call it,
the structures freed will become internal objects, and cease to
be whatever they were previously.
-------------------------------------
9 Poplog Structures as External Data
-------------------------------------
While any Poplog structure can be used as external data in the sense of
passing it to an external function, the procedure fill_external_ptr
allows any fixed-address Poplog structure to be used as an external
structure within Poplog itself (e.g. with exacc, etc).
A frequent need in this respect is to (a) create a pointer to an
uninitialised area of memory, (b) pass the pointer to some external
function to fill value(s) into the memory, and (c) on return to Poplog,
access the value(s) with exacc, etc. One way of doing this is for
example
exptr_init_fixed(nbytes, string_key) -> exptr
i.e. create a fixed-address string and a separate external pointer to
it.
However, the system provides a more direct method with the "exptr_mem"
structure. This is a fixed-address, external pointer-class structure
that combines both an area of memory and a pointer to it. (And if
desired, the memory it occupies can be freed directly after use by
calling sys_grbg_fixed.)
initexptr_mem(Nbytes, hold) -> exptr_mem [procedure]
initexptr_mem(Nbytes) -> exptr_mem
Returns an "exptr_mem" structure for an area of memory of size
Nbytes bytes. This is a fixed-address, external pointer-class
structure, whose pointer points into itself at the memory area
of the required size. The hold argument is as for cons_fixed,
etc.
Note that if Nbytes is greater than the size of a machine word,
the pointer is guaranteed to be doubleword aligned (i.e.
suitable for use with machine double floats). datalength applied
to an exptr_mem returns its size, i.e. Nbytes.
As described in REF * DEFSTRUCT, an interface to initexptr_mem
for use with typespec definitions is provided by the
EXPTRINITSTR macro, e.g.
p_typespec bytearray :byte[10];
vars bytearray = EXPTRINITSTR(:bytearray);
bytearray =>
** <exptr_mem>
exacc can then be used with this to access or update the
underlying memory:
`a` -> exacc bytearray[3];
exacc bytearray[3] =>
** 97
(N.B. For efficiency's sake, an exptr_mem structure does NOT
have an external_ptr_props field.)
isexptr_mem(item) [procedure]
Returns true if item is an "exptr_mem" structure, false if not.
fill_external_ptr(fixed_struct, exptr) -> exptr [procedure]
Given any fixed-address structure fixed_struct (i.e. for which
is_fixed is true), makes the pointer value of the external
pointer exptr be fixed_struct.
Note that, since the pointer value of an external pointer is an
external reference as far as garbage collection is concerned,
the problem of (1) in the section above potentially applies,
i.e. if fixed_struct has no other references remaining in the
system then a garbage collection will destroy it.
For this reason fixed_struct is also assigned to the
external_ptr_props of exptr, which at least ensures that if
exptr does not become garbage then neither will fixed_struct.
exptr_cons_fixed(arg1, ..., argN, key, hold) -> exptr [procedure]
exptr_cons_fixed(arg1, ..., argN, key) -> exptr
exptr_init_fixed(N, vec_key, hold) -> exptr [procedure]
exptr_init_fixed(N, vec_key) -> exptr
exptr_copy_fixed(struct, hold) -> exptr [procedure]
exptr_copy_fixed(struct) -> exptr
These three autoloadable procedures call cons_fixed, init_fixed
and copy_fixed respectively, and then use fill_external_ptr (see
above) to return an external pointer to the new fixed address
structure. For example, exptr_cons_fixed could be defined as:
define exptr_cons_fixed(/* args on stack*/) -> ptr_to_struct;
lvars new_struct, new_ptr, ptr_to_struct;
cons_fixed(/* args on stack */) -> new_struct;
consexternal_ptr() -> new_ptr;
fill_external_ptr(new_struct, new_ptr) -> ptr_to_struct;
enddefine;
Note that exptr will have the new vector or record in its
external_ptr_props field.
------------------------------
10 External Function Closures
------------------------------
An external function closure (dataword "exfunc_closure") is a record
that allows an argument value (an arbitrary Poplog item) to be `frozen'
into an external function.
Such a record contains a small piece of executable code, a pointer
to a base external function, and a frozen argument value. The code part
is at the pointer position of the structure, and so when passed out as
an external argument it represents an executable object; when called,
its action is to deposit the frozen argument value in the standard
external variable *pop_exfunc_closure_arg, and then hand over to the
base function (which gets whatever actual arguments were passed to the
closure).
The principal use of these closures is to surmount the problem of
externally-defined callbacks that make no allowance for `client' data to
be passed through to the called-back function. -- see the section
External Callback in REF * EXTERNAL for more details.
Note that external function closures are always fixed-address; also
note that they are NOT external pointer-class structures (they contain
actual code, not pointers to code). Since they are passed as their
structure pointers when given as arguments to external functions, the
net result is the same in that context; however, if you wanted to
execute a closure with exacc, etc you would first have to construct an
external pointer to it (using fill_external_ptr, see above).
make_exfunc_closure(F_exptrclass, arg_item, hold) -> efc [procedure]
Given an external pointer-class record F_exptrclass pointing to
an external function F, constructs an external function closure
efc for that function and the argument arg_item.
efc is always a fixed-address structure; the hold argument is
the same as for the other fixed-address constructors described
in the last section, i.e. true if efc should be retained on a
global list until freed. (Unlike those other procedures, the
hold argument is mandatory.)
The action of efc when executed as an external function is to
place arg_item in the external variable *pop_exfunc_closure_arg
and then call F with whatever arguments were supplied to the
call of the closure -- for details, see above and External
Callback in REF * EXTERNAL
is_exfunc_closure(arg) -> bool [procedure]
Given an arbitrary object arg, this procedure returns true if
arg is an external function closure, and false otherwise.
exfunc_export(proc, flags, hold) -> efc [procedure]
This procedure takes a Poplog procedure and returns an external
function closure which invokes the Poplog procedure using the
'generic' callback stub described in REF * EXTERNAL (External
Callback). proc is any valid first argument to pop_call, namely
a procedure or an identifier or ref record (which must contain a
procedure at call time). flags is a bit mask (ie an integer) of
PEF flags to be or-ed with pop_external_flags before proc is
called. This allows any of the user-assignable PEF flags (see
REF * EXTERNAL) to be set for a particular exported procedure.
Constants defining suitable values for flags are defined in
INCLUDE * EXTERNAL_FLAGS. hold is the same as for other
fixed-address structures.
When efc is called (from external code), it ors flags with
pop_external_flags and then invokes proc, passing a single
external pointer record to it. This external pointer points to
the array of args passed to efc, where each arg occupies an
"exptr" (or "exval") -sized field. When proc returns, efc
dereferences the pointer to recover the first arg and returns it
as result. So if proc wishes to return a result, it can do so by
modifying the first arg position.
-----------------
11 Miscellaneous
-----------------
external_NULL() -> <NULL> [procedure]
This procedure returns a constant value of machine zero, which
prints as <NULL>. It is provided solely to enable the assignment
of a null value into a "full" field of a data structure to be
processed externally (e.g. for terminating a vector of
null-terminated strings in Unix, etc). IT MUST NOT BE USED FOR
ANY OTHER PURPOSE: iscompound is true for this value, but any
procedure in the system which attempts to inspect its key will
cause an access violation.
In particular, you should not attempt to cache the value this
procedure returns by assigning it to a variable or a constant;
use a call of external_NULL() every time the value is needed.
(Note that null_external_ptr is a null pointer for assigning
into an "external_ptr" field.)
null_external_ptr -> exptr [constant]
This variable contains an external pointer with a zero in its
external address field. ie. a pointer to <NULL>. Note that you
cannot modify this address field. The null_external_ptr constant
is provided as a convenience so that libraries can use a
standard exptr when they want to pass an external routine a
pointer containing NULL.
external_ptr_key [constant]
exptr_mem_key [constant]
Constants holding the key structures for ordinary external
pointers and "exptr_mem" structures.
+-+ C.all/ref/external_data
+-+ Copyright University of Sussex 1994. All rights reserved.