SYSDOC NT Robert Duncan, July 1994
Porting Poplog to Windows NT 3.1.
Select headings to return to index
This document describes the port of Poplog 14.5003 to Windows NT 3.1 running on an Intel x86 PC.
The main interest is in the operating system interface, which is entirely new and based on the Win32 API. This is the ``32-bit Windows'' API recommended for Windows NT, and also for Chicago, so there must be some hope that this port will work on that too.
The Poplog system name is PCWNT, and the sysdefs.p file defines the following macros for conditional compilation:
WINDOWS = 3.1
WINDOWS_NT = 3.1
The host operating system
WIN32 = true
The target API
Building a system depends on the Windows NT SDK 3.1 and the Microsoft assembler MASM, version 6.1 or later.
New and changed master files for this port are contained in the directories:
C.all/
Versions of existing files which have had to be changed, usually
to extend existing conditional compilations.
C.80386/
Changes for Microsoft assembler syntax.
C.win32/
Pop11 and C source code for the Win32 interface.
C.windows/
Files for system building: mainly makefiles, plus
syscomp/os_comms.p.
S.pcwnt/
Assembly code files, plus syscomp/sysdefs.p. Initial versions of
the assembly code files were generated automatically from the
S.pcunix/ originals using the utoi (Unix-to-Intel) utilities in
/rsuna/pop/wnt/work, but have since been modified as required
for Windows.
None of these files have yet been installed in the masters proper. There are copies both in the directory /rsuna/pop/wnt at Sussex, and on the PC itself in G:\POPLOG. A full list of all new and changed files is given below.
There is a working system installed on the PC in G:\POPLOG. The pop/ directory structure is based on that of other systems, with the following changes:
adm/
Not present -- never used
com/
Not present -- I have so far avoided the need for both login
scripts and command files
doc/
help/
ref/
teach/
Not present -- this documentation depends on Ved, and it's not
yet clear what role Ved has in the final product. In any case,
there's an argument for moving the documentation to Windows Help
File format.
x/
Not present -- not relevant
lib/psv
Not present -- all the standard saved images have been moved to
$popsys, and $popsavelib redefined accordingly. This seems to be
the way things are going generally, with popc.psv, etc. now
located there, but also fits better with the Windows style of
having all the resources used by an application at run time
located in the same directory as the application itself.
lib/
This includes only files from C.all/, so is lacking several
common libraries. Also, no library code has been changed, so
nothing is guaranteed to work (e.g. because of Unix/VMS
assumptions).
extern/src/
This is new and contains a substantial amount of C code used for
interfacing to Win32. This code replaces that from c_core.c and
XtPoplog.c which are not used (a certain amount of code has been
copied from c_core.c: this would obviously be better off in a
separate, common file, if it's possible). Compiling these files
generates a number of static libraries pop*.lib which replace
the single libpop archive used on other systems.
win/
This is also new, and is meant as a replacement for the x/
directory used on other systems, containing GUI-specific files.
There's currently very little in it, because the GUI is not
developed: just an experimental system for providing basic Ved
functionality.
There is also a poplocal/ directory which contains some demonstration libraries for the Ved interface, and an example dialog box.
Running Poplog needs no special preparation: it's sufficient just to start up any of the executables in $popsys, and all initialisation is done automatically. This means there are no global environment variables set up by default, although if you do set any of them, they will be respected (see the discussion on the environment below).
There are currently four executables available:
corepop.exe
corepop11.exe
For system building. These are statically-linked, using just the
files in $popsrc; corepop.exe is an RSV-type image, with symbol
table removed.
pop11.exe
Console-mode Poplog, analogous to basepop11 on other systems.
Dynamically-linked, using the Poplog DLL (see below).
winpop11.exe
GUI Poplog, dynamically-linked. Does nothing by default, because
its standard I/O is null. It needs command-line arguments to
make anything happen (saved images, files, etc.). Saved images
built with pop11 can be used with winpop11 and vice versa.
There are no commands for the other languages -- prolog, clisp and pml
up these commands on Unix and VMS work for Windows. Of course, Windows users don't expect to run programs from a command-line anyway: I've provided a sample Poplog program group in Program Manager which provides some additional entry points. Setting up such a group should be done automatically as part of a proper installation process.
Saved images in $popsys include:
popc.psv
poplink.psv
poplibr.psv
Standard, built with corepop11.
winved.psv
Provides a demonstration ``Ved in a Window'' interface, built
with pop11.
prolog.psv
clisp.psv
pml.psv
Standard, built with pop11 +winved
Rather than recreate the large number of miscellaneous command files that Unix Poplog depends upon, I've gone for a solution based exclusively on the SDK nmake tool, with makefiles provided at various strategic points in the directory tree. It's an implicit assumption that users will not be rebuilding their systems (so no newpop, etc.).
There are makefiles in the following directories:
pop/
Builds the following executables and saved images:
corepop11/corepop .exe popc, poplink, poplibr .psv poplog .dll pop11/winpop11 .exe winved, prolog, clisp, pml .psv
These can be done individually, or
nmake all
will do the lot.
src/
ved/src
win/src
Builds the corresponding src.olb & src.wlb files in $popobjlib.
It will recreate .w/.o file pairs when the corresponding source
files change, but it doesn't know about source-file dependencies
on .ph files; use
nmake -a
to force recompilation.
The makefile in $popsrc/ will also assemble .a files and can build a newpop11.exe from the files in that directory, for porting.
extern/lib
extern/src
Together these build the pop*.lib libraries from the C sources.
Use
nmake nodebug=1
for optimised compilation.
All of these share a common include file -- $popsys/poplog.mak -- which defines common macros and inference rules.
The file poplink_cmnd generated by poplink is itself a makefile, not to be executed directly: instead, do
nmake -f poplink_cmnd IMAGE=<name>
where the IMAGE= argument is optional and overrides the default image name.
Suppose we're starting from $usepop, with .a files and a poplink_cmnd in $popsrc cross-compiled from a remote machine. To rebuild Poplog as it is now, do:
Although the .olb archives exist in $popobjlib, they're not used at present because of a problem with the Microsoft linker. Archive files encountered on the linker command line are not searched there and then as Unix does, but are saved up to the end of the link when all unresolved symbols are extracted at once. The resulting executable has all the symbols defined in object files placed first, followed by those extracted from libraries in an arbitrary order. This subverts Poplog's assumptions about memory layout, which rely on all the Poplog data area being in a contiguous chunk.
Possible solutions to this include:
For now, the Poplog executables are built directly from the .o files in the various src directories -- so don't delete these! This is OK for the standard configurations, but not for bespoke versions generated by popc which rely on selective archive extraction.
The current system configuration is experimental, in that it relies on a Poplog DLL (Dynamic-Link Library). This is generated by poplink and contains the whole of Poplog: effectively, it is Poplog. The executables pop11.exe and winpop11.exe are simple wrappers linked against the DLL which contain little more than their appropriate subsystem entry points (main or WinMain) which in turn call the Poplog entry point (pop_main) exported from the DLL. These executables are generated without the use of poplink.
The primary reason for using a DLL is that it is the only way of supporting general callback into Poplog from external code (see the discussion of external code below). But it does offer some advantages:
There are, inevitably, some corresponding disadvantages. It's possible to use a DLL only on the assumption that it will always be mapped into the same virtual address range. The preferred base address is given at link time, and for Poplog this is currently set at 8Mb: this sits comfortably above the default 4Mb address for executables, and so will cohabit with substantial applications. If the preferred address range is unavailable, the run-time linker will relocate the DLL on the fly into some suitably large free space. Obviously the DLL is kept internally consistent, but there are two potential reasons why the relocation might cause problems:
When running the same executable on the same system configuration, there's no obvious reason why the location of the DLL should change from one run to the next, but if the system configuration itself should change -- a new version of a system library installed, for example -- there's a possibility of its having a knock-on effect. Run-time linking is more suspect, because the location at which the DLL is mapped might vary depending on whether other DLLs have been linked in previously.
An entirely separate problem relates to run-time licensing: distributing the DLL as it stands means distributing the whole of Poplog, and there is no way to prevent users from writing their own executables linked against the DLL which could start up an interactive session. Applications would probably have to be delivered as stand-alone executables, built by popc (poplink retains the ability to do this -- subject to the linking problem described above).
corepop and corepop11 are stand-alone executables, linked against just the core source files, so avoid any of these problems.
The operating system (Windows) interface is provided by the pop*.lib archives in $popexternlib, built from C source files in extern/src. As far as possible, the interface is restricted to using pure Win32 API functions, as documented in the Microsoft Win32 Programmer's Reference. Use of functions from the C run-time library (CRT) has been kept to an absolute minimum, because
A major exception is the use of the ANSI C math functions -- sin, cos, etc. -- which have no Win32 equivalents.
The full library list used by poplink includes:
popwin.lib ;;; Poplog GUI functions popcore.lib ;;; Poplog core functions $(guilibsdll) ;;; Base Win32/CRT functions (expanded by nmake) advapi32.lib ;;; Registry/security functions winmm.lib ;;; Multimedia timers
Any Pop11 source file which depends on these libraries must include the header file $popsrc/win32defs.ph.
API functions under Windows conform to the Microsoft _stdcall calling convention: this means that the name of an API function, as it appears in an object file, is dependent on the size of the argument build, e.g.
GetLastError --> GetLastError@0
In addition, many API ``functions'' are actually defined as macros which select ASCII or UNICODE versions depending on compile-time flags, e.g.
GetUserName --> GetUserNameA@8 // ASCII
--> GetUserNameW@8 // UNICODE
To avoid Poplog having to do these tricky name conversions, the rule has been adopted that no API functions are called directly. Instead, every function which Poplog needs has a Poplog equivalent defined in one of the interface libraries, using the obvious naming convention:
GetLastError --> pop_get_last_error
Most of these are trivial wrappers, e.g.
DWORD pop_get_last_error(void) { return GetLastError(); }
There are, however, many additional library function which provide higher-level services, or facilities not supported directly by the API. For example, Poplog's notion of time -- seconds since 1/1/70 -- has to be synthesised from the Win32 GetSystemTime. Such conversions are most easily done in C.
It could be argued that there should be a different naming convention to distinguish the pure API wrappers from those functions which provide Poplog ``added value.''
To further narrow the interface between Poplog and the API, a second rule has been adopted which is that Poplog never directly accesses API structured types: the interface is entirely procedural, with additional C functions provided in the interface libraries as necessary. This avoids having to track changes in structure layout, etc. through API version changes.
Unfortunately, the same approach is impractical for macro (#define) and enumerated values, so the win32defs.ph header file does provide a set of Poplog equivalents for many Win32 macros. These have the same names, but prefixed with WIN32:
GENERIC_READ --> WIN32_GENERIC_READ
All macros are defined as pop (big)integers: use _: to get the non-pop equivalents.
To avoid having to check these values by hand for each version or platform, the header file contains instructions for how to generate them automatically on the target system: the header file contains just a list of macro names which are written out to a C program which prints out the matching Poplog definitions. These definitions go in a separate file -- win32macros.ph -- which is generated afresh for each system and so is not installed in the masters.
API functions indicate failure in a variety of ways -- check the documentation. When a failure does occur, the function GetLastError returns the appropriate error code. In Poplog, the macro GET_LAST_ERROR calls this function and assigns the result to the variable _syserror for subsequent testing. The conditions under which API functions may fail are very poorly documented (i.e., not at all) so there may be many failure conditions which Poplog could handle in principle, but which are not currently tested for because they're unknown.
Win32 dynamic memory management is closer in style to malloc and free rather than the more primitive Unix brk and sbrk. In particular, there are no default assumptions about where new space will be allocated. To ensure that Poplog's heap area is contiguous, it must be claimed in a single request (or, perhaps more generally, in a small number of requests, one per segment). This is done by pop_init_heap as part of process -- or DLL -- initialisation. The target heap size is hard-wired at 256Mb, but if this is unattainable, a binary chop is used to reduce to the largest available amount.
To avoid an excessively-large process size, the initial claim merely reserves the storage: this costs no system resources apart from some table entries, but ensures that the reserved space will not be used to satisfy subsequent allocation requests. Chunks of the reserved space are then committed -- i.e., turned into accessible memory -- and decommitted again as the heap expands and contracts.
Because the whole of the heap area is reserved, there are no problems of segmentation or fragmentation caused by externally-loaded code doing its own dynamic storage allocation, because those claims will be satisfied from outside the reserved area.
The choice of a 256Mb target heap size is completely arbitrary -- a supposed compromise between restricting heap size and occupying too much of the available address space. A Windows NT process can expand to a maximum 2Gb virtual address space -- currently with no quota limits -- but this can fragment quite quickly as DLLs are loaded at their allotted places and do their own private storage allocation.
Memory protection is supported, so allows the use of a guard page at the base of the user stack for trapping stack underflow.
In addition to the space reserved for the Poplog heap, pop_init_heap also allocates a private heap object (using the Win32 API function HeapCreate) which is used for storage allocation by functions in the pop*.lib libraries, e.g. for device buffers, window data, etc. This is entirely transparent to Poplog itself.
Win32 supports a notion of environment similar to that of Unix: an association of strings names to string values, inherited by child processes. So this is the natural implementation for systranslate. There are API functions which add, change and delete entries in the environment, so there's no need for Poplog to maintain an equivalent of popenvlist.
It would be bad style to expect PC users to have to run some special login script to set up Poplog's environment, as is currently done under Unix and VMS. Instead, Poplog extracts the information it needs from the Registry. Poplog's registry key is
Software\Integral Solutions Ltd.\Poplog\pop_internal_version
where pop_internal_version is the usual six-digit number. There may be several versions of Poplog active at once and each can have its own pop_internal_version key, but when searching for a key Poplog always looks for the largest value less than or equal to pop_internal_version so that minor version changes don't have to have new keys.
Within the Poplog key, the sub-key
Environment
contains version-specific settings for environment variables.
The environment is initialised on startup by the external function pop_init_environment. This walks a built-in table of essential variables and their hard-wired defaults, and sets each according to the rules:
The set of ``essential'' variables currently consists of:
--------------------------------------------------------------------
The only non-obvious entry here is the equivalence of $popsavelib with $popsys, as discussed above. Note the ';' character used as a separator in $popsavepath and $popcomppath: this is the same as in the DOS/Windows PATH variable.
For the system currently installed on the PC, registry entries have been set up by hand as follows:
HKEY_LOCAL_MACHINE\Software\Integral Solutions Ltd.\Poplog\145003
usepop = G:\poplog
poplocal = G:\poplog\poplocal
HKEY_CURRENT_USER\Software\Integral Solutions Ltd.\Poplog\145000
poplib = G:\poplog\poplib
This should be done automatically as part of an installation process. Also, the updater of systranslate should possibly be extended (a la VMS) to allow updating of Registry entries for saving per-user preferences (e.g. from a dialog box).
Windows doesn't provide any command-line processing, so Poplog has to do it for itself. The method used is a simplification of that used on VMS, and supports quoting, wildcard filename expansion and options files indicated by a leading '@' (this convention is also observed by tools in the SDK and Borland C++, so can probably be considered ``standard''). The text of the command line is obtained from the API function GetCommandLine which works identically for console and GUI applications, unlike the argc/argv mechanism which is for the console only. Wildcard expansion uses the API functions FindFirstFile/FindNextFile.
sysfileok performs the following translations on file names:
o The first component of the filename is Unix-converted:
-- $xyz is replaced with %xyz% -- ~ is replaced with the value of popdirectory -- /tmp is replaced with %TEMP%
As a special case, the name /dev/null is replaced with nul.
If the second argument to sysfileok is <true>, the filename is decomposed into the following parts:
host
a leading '\\' followed by everything up to the next '\' or the
end of the name
disk
a leading single character followed by a ':' (host and drive
components are currently mutually exclusive: that may not be
right)
directory
everything after the host/disk up to and including the last '\'
name
everything after the directory and before the last '.'
extension
everything from the last '.' excluding the version
version
a trailing sequence of zero or more '-'
The value of current_directory is obtained and changed using the API functions GetCurrentDirectory/SetCurrentDirectory which eliminates the need for the kind of directory traversal performed by Unix Poplog, while popdirectory is set initially to the expansion of %HOMEDRIVE%%HOMEPATH%
The following file management procedures are supported:
syscreate sysdelete sysisdirectory sysopen sys_file_move sys_file_copy sys_file_stat
File versioning is identical to that on Unix, with back versions indicated by one or more trailing hyphens, and the number of maintained versions controlled by pop_file_versions. File links (hard and symbolic) are not directly supported by Win32 and so are completely ignored, even though they may appear in the file system if created by POSIX applications.
The API functions MoveFile and DeleteFile fail if the file is currently open, so both sys_file_move and sysdelete have to garbage collect in this situation in case the file is held open by some garbage device within Poplog.
sys_file_stat is compatible with Unix and VMS versions only as far as field 2 in the result vector (the last modified time). Fortunately this is sufficient to support the only standard library functions which use sys_file_stat, viz. sysfilesize and sysmodtime.
sys_file_match is not yet supported.
The following functions related to device I/O are supported:
sysread syswrite sysflush sysclose systrmdev sys_input_waiting sys_clear_input
The majority of I/O code is written in C (in "device.c") for clarity and efficiency, and because it will make it easier to introduce multi- threaded and/or asynchronous I/O in the future. The D_CTRL_BLK structure in the Poplog device record is used to communicate between Pop and C and contains the underlying file handle for the device, its mode and status. Device buffers are allocated in C from the private heap rather than by Poplog so are naturally fixed-address, and are stored in the control block rather than in the D_IN/D_OUT_BUFFER fields of the device record which are always <false>.
Device I/O operates in three independent modes:
Text mode
CR/NL pairs are mapped to a single NL on input and vice versa on
output; ^Z is seen as end-of-file. This is the same behaviour as
provided by the CRT file I/O functions in their default mode and
is traditional in DOS/Windows. In non-text (binary) mode, bytes
are transferred without conversion.
Line mode
Input stops after a NL is seen, regardless of how many bytes
have been read (same as Unix).
Cooked mode (console ONLY)
Supports echo, command-line editing and recall, keyboard-
generated interrupt and quit signals -- though see below for
caveats -- and prints a prompt at the start of each line. In
non-cooked, (raw) mode, all input is passed directly to Poplog
-- including the interrupt characters -- with no echo.
The modes are determined when the device is created from the org argument to syscreate/sysopen, according to the following table:
--------------------------------------
On each console I/O operation, the current console mode is compared against the desired cooked/raw mode of the device and changed if necessary; the complexities of Unix tty mode-changing are avoided simply because the overhead of testing and changing the mode is minimal.
Device I/O is implemented in terms of the API functions ReadFile and WriteFile. These -- like all API functions -- are non-interruptible: i.e., once the function has been called, it will return only when the operation has completed, either correctly or with some error. A read from a device which initiates a long-term wait -- such as a pipe, or possibly a remote-mounted file -- will cause Poplog to hang. There is, currently, no solution to this problem, since the ability to interrupt or cancel API calls is an omission from the API itself. It may be fixed in some subsequent release. If not, it might be possible to contrive a solution by using multi-threaded I/O, using a separate thread to read from each device. A long-term wait would then hang only the reading thread, but allow Poplog to recover and continue.
An exception to the no-interrupt rule is that a read from the console will terminate immediately on receipt of a control event (CTRL+C, etc.). Unfortunately, there is no way to distinguish this termination from a normal termination on end-of-file: the associated interrupt signal is raised from a separate thread, and may not arrive until some time later, depending on how the threads are scheduled (see below for a discussion of interrupts). Currently, there's a hack in the code which causes the reading thread to sleep for a short time on seeing end-of-file, giving the handler thread a chance to run and signal the interrupt. This sometimes works and sometimes doesn't; the upshot is that an interrupt at the console prompt can terminate Poplog.
Asynchronous I/O is not currently supported.
systrmdev returns <true> for a console device only.
sys_input_waiting and sys_clear_input are inadequate, in that they treat only characters already in the device buffers, and not any information waiting to be read.
Synchronous error conditions -- access violation and the like, what Poplog calls ``error signals'' -- manifest themselves in Win32 as exceptions (EXCEPTION_ACCESS_VIOLATION, etc.). For Poplog to handle these, all Poplog code must be executed within the scope of an appropriate handler using the C function pop_exception_filter as the exception filter. This does the following:
This has the effect of running __pop_errsig at the immediate point at which the exception arose, allowing Pop to do a proper stack unwind. The information in __pop_sigcontext is ultimately decoded by System_error.
An initial handler for the main Poplog thread is set up by renaming what has traditionally been called main in "amain.s" as pop_main and then calling this from main or WinMain within a try/except block.
A secondary handler must also be set up around any callback code, in case the calling code has set up handlers of its own which will take precedence over the main handler (being closer to the event). This is done by defining _pop_external_callback in C to call the true callback function Sys$- _external_callback_func inside another try/except block.
The following events are handled by this mechanism:
------------------------------------------------
The signal numbers (SIG_SEGV, etc.) defined in INCLUDE * SIGDEFS are made up just for this purpose; Unix-style names and numbers are used for familiarity.
The one standard error signal not included here is QUIT. Ideally, this would be associated with a CTRL+BREAK event generated from the console, but the reason this can't be done is discussed below. Currently, the QUIT mechanism for doing a hard reset or exit is not supported.
The occurrence of an asynchronous event is communicated to Poplog by calling the C function _pop_add_ast: this adds a notification of the event to a low-level AST (ASynchronous Trap) queue and sets the _trap flag to show that the queue is non-empty. Under normal execution, Poplog checks the _trap flag at regular, safe points and diverts to processing the event queue whenever the flag is set; if Poplog is in a wait state
important that the arrival of an AST should interrupt the wait so that the queue can be processed immediately. Poplog calls the function _pop_rem_ast to remove an event from the queue for processing, and to reset the _trap flag if the queue is empty.
Asynchronous events in Win32 are associated with multi-threading: the occurrence of an event will typically create a new thread to handle the event, or awaken an existing thread waiting for the event to happen. As an example, initialising the timer mechanism creates a new thread which runs the timer callback function whenever a timer expires. This means that _pop_add_ast will almost always be called from a thread other than the main Poplog thread, so a new version has had to be written for Win32 which provides the correct degree of synchronisation.
To alert the main thread to the arrival of an AST, a Win32 manual-reset Event object is used to indicate that the AST queue is non-empty: this is in addition to the _trap flag, and an invariant property of the system is that the Event is signalled if-and-only-if the _trap flag is set. The theory is that whenever the main thread enters a wait state, it should include this event object in its wait set, so that the wait will terminate whenever the event is signalled. A CriticalSection object is used to guard the AST queue, the _trap flag and the Event to properly synchronise concurrent access.
This mechanism works for events occurring during normal Poplog execution and during wait states entered through the procedures:
It does not -- and cannot -- work during wait states entered through API calls, since these are unaffected by Event states and cannot be interrupted.
Nor does it currently work for events occurring during execution of external code, since this will check neither the _trap flag nor the state of the AST queue. If externally-loaded code enters an infinite loop or wait state, then Poplog will hang. It should be technically possible to interrupt a running thread by use of the API sequence:
SuspendThread GetThreadContext SetThreadContext ResumeThread
the idea being to change the thread's execution context to run the AST handler there and then. The implementation is left as an exercise for the reader.
These restrictions mean that Win32 Poplog cannot guarantee to respond to ASTs within any reasonable time period, if at all.
The supported AST types are
AST_SIGNAL AST_TIMER
Unsupported are
AST_APP_PENDING AST_DEV_*
An AST_SIGNAL is indicated by the following events:
The process case is discussed below. The keyboard events are signalled by _pop_keyboard_trap which maps console CTRL events to Poplog signals according to the following table:
------------------------------------ | Win32 Console Event | Pop Signal | |---------------------+------------| | | |
Poplog's console control handler calls _pop_keyboard_trap with the appropriate event code whenever an event occurs; this runs in a separate thread created by the console server. The control handler is installed by the external function pop_init_console, called the first time a console device is created.
In the Ved demo, the base window procedure calls _pop_keyboard_trap with argument CTRL_C_EVENT in response to the designated hot-key F4.
Currently, both CTRL_C_EVENT and CTRL_BREAK_EVENT map to the same Poplog signal. Ideally, a CTRL+BREAK would generate a QUIT signal for hard reset or exit, but making this work depends upon the control handler thread being able to interrupt the main Poplog thread to get immediate attention: this is the same problem discussed above with regard to interrupting external code.
An AST_TIMER is indicated on the expiry of a timer set by pop_timer. The full mechanism is supported, including multiple and repeating timers, in real and virtual time. The implementation is based on the Win32 multimedia timers because of their extra accuracy, and because they work identically in console and GUI applications.
The code for pop_timer is used unchanged from that in "c_core.c". However, this is the only piece of code from that file which is used unchanged, so is currently copied verbatim into a separate "pop_timer.c" file. It should be taken out into a separate C.all/ file.
pop_timer requires that the indication of timer events should be independent of the actual expiry of a timer: a single timer expiry may generate several timer events. This follows from the Unix model, where timer expiry sends a signal to the process, but the same signal can be sent irrespective of whether a timer has expired or not.
The Win32 implementation uses two Semaphores -- one for real and one for virtual time -- where the value of the semaphore indicates the number of timer events outstanding. A timer event is indicated by incrementing the appropriate semaphore value. This is done implicitly by the timer callback function, but can be done explicitly by TIMER_RAISE_TRAP.
To convert semaphore values into ASTs, a separate thread runs in a loop, repeatedly waiting for one or other semaphore to be signalled (i.e., to go non-zero); each time this happens, the thread calls the appropriate timer handler routine which raises the AST. This thread is created when pop_timer is called for the first time; it runs at a slightly higher priority then the main thread so that timer events are serviced as soon as possible. This thread is created by Poplog, and is distinct from the thread created when the multimedia timers are initialised, and which is used to run the timer callback function.
The use of multiple threads requires that the data structures used by pop_timer be protected by a CriticalSection object.
External load uses a shared library mechanism similar to that used in Solaris, etc. It depends on the following API functions:
LoadLibrary
maps a shared library into the process address space
GetProcAddress
obtains the value of a symbol exported from a shared library
FreeLibrary
unmaps a previously-loaded library (for external_unload)
There is no need for any temporary files, '.stb' files or for any separate assembling/linking step.
The mechanism works exclusively with dynamically-linked libraries (DLLs). The input_file_list argument to exload can contain DLL names and nothing else. Each of these is passed directly to the LoadLibrary function with no conversion other than that provided by sysfileok. The API has its own rules for searching for libraries without directory names, so Poplog does no additional searching of its own. To fit in with that, it makes sense for any Poplog DLLs to go in $popsys rather than in a separate directory, since the default search path always includes the directory in which the executable itself was found.
One feature of DLLs is that they must be self-contained: you can't build a DLL with unresolved references, as you can with a DSO under Solaris, for example. This means that the issue of incremental loads -- where an externally-loaded library depends on names exported from previous loads, or from the Poplog image itself -- does not arise. Poplog reflects this restriction by requiring that each exload be self-contained too: all the identifier_specs must be resolved from the libraries given in the accompanying input_file_list. It's quite in order to specify the same DLL in multiple exload calls.
Building a DLL is not trivial, unfortunately, and the rules for loading them are probably sufficiently different from those on existing systems that it might pay to create a separate document ``How to do External Load under Windows''.
The fact that DLLs must be self-contained causes a problem for external programs which want to re-enter Poplog using callback functions such as pop_call. For the external program to be loaded into Poplog it must be built into a DLL; that requires the callback functions and their dependencies to be defined at link time, but those functions necessarily refer to symbols such as _pop_external_callback which are defined only within Poplog itself.
In an existing system such as Solaris, the callback functions are defined in an archive library, libpop.a; the external code is linked against this library, but the consequent references into Poplog are left unresolved until the resultant shared object is loaded. This is not possible with a DLL.
The solution is to have the whole of Poplog itself as a DLL -- poplog.dll -- with the callback functions included in its export list. The associated import library poplog.lib can be used as the defining module when creating a DLL for external load: this creates an implicit dependency on Poplog such that whenever the DLL is loaded, the Poplog DLL is brought in too. Of course, if the Poplog DLL has already been loaded by the application, then the existing instance is shared. This is the case with a normal external load, run from a Poplog executable which is itself linked against the Poplog DLL (such as pop11.exe).
This combination of external load plus callback will not work from statically-linked executables such as corepop11.exe, since loading the external code will also load the Poplog DLL -- duplicating code and data present in the executable itself. Ideally, external load would be disabled in these executables.
A second --- and entirely unrelated -- problem with callbacks concerns the treatment of abnormal exits initiated by Poplog within the callback (e.g., in response to a mishap). The default strategy is simply to wipe out the block of stack frames associated with the external calls, but this is incorrect for Win32. A well-designed external function will include try/finally blocks to provide guaranteed execution of exit code (e.g. to release allocated resources) and these will be subverted by Poplog's strategy.
An ideal behaviour would be for Poplog to unwind in the normal way back to the callback entry, raise a Poplog-specific exception to abort the external calls and then continue the abnormal exit once control has returned to Poplog. This would require significant coordination between _call_external and Callback.
As a short-term alternative, the definition of _pop_external_callback has been changed to set PEF_RETURN_ABEXIT_NEXT in pop_external_flags immediately before calling back; this causes the callback function to return 0 on failure, and the abnormal exit is continued only when control has been returned to Poplog. Users can clear this flag and restore the abort behaviour where they know it to be safe.
The procedure sys_create_process provides a simplified interface to the API function CreateProcess and is the basic means of creating a child process. In line with recent changes to Unix sys_fork and VMS sys_spawn, the argument list includes will_do_wait and ast_p to determine what happens when the new process terminates. To make this work, Poplog expects to be delivered an AST_SIGNAL (SIG_CHLD) on termination. This is not supported directly by the API, so is contrived as follows.
If will_do_wait and ast_p are both <false>, the process is created detached and is subsequently ignored; unlike the other systems, the process ID never even appears in Poplog's process list, because there's no danger of it becoming a zombie. Otherwise, a new thread is created within Poplog, whose task is simply to wait on the process' termination and then raise the appropriate AST.
A Win32 process is identified by an opaque handle where Poplog expects a process ID. Process IDs do exist in Win32, but they are unique only while the process is executing: once a process has terminated, its ID can be immediately reused elsewhere, even if there still exist handles to the original process (surely some mistake?). This means it's unsafe to use the process ID to obtain the process' exit status in the way that Poplog expects. Instead, a fixed-address PROCESS_INFORMATION structure is passed to the thread waiting for the process termination, from which it determines which process to wait for and automatically fills in the exit status on completion. That is subsequently delivered to sys_wait by the procedure Waitpid. Needless to say, this is incompatible with the generic scheme implemented in "sysfork.p", so some of that has been temporarily commented out and will need reorganising at some point.
Even so, it is possible to end up in the confusing position of having duplicate IDs in the process list.
sysobey uses sys_create_process to run the command interpreter (cmd.exe) to execute a command string. This will cope with built-in commands such as DIR, pipelines, output redirection, etc.
At the console prompt or from the Ved command line
$ command
is executed as
sysobey('command')
as under Unix and VMS; the '$' character has no particular significance in Windows, but is as good as anything else. A '$' on its own runs the command interpreter in interactive mode.
There are two demonstration libraries in $poplocal/local/lib which give some ideas about Windows programming within Poplog:
lib winved
Terminal-style Ved running in a separate window
lib memory_dialog
A dialog box allowing changes to popmemlim, etc.
This library is pre-compiled into the winved saved image.
The idea is to provide a simple terminal emulator window within which a normal, non-windowing Ved can function with minimal change. It depends upon modules from the Poplog DLL compiled from the files
edit.c
simple editor window class (PopEdit)
base.c
base (frame) window class (PopBase)
windows.c
C side of the Poplog/Windows interface
together with the file win/src/windows.p which provides the Pop11 half of the interface. The majority of code is in C: the interface to Pop has been made deliberately narrow and fairly trivial.
Calling popwin_create_base_window creates a PopBase window whose client area is entirely occupied by a child PopEdit window. The window is managed by a separate thread: this is a simple device to ensure that the window can respond to paint, resize and keyboard events even while Poplog is busy. The window thread runs the standard Windows message- processing loop. Most messages are handled locally by the window procedure, but significant events are passed on to Poplog by posting messages to the Poplog thread. Currently, the set of significant events includes just
POPM_EDIT_CHAR
a character key press in the PopEdit window
POPM_EDIT_FUNCTION
a function key press in the PopEdit window
POPM_EDIT_SIZE
window size changed
POPM_CLOSE
a WM_CLOSE request generated from the system menu, or Alt+F4,
etc.
The POPM_* message codes are Poplog-specific values above WM_USER.
The key F4 is registered as a hot-key for the base window: if this is pressed, the window procedure calls _pop_keyboard_trap with argument CTRL_C_EVENT to bypass the message-passing protocol and interrupt Poplog directly.
Poplog retrieves messages from its queue by calling popwin_get_message. This has to be called explicitly: there is no asynchronous or automatic message-processing facility. popwin_get_message uses the API functions PeekMessage and MsgWaitForMultipleObjects rather then the usual GetMessage so that it can remain alert to ASTs.
In the winved library, popwin_get_message is called by vedscr_read_input which converts the messages it receives into characters or procedures. It follows that messages will only be processed when Ved is waiting for input.
To initiate output on the window, vedscr_flush_output calls pop_send_message to send one or more edit-control messages to the edit window. Message are sent rather than posted which forces synchronisation between the Poplog and window threads: this is necessary, because the messages contain pointer data. Essentially, the display is updated by sending a string which contains mixed printing characters and control characters for moving the cursor, clearing the screen, etc.
The memory dialog is externally loaded into Poplog from popdlg.dll in $poplocal/local/extern/lib. The box itself was created with the SDK dialog editor and linked into the DLL. Other dialogs could be added into the DLL as required, rather than create a separate library for each.
The dialog procedure is written in Pop11 and converted to a Win32 DialogProc by the procedure make_dialog_proc. This constructs an external function closure of pop_dialog_wrapper and is intended to be a generic interface which should work for all dialogs. The wrapper is essential to convert from the normal C-style calling convention to the _stdcall convention expected of Windows callback procedures. It also packages up the arguments to the dialog procedure into a single structure argument for the Poplog callback, and terminates the dialog box cleanly if the callback fails for any reason.
extern/
src/
ast.c
base.c
console.c
device.c
dllmain.c
edit.c
environment.c
exception.c
external.c
file.c
main.c
memory.c
misc.c
pop_timer.c
popcore.h
popwin.h
process.c
test.c
time.c
util.asm
windows.c
winmain.c
src/
devio.p
sys_create_process.p
sys_file_copy.p
sys_file_move.p
sys_file_stat.p
sys_host_id.p
sys_host_name.p
sys_input_waiting.p
sys_real_time.p
sysdate.p
sysfileok.p
sysio.p
sysisdirectory.p
sysobey.p
syspipe.p
systime.p
systranslate.p
sysutil.p
win32_dir.p
win32defs.ph
win32extern.p
win/
src/
popc.args
windows.p
windows.ph
extern/
lib/
makefile
src/
makefile
poplog.def
src/
syscomp/
os_comms.p
makefile
pop/
makefile
poplog.mak
ved/
src/
makefile
win/
src/
makefile
src/
syscomp/
asmout.p
sysdefs.p
aarith.s
aextern.s
afloat.s
alisp.s
amain.s
amisc.s
amove.s
aprocess.s
aprolog.s
asignals.s
lib/
include/
sigdefs.ph
sysdefs.ph
src/
syscomp/
poplink_main.p
w_util.p
dirs.ph
errors.p
extern_load.p
extern_ptr.p
getstore.p
init_args.p
setpop.p
signals.p
sr_sys.p
sysfork.p
ved/
src/
vdfiles.p
vdinitseq.p
vdprocess.p
src/
syscomp/
genproc.p
--- sysdoc/nt --- Copyright University of Sussex 1994. All rights reserved.