Flexible environments in dynamic lexical analysis systems

Abstract

In this thesis, a system for studying human/computer interfaces is introduced. The human/computer interface provides several features, the most noteable of which is TOKEN COMPLETION. These features permit the user to define and/or redefine command tokens, define and/or delete synonym and noiseword tokens, and to establish a terminal environment. The terminal environment includes the ability to specify automatic comment blocking, token look-ahead, and to control the source of data input (keyboard, VMS file, or I/O buffer).

The ability to token complete is based on a forest of generalized trees used to implement dynamic deterministic finite state automata (DDFA). These trees are built during IPL and loaded with command, synonym, and noiseword tokens, all of which are stored in separate VMS files. Synonym and noiseword translation is carried out in the lexical analysis process, thereby negating any need to specify these functions in the grammar of the language. Insertion and deletion into the forest may be executed at any time, permitting the dynamic definition and deletion of synonyms and noisewords. During synonym and/or noiseword definition, lexical analysis switches to a deterministic finite state automata (DFA) mode of operation. Upon completion, lexical analysis reverts to DDFA mode.

A sample grammar is provided in the APPENDICES. The lexical analysis process is not tied into this grammar but rather is very general and will process any tokens stored in the command, synonym, and noiseword files. The sample grammar is LALR(1).