Release 0.11

• Major incompatible change

  A children parameter has been added to the lambda list of the generic function eclector.parse-result:make-skipped-input-result so that results which represent skipped material can have children. For example, before this change, a eclector.parse-result:read call which encountered the expression #+no-such-feature foo bar potentially constructed parse results for all (recursive) read calls, that is for the whole expression, for no-such-feature, for foo and for bar, but the parse results for no-such-feature and foo could not be attached to a parent parse result and were thus lost. In other words the shape of the parse result tree was

  skipped input result #+no-such-feature foo
  expression result    bar
  

  With this change, the parse results in question can be attached to the parse result which represents the whole #+no-such-feature foo expression so that the entire parse result tree has the following shape

  skipped input result #+no-such-feature foo
    skipped input result no-such-feature
    skipped input result foo
  expression result    bar
  

  Since this is a major incompatible change, we offer the following workaround for clients that must support Eclector versions with and without this change:

  (eval-when (:compile-toplevel :load-toplevel :execute)
    (let* ((generic-function #'eclector.parse-result:make-skipped-input-result)
           (lambda-list      (c2mop:generic-function-lambda-list
                              generic-function)))
      (when (= (length lambda-list) 5)
        (pushnew 'skipped-input-children *features*))))
  (defmethod eclector.parse-result:make-skipped-input-result
      ((client client)
       (stream t)
       (reason t)
       #+PACKAGE-THIS-CODE-IS-READ-IN::skipped-input-children (children t)
       (source t))
    ...
    #+PACKAGE-THIS-CODE-IS-READ-IN::skipped-input-children (use children)
    ...)

  The above code pushes a symbol that is interned in a package under the control of the respective client (as opposed to the KEYWORD package) onto *features* before the second form is read and uses that feature to select either the version with or the version without the children parameter of the method definition. See Maintaining Portable Lisp Programs by Christophe Rhodes for a detailed discussion of this technique.

• The new condition type eclector.reader:state-value-type-error can be used to indicate that a value of an unsuitable type has been provided for a reader state aspect.

• The reader state protocol now provides the generic function (setf eclector.reader:state-value) which allows clients to set reader state aspects in addition to establishing dynamically scoped bindings.

• The macros eclector.reader:unquote and eclector.reader:unquote-splicing now signal sensible errors when used outside of the lexical scope of a eclector.reader:quasiquote macro call. Note that the name of the associated condition type is not exported for now since quasiquotation will be implemented in a separate module in the future.

  Such invalid uses can happen when the above macros are called directly or when the ,, ,@ and ,. reader macros are used in a way that constructs the unquoted expression in one context and then "injects" it into some other context, for example via an object reference #N# or read-time evaluation #.(...). Full example:

  (progn
    (print `(a #1=,(+ 1 2) c))
    (print #1#))

  Another minor aspect of this change is that the condition types eclector.reader:unquote-splicing-in-dotted-list and eclector.reader:unquote-splicing-at-top are no longer subtypes of common-lisp:stream-error. The previous relation did not make sense since errors of those types are signaled during macro expansion.

• Eclector now uses the reader state protocol instead of plain special variables to query and track the legality of quasiquotation operations and the consing dot. The additional reader state aspects are documented but remain internal for now.

  The (internal) macro eclector.reader::with-forbidden-quasiquotation is deprecated as of this release. Clients which really need a replacement immediately can use the new (internal) macro eclector.reader::with-quasiquotation-state.

• Eclector no longer returns incorrect parse results when custom reader macros bypass some reader functionality and the input contains labeled object definitions or references.

  An example of a situation that was previously handled incorrectly is the following

  (defun bypassing-left-parenthesis (stream char)
    (declare (ignore char))
    (loop for peek = (eclector.reader:peek-char t stream t nil t)
          when (eq peek #\))
            do (eclector.reader:read-char stream t nil t)
               (loop-finish)
          collect (let ((function (eclector.readtable:get-macro-character
                                   eclector.reader:*readtable* peek)))
                    (cond (function
                           (eclector.reader:read-char stream t nil t)
                           (funcall function stream peek))
                          (t
                           (eclector.reader:read stream t nil t))))))
  
  (let ((eclector.reader:*readtable* (eclector.readtable:copy-readtable
                                      eclector.reader:*readtable*)))
    (eclector.readtable:set-macro-character
     eclector.reader:*readtable* #\( #'bypassing-left-parenthesis)
    (describe (eclector.parse-result:read-from-string
               (make-instance 'eclector.parse-result.test::simple-result-client)
               "(print (quote #1=(member :floor :ceiling)))")))
  ;; [...]
  ;; Slots with :INSTANCE allocation:
  ;;   %RAW                           = (PRINT '(MEMBER :FLOOR :CEILING))
  ;;   %SOURCE                        = (0 . 43)
  ;; [...]
  ;; The %RAW slot used to contain (MEMBER :FLOOR :CEILING) instead of
  ;; (PRINT '(MEMBER :FLOOR :CEILING)).

• The reader macros for non-decimal radices now accept + in the sign part. For example, Eclector now accepts #x+10 as a spelling of 16.

• The reader macros for non-decimal radices now treat non-terminating macro characters that are valid digits for the respective rational syntax as digits instead of signaling an error. This is in line with the behavior for tokens outside of those reader macros.

  As an example, the following signaled an error before this change:

  (let ((eclector.reader:*readtable*
          (eclector.readtable:copy-readtable eclector.reader:*readtable*)))
    (eclector.readtable:set-macro-character
     eclector.reader:*readtable*
     #\1
     (lambda (stream char)
       (declare (ignore stream char))
       1)
     t) ; non-terminating
    (eclector.reader:read-from-string "#x01"))

• When producing parse results and recovering from an invalid input of a form like

  #1=
  ;; a
  ;; b
  <eof>

  Eclector no longer returns an invalid parse result graph.

• When producing parse results and recovering from an invalid input of a form like #1=#1#, Eclector no longer returns an invalid parse result graph.

• The new generic function eclector.reader:new-value-for-fixup is called by eclector.reader:fixup to compute the replacement value for a labeled object marker, both in ordinary objects and in parse results. Clients can define methods on the new generic function to customize such replacements which is probably only useful when parse results are processed since there is not a lot of leeway in the processing of ordinary objects.

• There is now a default method on eclector.reader:fixup-graph-p which returns true if eclector.reader:labeled-object-state indicates that the labeled object in question is final and circular.

• When eclector.parse-result:parse-result-client is used, eclector.reader:labeled-object-state now returns inner labeled object as its fourth value.

• Elector now breaks up long chains of recursive eclector.reader:fixup calls in order to avoid exhausting available stack space. As a consequence, methods on the generic function eclector.reader:fixup can no longer assume an unbroken chain of recursive calls that correspond to the nesting structure of the object graph that is being fixed up. In particular, a call for an inner object cannot rely on the fact that a particular dynamic environment established by a call for an outer object is still active.

Release 0.10

• The deprecated generic functions eclector.parse-result:source-position and eclector.parse-result:make-source-range have been removed. Clients should use eclector.base:source-position and eclector.base:make-source-range respectively instead.

• The new reader eclector.base:range-length can be applied to conditions of type eclector.base:stream-position-condition (which includes almost all conditions related to syntax errors) to determine the length of the sub-sequence of the input to which the condition in question pertains.

• Minor incompatible change

  The part of the labeled objects protocol that allows clients to construct parse results which represent labeled objects has been changed in an incompatible way. The change allows parse results which represent labeled objects to have child parse results but requires that clients construct parse results which represent labeled objects differently: instead of eql-specializing the result parameters of methods on eclector.parse-result:make-expression-result to eclector.parse-result:**definition** and eclector.parse-result:**reference** and receiving the labeled object in the children parameters, the result parameters now have to be specialized to the classes eclector.parse-result:definition and eclector.parse-result:reference respectively. The object passed as the result argument now contains the labeled object so that the children parameter can receive child parse results.

  This change is considered minor since the old mechanism described above was not documented. For now, the new mechanism also remains undocumented so that the design can be validated through experimentation before it is finalized.

• The new syntax-extensions module contains a collection of syntax extensions which are implemented as either mixin classes for clients or reader macro functions.

• The extended package prefix extension allows prefixing an expression with a package designator in order to read the expression with the designated package as the current package. For example

  my-package::(a b)

  is read as

  (my-package::a my-package::b)

  with this extension.

• A new syntax extension which is implemented by the reader macro eclector.syntax-extensions.s-expression-comment:s-expression-comment allows commenting out s-expressions in a fashion similar to SRFI 62 for scheme. One difference is that a numeric infix argument can be used to comment out a number of s-expressions different from 1:

  (frob r1 r2 :k3 4 #4; :k5 6 :k6 7)

• The concrete-syntax-tree module now produces a better tree structure for certain inputs like (0 . 0). Before this change the produced CST had the same concrete-syntax-tree:atom-cst object as the concrete-syntax-tree:first and concrete-syntax-tree:rest of the outer concrete-syntax-tree:cons-cst node. After this change the concrete-syntax-tree:first child is the concrete-syntax-tree:atom-cst which corresponds to the first 0 in the input and the concrete-syntax-tree:rest child is the concrete-syntax-tree:atom-cst which corresponds to the second 0 in the input. In contrast to the previous example, an input like (#1=0 . #1#) continues to result in a single concrete-syntax-tree:atom-cst in both the concrete-syntax-tree:first and concrete-syntax-tree:rest slots of the outer concrete-syntax-tree:cons-cst object.

Release 0.9

• The deprecated function eclector.concrete-syntax-tree:cst-read has been removed.

• eclector.reader:find-character receives characters names with unmodified case and is also called in the #\<single character> case so that clients have more control over character lookup.

• The new generic function eclector.base:position-offset allows interested clients to refine the source positions of errors obtained by calling eclector.base:stream-position.

• Some condition and restart reports have been improved.

• A discussion of the relation between circular objects and custom reader macros has been added to the manual.

• Problems in the eclector.reader:fixup method for hash tables have been fixed: keys were not checked for circular structure and circular structures in values were not fixed up in some cases.

• Eclector provides a new protocol for handling labeled objects, that is the objects defined and referenced by the #= and ## reader macros respectively.

• Eclector now avoids unnecessary fixup processing in object graphs with complicated definitions and references.

  Before this change, cases like

  #1=(1 #1# #2=(2 #2# ... #100=(100 #100#)))

  or

  #1=(1 #2=(2 ... #2#) ... #1#)

  lead to unnecessary and/or repeated traversals during fixup processing.

• Fixup processing is now performed in parse result objects.

  Before this change, something like

  (eclector.concrete-syntax-tree:read-from-string "#1=(#1#)")

  produced a CST object, say cst, which failed to satisfy

  (eq (cst:first cst)       cst)
  (eq (cst:raw (first cst)) (cst:raw cst))

  The properties now hold.

• Clients can use the new mixin classes eclector.concrete-syntax-tree:definition-csts-mixin and eclector.concrete-syntax-tree:reference-csts-mixin to represent labeled object definitions and references as instances of eclector.concrete-syntax-tree:definition-cst and eclector.concrete-syntax-tree:reference-cst respectively.

• The stream position in conditions signaled by eclector.reader::sharpsign-colon is now always present.

• When Eclector is used to produce parse results, it no longer confuses end-of-input with having read nil when nil is used as the eof-value (nil makes sense as an eof-value in that case since nil is generally not a possible parse result).

• A detailed description of the constraints on return values of the generic functions in the Reader behavior protocol has been added to the manual.

• The eclector-concrete-syntax-tree system now works with and requires version 0.2 of the concrete-syntax-tree system.

• Eclector provides a new protocol for querying and binding behavior-changing aspects of the current state of the reader such as the current package, the current readtable and the current read base.

  Clients can use this protocol to control the reader state in other ways than binding the Common Lisp variables, for example by storing the values of reader state aspects in context objects.

  Furthermore, implementations which use Eclector as the Common Lisp reader can use this protocol to tie the cl:*readtable* aspect to the cl:*readtable* variable instead of the eclector.reader:*readtable* variable.

  The new protocol subsumes the purpose of the generic function eclector.reader:call-with-current-package which is deprecated as of this Eclector version.

• Eclector now provides and uses by default a relaxed version of the eclector.reader::sharpsign-s reader macro function which requires the input following #S to be read as a list but not necessarily be literally written as (TYPE INITARG₁ VALUE₁ …).

  A detailed discussion of the topic has been added to the manual.