The value of SYMBOL is NIL, which is not of type (AND SYMBOL (NOT NULL)).

[qubit55]

Hello, I'm trying to do the following:

(cffi:defcstruct llmodel-prompt-context
  (logits :pointer)
  (logits-size :size)
  (tokens :pointer)
  (tokens-size :size)
  (n-past :int32)
  (n-ctx :int32)
  (n-predict :int32)
  (top-k :int32)
  (top-p :float)
  (min-p :float)
  (temp :float)
  (n-batch :int32)
  (repeat-penalty :float)
  (repeat-last-n :int32)
  (context-erase :float))

(cobj:define-cobject-class (llmodel-prompt-context (:struct llmodel-prompt-context))) 
;; The above line raises: The value of SYMBOL is NIL, which is not of type (AND SYMBOL (NOT NULL)). 

My understanding is that it doesn't work with :pointer field types. Is there a walk-around?

My system:
SBCL 2.3.4
MacOS 11.7.10

[bohonghuang]

Thank you for the report. Pointer types are supported. You may need to use (:pointer POINTER-TYPE) to declare a pointer field, otherwise cffi-object won't know the type it's dereferencing. Anyways, I will fix the case you provided and treat :pointer as (:pointer :void).

[qubit55]

Ok, I see. I thought that (:pointer pointer-type) was a SBCL-specific feature, so was trying to avoid it.

[qubit55]

Btw, awesome work, incredible library. Thank you!

[bohonghuang]

Ok, I see. I thought that (:pointer pointer-type) was a SBCL-specific feature, so was trying to avoid it.

This is actually the declaration syntax supported by CFFI, and it is available on any implementation supported by CFFI, while the sb-alien package provides an FFI syntax specific to SBCL.

[qubit55]

Ah, you are absolutely right, I've completely misread the CFFI documentation on types.

[qubit55]

I redefined the struct as:

(cffi:defcstruct llmodel-prompt-context
  (logits (:pointer :float))
  (logits-size :size)
  (tokens (:pointer :int))
  (tokens-size :size)
  (n-past :int)
  (n-ctx :int)
  (n-predict :int)
  (top-k :int)
  (top-p :float)
  (min-p :float)
  (temp :float)
  (n-batch :int)
  (repeat-penalty :float)
  (repeat-last-n :int)
  (context-erase :float))

and the following works now:

(cobj:define-cobject-class (llmodel-prompt-context (:struct llmodel-prompt-context)))

Thanks!

[qubit55]

Another question I have is how to pass null pointers to the logits and tokens fields?

My attempt that doesn't work:

(make-llmodel-prompt-context
   :logits (cobj:cobject-pointer (cffi:null-pointer))
   :logits-size 0
   :tokens (cobj:cobject-pointer (cffi:null-pointer))
   :tokens-size 0
   :n-past 0
   :n-ctx 1
   :n-predict 50
   :top-k 40
   :top-p 0.9
   :min-p 0.0
   :temp 0.1
   :n-batch 8
   :repeat-penalty 1.2
   :repeat-last-n 10
   :context-erase 0.75)

[bohonghuang]

Another question I have is how to pass null pointers to the logits and tokens fields?

My attempt that doesn't work:

(make-llmodel-prompt-context
   :logits (cobj:cobject-pointer (cffi:null-pointer))
   :logits-size 0
   :tokens (cobj:cobject-pointer (cffi:null-pointer))
   :tokens-size 0
   :n-past 0
   :n-ctx 1
   :n-predict 50
   :top-k 40
   :top-p 0.9
   :min-p 0.0
   :temp 0.1
   :n-batch 8
   :repeat-penalty 1.2
   :repeat-last-n 10
   :context-erase 0.75)

Try (cobj:pointer-cpointer (cffi:null-pointer) '(signed-byte 32)) and (cobj:pointer-cpointer (cffi:null-pointer) 'single-float). It's viable for these fields to accept cobj:carrays, whose lifetime should be manually handled however.

[qubit55]

Getting a type error:

Error (SIMPLE-TYPE-ERROR) during printing: #<SIMPLE-TYPE-ERROR {10011C6D13}>
   [Condition of type SIMPLE-TYPE-ERROR]

Here's the C struct definition if it helps:

;; struct llmodel_prompt_context {
;;     float *logits;          // logits of current context
;;     size_t logits_size;     // the size of the raw logits vector
;;     int32_t *tokens;        // current tokens in the context window
;;     size_t tokens_size;     // the size of the raw tokens vector
;;     int32_t n_past;         // number of tokens in past conversation
;;     int32_t n_ctx;          // number of tokens possible in context window
;;     int32_t n_predict;      // number of tokens to predict
;;     int32_t top_k;          // top k logits to sample from
;;     float top_p;            // nucleus sampling probability threshold
;;     float min_p;            // Min P sampling
;;     float temp;             // temperature to adjust model's output distribution
;;     int32_t n_batch;        // number of predictions to generate in parallel
;;     float repeat_penalty;   // penalty factor for repeated tokens
;;     int32_t repeat_last_n;  // last n tokens to penalize
;;     float context_erase;    // percent of context to erase if we exceed the context window
;; };

and on the CL's side:

(cffi:defcstruct llmodel-prompt-context
  (logits (:pointer :float))
  (logits-size :size)
  (tokens (:pointer :int))
  (tokens-size :size)
  (n-past :int)
  (n-ctx :int)
  (n-predict :int)
  (top-k :int)
  (top-p :float)
  (min-p :float)
  (temp :float)
  (n-batch :int)
  (repeat-penalty :float)
  (repeat-last-n :int)
  (context-erase :float))

(cobj:define-cobject-class (llmodel-prompt-context (:struct llmodel-prompt-context)))

(make-llmodel-prompt-context
   :logits (cobj:pointer-cpointer (cffi:null-pointer) 'single-float)
   :logits-size 0
   :tokens (cobj:pointer-cpointer (cffi:null-pointer) '(signed-byte 32))
   :tokens-size 0
   :n-past 0
   :n-ctx 1
   :n-predict 50
   :top-k 40
   :top-p 0.9
   :min-p 0.0
   :temp 0.1
   :n-batch 8
   :repeat-penalty 1.2
   :repeat-last-n 10
   :context-erase 0.75)

[qubit55]

Ah, never mind, I've restarted the REPL and everything is working now

[qubit55]

Another question I have, do I need to worry about manually freeing up memory
when instantiating context or the GC will take care of it automatically?

[bohonghuang]

Another question I have, do I need to worry about manually freeing up memory
when instantiating context or the GC will take care of it automatically?

For the structure itself, you do not need to be responsible for its deallocation since the GC will handle it. However, for the pointers within it, the GC will not manage them. If the memory for these pointers is allocated by Lisp, it is best to use cobj:carray, which is a subtype of cobj:cpointer but can be managed by the GC by default. When you use cobj:carrays as pointers to construct a context, you need to ensure that the references to these cobj:carrays remain valid as long as the reference to the context is valid, to avoid dangling pointers. There are several methods to achieve this; for example, you can place the constructed context and the passed-in cobj:carrays together in a Lisp structure.