Skip to content

Period not correctly tokenized? #1

Description

@huangjs

Here's an example:

LANGUTILS> (tokens-for-ids (vector-document-words (vector-tag "Hello world. I'm here.")))
("Hello" "world." "I" "'" "m" "here.")

I think it should be:

("Hello" "world" "." "I" "'" "m" "here" ".")

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions