Skip to content

Tokenization max length #11

@kirilltobola

Description

@kirilltobola

What if column is almost empty, then we reduce max_length for other columns. With respect to current formula: max_length = 512 // num_cols.

Maybe calculate dynamically this parameter?

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions