[Code scan] Make UniMol tokenizer save and load self-contained

[njzjz]

This issue is a result of a Codex global repository scan.

Summary

UnimolTokenizer.save_pretrained writes tokenizer_config.json and mol.dict.txt, but from_pretrained requires UnimolConfig.from_pretrained. A tokenizer-only saved directory lacks config.json, so it cannot be reloaded as a standalone tokenizer. The implementation also bypasses Transformers vocab-file resolution by directly joining local paths.

Code references

unimol_tools/unimol_hf/tokenization_unimol.py

Lines 122 to 158 in 4596596

	def save_pretrained(self, save_directory, **kwargs):
	os.makedirs(save_directory, exist_ok=True)
	tokenizer_config = {
	"auto_map": {
	"AutoTokenizer": ["unimol_hf.tokenization_unimol.UnimolTokenizer", None],
	},
	"conformer_seed": self.conformer_seed,
	"conformer_mode": self.conformer_mode,
	"remove_hs": self.remove_hs,
	"model_max_length": self.model_max_length,
	"cls_token": self.cls_token,
	"sep_token": self.sep_token,
	"pad_token": self.pad_token,
	"unk_token": self.unk_token,
	"tokenizer_class": self.__class__.__name__,
	}
	with open(os.path.join(save_directory, "tokenizer_config.json"), "w", encoding="utf-8") as f:
	json.dump(tokenizer_config, f, indent=2)
	self.save_vocabulary(save_directory)
	@classmethod
	def from_pretrained(cls, pretrained_model_name_or_path, config=None, **kwargs):
	if config is None:
	config = UnimolConfig.from_pretrained(pretrained_model_name_or_path)
	dict_path = config.resolve_dict_path(pretrained_model_name_or_path)
	local_dict = os.path.join(pretrained_model_name_or_path, "mol.dict.txt")
	if os.path.isfile(local_dict):
	dict_path = local_dict
	tokenizer_kwargs = {
	"dict_path": dict_path,
	"conformer_seed": config.conformer_seed,
	"conformer_mode": config.conformer_mode,
	"remove_hs": config.remove_hs,
	}
	tokenizer_kwargs.update(kwargs)
	return cls(**tokenizer_kwargs)

Impact

Users who save just the tokenizer, or load a tokenizer from a cached/remote Transformers layout, can hit config-loading failures even though the saved tokenizer directory contains the dictionary and tokenizer metadata.

Suggested fix

Let PreTrainedTokenizer handle save/load metadata where possible, read tokenizer_config.json directly, accept resolved vocab_file or dict_path, and treat model config as optional defaults rather than a hard requirement.