ERROR when saving ckpt

[syj2908]

Hi, I meet this error when saving the checkpoint, is there any solution?

Traceback (most recent call last):
  File "qst.py", line 944, in <module>
    train()
  File "qst.py", line 905, in train
    train_result = trainer.train()
  File "/home/anaconda3/lib/python3.8/site-packages/transformers/trainer.py", line 1553, in train
    return inner_training_loop(
  File "/home/anaconda3/lib/python3.8/site-packages/transformers/trainer.py", line 1927, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/anaconda3/lib/python3.8/site-packages/transformers/trainer.py", line 2265, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/home/anaconda3/lib/python3.8/site-packages/transformers/trainer.py", line 2322, in _save_checkpoint
    self.save_model(output_dir, _internal_call=True)
  File "/home/anaconda3/lib/python3.8/site-packages/transformers/trainer.py", line 2803, in save_model
    self._save(output_dir)
  File "/home/anaconda3/lib/python3.8/site-packages/transformers/trainer.py", line 2861, in _save
    self.model.save_pretrained(
  File "/home/anaconda3/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1832, in save_pretrained
    raise NotImplementedError(
NotImplementedError: You are calling `save_pretrained` on a 4-bit converted model. This is currently not supported

[geniedan]

please try the orignal code on hf models

[paprika0741]

Hi, I’m facing the same issue when saving a model checkpoint using Llama-2-7b-chat-hf. The training works fine, but I encountered the error mentioned above while saving the checkpoint.

I tried switching to Llama-2-7b-hf, but I encountered another issue as below:

Traceback (most recent call last):
  File "/root/nfs/codespace/QST/qst.py", line 942, in <module>
    train()
  File "/root/nfs/codespace/QST/qst.py", line 903, in train
    train_result = trainer.train()
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 1553, in train
    return inner_training_loop(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 1835, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 2679, in training_step
    loss = self.compute_loss(model, inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 2704, in compute_loss
    outputs = model(**inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/accelerate/utils/operations.py", line 632, in forward
    return model_forward(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/accelerate/utils/operations.py", line 620, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/root/nfs/codespace/QST/modeling_llama_qst.py", line 1301, in forward
    outputs = self.model(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/nfs/codespace/QST/modeling_llama_qst.py", line 877, in forward
    inputs_embeds = self.embed_tokens(input_ids)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

[geniedan]

Hi, I’m facing the same issue when saving a model checkpoint using Llama-2-7b-chat-hf. The training works fine, but I encountered the error mentioned above while saving the checkpoint.

I tried switching to Llama-2-7b-hf, but I encountered another issue as below:

Traceback (most recent call last):
  File "/root/nfs/codespace/QST/qst.py", line 942, in <module>
    train()
  File "/root/nfs/codespace/QST/qst.py", line 903, in train
    train_result = trainer.train()
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 1553, in train
    return inner_training_loop(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 1835, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 2679, in training_step
    loss = self.compute_loss(model, inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 2704, in compute_loss
    outputs = model(**inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/accelerate/utils/operations.py", line 632, in forward
    return model_forward(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/accelerate/utils/operations.py", line 620, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/root/nfs/codespace/QST/modeling_llama_qst.py", line 1301, in forward
    outputs = self.model(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/nfs/codespace/QST/modeling_llama_qst.py", line 877, in forward
    inputs_embeds = self.embed_tokens(input_ids)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

what GPU are you using?

[paprika0741]

Hi, I’m facing the same issue when saving a model checkpoint using Llama-2-7b-chat-hf. The training works fine, but I encountered the error mentioned above while saving the checkpoint.
I tried switching to Llama-2-7b-hf, but I encountered another issue as below:

Traceback (most recent call last):
  File "/root/nfs/codespace/QST/qst.py", line 942, in <module>
    train()
  File "/root/nfs/codespace/QST/qst.py", line 903, in train
    train_result = trainer.train()
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 1553, in train
    return inner_training_loop(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 1835, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 2679, in training_step
    loss = self.compute_loss(model, inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 2704, in compute_loss
    outputs = model(**inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/accelerate/utils/operations.py", line 632, in forward
    return model_forward(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/accelerate/utils/operations.py", line 620, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/root/nfs/codespace/QST/modeling_llama_qst.py", line 1301, in forward
    outputs = self.model(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/nfs/codespace/QST/modeling_llama_qst.py", line 877, in forward
    inputs_embeds = self.embed_tokens(input_ids)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

what GPU are you using?

Two GPU . 4090 and A100

[YouAreSpecialToMe]

Hi, I’m facing the same issue when saving a model checkpoint using Llama-2-7b-chat-hf. The training works fine, but I encountered the error mentioned above while saving the checkpoint.
I tried switching to Llama-2-7b-hf, but I encountered another issue as below:

Traceback (most recent call last):
  File "/root/nfs/codespace/QST/qst.py", line 942, in <module>
    train()
  File "/root/nfs/codespace/QST/qst.py", line 903, in train
    train_result = trainer.train()
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 1553, in train
    return inner_training_loop(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 1835, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 2679, in training_step
    loss = self.compute_loss(model, inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 2704, in compute_loss
    outputs = model(**inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/accelerate/utils/operations.py", line 632, in forward
    return model_forward(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/accelerate/utils/operations.py", line 620, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/root/nfs/codespace/QST/modeling_llama_qst.py", line 1301, in forward
    outputs = self.model(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/nfs/codespace/QST/modeling_llama_qst.py", line 877, in forward
    inputs_embeds = self.embed_tokens(input_ids)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

what GPU are you using?

Two GPU . 4090 and A100

Hi, I've updated the code. Now it works well on two A100 GPUs on my side. Can you update the code and try it again?

[paprika0741]

Hi, I’m facing the same issue when saving a model checkpoint using Llama-2-7b-chat-hf. The training works fine, but I encountered the error mentioned above while saving the checkpoint.
I tried switching to Llama-2-7b-hf, but I encountered another issue as below:

Traceback (most recent call last):
  File "/root/nfs/codespace/QST/qst.py", line 942, in <module>
    train()
  File "/root/nfs/codespace/QST/qst.py", line 903, in train
    train_result = trainer.train()
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 1553, in train
    return inner_training_loop(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 1835, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 2679, in training_step
    loss = self.compute_loss(model, inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/transformers/trainer.py", line 2704, in compute_loss
    outputs = model(**inputs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/accelerate/utils/operations.py", line 632, in forward
    return model_forward(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/accelerate/utils/operations.py", line 620, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/root/nfs/codespace/QST/modeling_llama_qst.py", line 1301, in forward
    outputs = self.model(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/nfs/codespace/QST/modeling_llama_qst.py", line 877, in forward
    inputs_embeds = self.embed_tokens(input_ids)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/root/anaconda3/envs/qst/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

what GPU are you using?

Two GPU . 4090 and A100

Hi, I've updated the code. Now it works well on two A100 GPUs on my side. Can you update the code and try it again?

Thanks a lot for your kind reply.

[YouAreSpecialToMe]

Hi, I've tried Llama-2-7b-hf and Llama-2-7b-chat-hf, and both of them worked well on my site using 1-3 A100 GPUs. Can you update the code and try again? Or maybe give more detail information about that?