python (65.1k questions)

javascript (44.2k questions)

reactjs (22.7k questions)

java (20.8k questions)

c# (17.4k questions)

html (16.3k questions)

r (13.7k questions)

android (12.9k questions)

Questions - huggingface-transformers

Using the encoder part only from T5 model

I want to build a classification model that needs only the encoder part of language models. I have tried Bert, Roberta, xlnet, and so far I have been successful. I now want to test the encoder part on...
test-img

ls_grep

deep-learning

pytorch

huggingface-transformers

transformer-model

encoder-decoder

Votes: 0

Answers: 1

Latest Answer

Load T5 encoder checkpoint only: from transformers import T5EncoderModel T5EncoderModel._keys_to_ignore_on_load_unexpected = ["decoder.*"] auto_model = T5EncoderModel.from_pretrained("t...
test-img

Dương Xuân Hiếu

How does the finetune on transformer (t5) work?

I am using pytorch lightning to finetune t5 transformer on a specific task. However, I was not able to understand how the finetuning works. I always see this code : tokenizer = AutoTokenizer.from_pret...
test-img

Chan Wing

deep-learning

nlp

pytorch

huggingface-transformers

seq2seq

Votes: 0

Answers: 1

Latest Answer

If you are using PyTorch Lightning, then it won't freeze the head until you specify it do so. Lightning has a callback which you can use to freeze your backbone and training only the head module. See ...
test-img

Aniket Maurya

Hugging Face not able to reload all weights after training

I recently being using a RobertaLarge model, which I perform a down stream Training, using "Trainer" package. All goes well, I see the loss going down, and compare manually some results with...
test-img

Joao de Oliveira

python

huggingface-transformers

roberta

Votes: 0

Answers: 1

Latest Answer

I found out what was wrong. Will share with others, given others may have the same issue. My problem was that I wrapped my model into a DataParallel model = nn.DataParallel(model) So it seems that Tra...
test-img

Joao de Oliveira

How to load huggingface's BERT after fine-tuning with Pytorch Lightning?

I fine-tuned a pre-trained BERT model from Huggingface on a custom dataset for 10 epochs using pytorch-lightning. I did logging with Weights and Biases logger. When I load from checkpoint like so: mod...
test-img

Ilia

pytorch

huggingface-transformers

pytorch-lightning

Votes: 0

Answers: 1

Latest Answer

You can make prediction on a sample and save the results. To test checkpoint loading you can use the saved results to match model output.
test-img

Aniket Maurya

Posts

Questions

Blogs

Jobs