Using VGG nets

[skaae]

I'm trying to use the VGG-16 net with pretrained weights.

The link https://s3.amazonaws.com/lasagne/recipes/pretrained/imagenet/vgg16.pkl does not seem to be public?

@ebenolson : I can download the file if I log in with the information you gave me.

I'm not sure how i should pre-process my data to make the model work. I looked at the preprocessing description in the repo:

In the paper, the model is denoted as the configuration D trained with scale jittering. The input images should be zero-centered by mean pixel (rather than mean image) subtraction. Namely, the following BGR values should be subtracted: [103.939, 116.779, 123.68].

I guess i should do something like (not tested):

MEAN_VALUE = np.array([103.939, 116.779, 123.68])   # BGR
def preprocess(img):
    # img is (channels, height, width), values are 0-255
    img = img[::-1]  # switch to BGR
    img -= MEAN_VALUE
    return img

Maybe we should at preprocess functions to the modelzoo?

[ebenolson]

Thanks, forgot to make that one public. Should be good now.

There's an example of preprocessing (for VGG_S) here, but your preprocess function looks correct to me. Are you having problems?

Adding something like that to the modelzoo files seems like a good idea to me.

[skaae]

I haven't gotten that far yet. I'll try it out and if it works I'll add a preprocessing function for VGG. Any clue why do they use BGR ?

The model setup is nice and so far very easy to use :)

[ebenolson]

Apparently it's because Caffe uses OpenCV which uses BGR. We could swap it in the weights, but I think that could confuse people who looked at the Caffe Model Zoo page.

[rbrthogan]

I have a follow up question then. Have all of the models in the model zoo here been trained on BGR or just VGG? Do you think this will remain a standard?

[henridwyer]

I ran into an issue when trying to unpickle VGG19 normalized https://s3.amazonaws.com/lasagne/recipes/pretrained/imagenet/vgg19_normalized.pkl in python 3 - no problems in python 2. Is that a compatibility problem with pickle itself?

[dnouri]

I ran into an issue when trying to unpickle VGG19 normalized

What issue? What's the output you're getting? Probably best to add a separate ticket, too.

[benanne]

In the MNIST example there is some trickery to make the dataset (pickle file) load properly in Python 3. I guess it's because Python 3 assumes utf-8 encoding unless otherwise instructed: https://github.com/Lasagne/Lasagne/blob/master/examples/mnist.py#L34-L45
Maybe the issue you're seeing is related to that.

[henridwyer]

Here were my attempts, before I decided to use numpy's save and load in order to get the data into python 3.

values = pickle.load(open('vgg19_normalized.pkl'))['param values']

output:

UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input> in <module>()
----> 1 values = pickle.load(open('vgg19_normalized.pkl'))['param values']

/home/python3/python3_install/lib/python3.4/codecs.py in decode(self, input, final)
    317         # decode input (taking the buffer into account)
    318         data = self.buffer + input
--> 319         (result, consumed) = self._buffer_decode(data, self.errors, final)
    320         # keep undecoded input until the next call
    321         self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

using the mnist trickery:

values = pickle.load(open('vgg19_normalized.pkl', encoding='latin-1'))['param values']

output:

TypeError                                 Traceback (most recent call last)
<ipython-input> in <module>()
----> 1 values = pickle.load(open('vgg19_normalized.pkl', encoding='latin-1'))['param values']

TypeError: 'str' does not support the buffer interface

Reading as bytes:

values = pickle.load(open('vgg19_normalized.pkl', 'rb'))['param values']

output:

UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-10> in <module>()
----> 1 values = pickle.load(open('vgg19_normalized.pkl', 'rb'))['param values']

UnicodeDecodeError: 'ascii' codec can't decode byte 0xbc in position 1: ordinal not in range(128)

Am I doing something wrong?

[f0k]

Am I doing something wrong?

Yes, you didn't correctly copy the MNIST trickery. It should be pickle.load(open('vgg19_normalized.pkl'), encoding='latin-1')['param values'], not pickle.load(open('vgg19_normalized.pkl', encoding='latin-1'))['param values']. The full and truthful MNIST trickery would even be:

with open('vgg19_normalized.pkl', 'rb') as f:
    values = pickle.load(f, encoding='latin-1')['param values']

I.e., pass 'rb' to open, and encoding='latin1' to pickle.load.

In the discussion of #21, we've pondered converting the models to numpy's .npz format. That would avoid the compatibility and security issues around pickles (and maybe even reduce download traffic when saved with np.savez_compressed()).

[skaae]

Could/should we provide a layer for converting from RGB to (bhwc) to BGR (bchw) similar to

https://github.com/nicholas-leonard/dpnn/blob/master/Convert.lua

[zaheersm]

@ebenolson any ideas on how could we use a pre-trained ImageNet model to fine-tune a dataset of images with 4-channels? (RGB + NIR)

Would using PCA to transform the 4-channels into 3 would work? Are there any other ideas?

[f0k]

Would using PCA to transform the 4-channels into 3 would work?

I doubt so. The PCA components would probably be too different from RGB.

Are there any other ideas?

You could try extending the first-layer filter tensor to have 4 input channels, initialize the additional filters randomly with a carefully-chosen scale and then slowly train. If you fear this would destroy existing weights or not give the extra information enough consideration, you could try to add a separate branch and fuse the additional infrared information later. Good luck!

[oysteijo]

Hi!
I'm trying the ImageNet VGG16 example in Recipes, and I get the same error as henridwyer

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-23-ac5b1f011bed> in <module>()
      1 import pickle
      2 
----> 3 model = pickle.load(open('vgg_cnn_s.pkl'), encoding='latin-1')
      4 CLASSES = model['synset words']
      5 MEAN_IMAGE = model['mean image']

/usr/lib/python3.6/codecs.py in decode(self, input, final)
    319         # decode input (taking the buffer into account)
    320         data = self.buffer + input
--> 321         (result, consumed) = self._buffer_decode(data, self.errors, final)
    322         # keep undecoded input until the next call
    323         self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

This happens both with and without the encoding='latin-1' parameter to pickle.load(). Am I doing something wrong? Can you also please post the correct md5sum to the pickle-file such that I can rule out any problems with the download? Thanks!