pickle.dump gives SystemError

[xdvx]

picke.dump gives me SystemError if I try larger batch of images. Thought it is memory issue, tried same batch on 32GB ram machine, same error.

[bakwc]

Thanks for report! Could you please provide more details? What are you trying to run - pcr.py or nnpcr.py? Are you trying to train a new model? What exact SystemError do you have (please attach full exception trace)? How large is your training set?

[xdvx]

I got this running nnpcr.py

This is an error message:
pickle.dump(obj, open(fileName + '.tmp', 'wb'), -1) SystemError: error return without exception set

I had 10k positive / 10k negative images. While googling I've found only one solution to run same script on Python 3. Took me awhile but no error message anymore.

I have one more question, I see that this is set to constant number:
def train(self, numIterations=1500):

Should this represent the size of training set?

[bakwc]

Seems like this dataset is too large to fit into in-memory cache. You can comment line 114 (saveCache((trainX, trainY, testX, testY), 'nncache.bin')) - this cache is only used to improve speed if running train multiple times.

I have one more question, I see that this is set to constant number:
def train(self, numIterations=1500):

Should this represent the size of training set?

No (at least directly). But you can try to set different numbers here. If numIterations too little or too large - the accuracy will be poor. 1500 was optimal for my training set (3K images total).

[bakwc]

BTW, what accuracy do you have with your dataset? Could you share your model (or how you gather it)?

[xdvx]

Didn't get over 80% yet. Looking for better dataset. Basically I aim to train to recognize photos which are not suitable for advertisement networks. So even minor nudity is not accepted here.

Should I train it only with people photos or should I provide with all kind of different samples as negative ones? Do photo dimensions matter or I can collect photos with smaller resolution? At final stage I hope I could use this script to go through 20 million photos on my website and mark which ones are not suited for showing advertisements.

I am collecting data samples by crawling reddit. So I could gather even huge datasets like hundrends of thousands images.

[bakwc]

Should I train it only with people photos or should I provide with all kind of different samples as negative ones?

Not only people - just arbitrary images. The more - the better.

Do photo dimensions matter or I can collect photos with smaller resolution?

Currently all images converted to 128x128. You could try smaller one, they should will be upscaled.

I am collecting data samples by crawling reddit. So I could gather even huge datasets like hundrends of thousands images.

Crawling reddit is a good idea, may be I'll try too later.

[xdvx]

Would it be smart to double image size? Would it be enough to change this constant?
IMG_SIZE = 128
to
IMG_SIZE = 256

Must positive and negative data samples be at same size?

[bakwc]

Would it be smart to double image size? Would it be enough to change this constant?
IMG_SIZE = 128
to
IMG_SIZE = 256

It's not that simple - you also need to change architecture of neural network - eg. add additinal conv & max_pool layers.

Must positive and negative data samples be at same size?

Yep.

[xdvx]

def train(self, numIterations=1500):

Is this enough for batch of 60K of training data?

[bakwc]

Not sure. Try to increase to 3K and check if quality is better or worse than with 1.5k.

[xdvx]

It really takes lot of ram. I could only run 40k sample on 32gb ram. Will try 80k on 64gb tomorrow. My model size always end up being 13mb, is this right? I still can't get accuracy over 80%.

[bakwc]

It really takes lot of ram. I could only run 40k sample on 32gb ram.

Seems like it currently not optimized for large datests. Currently it is loading the whole dataset in-memory, need to fix working with dataset.

My model size always end up being 13mb, is this right? I still can't get accuracy over 80%.

Current network architecture is not very complex - may be you should try more complicated architectures, eg Inception. You can also try following:

• tune iterations number
• tune size of pre-outer layer (now 1024 - you can try to increase or decrease it)
• tune size of channels for convolutional layers (6, 12, 24)
• add additional layers
• try 5x5 kernel instead 3x3 one (shape=[3, 3, ...] => shape=[5, 5, ...] )

[xdvx]

By the way in new TensorFlow libarry you have to call

init_ops.zeros_initializer

like this, otherwise you will get error
init_ops.zeros_initializer()

I'll test with all those different parameters.

Also I get these warnings:

Use tf.losses.softmax_cross_entropy instead. [2017-03-01 13:01:03,853 deprecation.py:116 WARNING] From nnpcr.py:216: softmax_cross_entropy (from tensorflow.contrib.losses.python.losses.loss_ops) is deprecated and will be removed after 2016-12-30. Instructions for updating: Use tf.losses.softmax_cross_entropy instead. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/losses/python/losses/loss_ops.py:394: compute_weighted_loss (from tensorflow.contrib.losses.python.losses.loss_ops) is deprecated and will be removed after 2016-12-30. Instructions for updating: Use tf.losses.compute_weighted_loss instead. [2017-03-01 13:01:03,865 deprecation.py:116 WARNING] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/losses/python/losses/loss_ops.py:394: compute_weighted_loss (from tensorflow.contrib.losses.python.losses.loss_ops) is deprecated and will be removed after 2016-12-30. Instructions for updating: Use tf.losses.compute_weighted_loss instead. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/losses/python/losses/loss_ops.py:151: add_loss (from tensorflow.contrib.losses.python.losses.loss_ops) is deprecated and will be removed after 2016-12-30. Instructions for updating: Use tf.losses.add_loss instead. [2017-03-01 13:01:03,879 deprecation.py:116 WARNING] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/losses/python/losses/loss_ops.py:151: add_loss (from tensorflow.contrib.losses.python.losses.loss_ops) is deprecated and will be removed after 2016-12-30.

[xdvx]

Which line is for pre-outer layer?

[bakwc]

W_fc1 = tf.get_variable("W_fc1", shape=[8 * 8 * 24, 1024], initializer=xavier())