Validate with Existing Model Trained on Both Classes

[rbunn80110]

This is really great work! I noticed you appear to be using this code for creating new examples of pathology images. I am doing something similar, just not pathology. Let's say I already have a separate model trained to classify these types of images and I want to run it against a validation set every epoch and determine how well the cyclegan is performing in generating new examples that fool an existing classifier. I'm trying to figure out where the best place might be to add code which does that. I can see a few places where it would probably work, but was curious if you have already thought about adding this functionality to monitor training progress?

Thanks,

Bob

[tmabraham]

Thank you for your interest!

Let me clarify one thing, are you trying to classify how well CycleGAN-generated images fools an existing model that classifies between real and fake images?

I have done something similar, but at the end of training.

[rbunn80110]

I currently have a model that classifies images reasonably well on the two classes being fed to the GAN (in addition to other classes in my case). I'm just curious to watch stats on how well it can create fake images that are correctly classified by my existing model. Not interested in predicting real or fake. Just generating a bunch of fake images from both classes then seeing how well those fake images work on an existing classifier.

[tmabraham]

Okay. This may be a little hard to do with the current version of the library. I plan to add metrics soon (#3) and maybe it's possible to treat this as a metric? But the problem is that you will be having separate ground-truth labels, which makes it complicated, because the DataLoader is set up such that the domain A image is the input, and the domain B image is the label.

I think a good alternative might be to create a separate callback. In this callback, you can have a new DataLoader with your images and corresponding ground-truth labels, pass it into the generator and the classifier, and calculate the accuracy. I think once I finish #3, then you could add the calculated accuracy as a metric for it to show up as part of the progress bar.

Sorry that this is a little bit hard because I hadn't thought about this particular use case. But you can do virtually anything you think of with callbacks so hopefully something can be done there.

Let me know if you have any questions!

[rbunn80110]

Sounds cool! I'll definitely try to do that once you are done with #3. Thanks!