Why not copy the directory "src/" in Dockerfile

[dearchill]

When I use docker run command in an interactive way, I can't prepare data or train the model for the reason "python3: can't open file 'src/train_rnn.py': [Errno 2] No such file or directory". And in the image directory I can't find prepare_ddi.sh and pipeline_ddi.sh. So maybe there's any better choice to run the docker image you provided, could you please offer any instructions specifically? I'm very first to docker and thank's a lot!

[AndreLamurias]

Hi Yachao, I think you can solve this issue if you first clone the repository to your machine and then mount the repository directory using the -v option, so something like:

docker run -v /path/to/BOLSTM/:/BOLSTM bolstm bash

then you can run all the scripts inside /BOLSTM. let me know if that works.

[dearchill]

@AndreLamurias Thanks for your quick reply! I can move the repository's scrips to the image now, but I can't find a proper way to run it. I means if I use "docker run -v /path/to/BOLSTM/:/BOLSTM bolstm bash" command then a container is created but unfortunately it exits anyway. I've seen this with "docker ps -a". I search the google and find the reason's there's already a bash file in the image. Another way is an interactive mode "docker run -it -v /path/to/BOLSTM/:/BOLSTM bolstm bash", then I can run all the scripts but when I exit the bash, all changes are gone. I don't know how to do now. Do you have any other solutions?

[AndreLamurias]

Sorry, I forgot to include the -it option. The changes you make inside /BOLSTM should persist. What are you loosing? I guess another option would be to follow the instructions of the Dockerfile and configure your system manually without docker

[mjlorenzo305]

Hi Andre, @AndreLamurias
Firstly, great work on your paper, I find your method and approach very interesting and would like to see if I can get your BOLSTM model to run.
I too have run into some issue with the docker image.
As @dearchill pointed out, the source is not contained within the docker image. Instead of mounting to the local filesystem I simply did a GIT clone on the repo from within the docker image to pull that down. I then ran into missing dependency issue with DiShIn missing, so I located the repo and cloned that using GIT clone from within the docker image. I then hit issues with the load_chebi and locating the chebi obo file. I could ftp it that as well directly from the source (ftp://ftp.ebi.ac.uk/pub/databases/chebi/ontology/chebi.obo).
I seem to continue to run into issues in getting the chebi obo to generate the chebi.db and the pickled version used by your code to load during train.sh (or train_rn.py).
I think it would help to have some of these dependencies included in the docker image.
Next I am going to try to just set it up locally on my machine without docker since the docker is not really serving it's purpose. Let me know if you have any advice or recommendations.
Thanks!

[dearchill]

@AndreLamurias Hi Andre, now I can save the changes using docker commit command, and I'm confronted with the same issue with @mjlorenzo305 , I can't generate the chebi.db with "python dishin.py" in the docker or locally. Looking for your reply and solutions! Thank you!

[AndreLamurias]

@dearchill @mjlorenzo305 I am working on an updated Dockerfile to include all the dependencies, should be ready next week

[mjlorenzo305]

@AndreLamurias (@dearchill ) ,
That's great news. I've made some progress towards getting deep into the model training stage but I am hitting an issue that may actually be a Keras/Tensorflow bug. But first let me list some of the dependencies I've had to track down in order to make progress that ideally could be included within the new docker image:

• DIShin
• Obonet
• fuzzy wuzzy
• Gensim
• ssi_light
• spacy and en_core_web_sm pretrained embedding matrix
• sem eval DDI task 9 corpus data (drug bank and medline)
• pubmed w2v binary

Also, I noticed the code expects certain directories to be precreated (temp, models, data, etc) or it will fail during preprocessing or training.

Hope this is helpful

[mjlorenzo305]

I am currently hitting an issue (as mentioned above) when in the middle of training the model. The error looks like this: AttributeError: 'ProgbarLogger' object has no attribute 'log_values'

I have searched around and see others have hit this problem but I haven't found any solution yet.

Here is the trace back console output:
Total params: 505,162,683
Trainable params: 34,821,483
Non-trainable params: 470,341,200

---

None
Train on 0 samples, validate on 0 samples
Epoch 1/100
Traceback (most recent call last):
File "src/train_rnn.py", line 809, in
main()
File "src/train_rnn.py", line 547, in main
train(sys.argv[3], sys.argv[4:], train_inputs, id_to_index)
File "src/train_rnn.py", line 416, in train
batch_size=batch_size, verbose=2, callbacks=[metrics, checkpointer])
File "C:\Users\Mario\Anaconda3\envs\DL4D\lib\site-packages\keras\engine\training.py", line 1039, in fit
validation_steps=validation_steps)
File "C:\Users\Mario\Anaconda3\envs\DL4D\lib\site-packages\keras\engine\training_arrays.py", line 217, in fit_loop
callbacks.on_epoch_end(epoch, epoch_logs)
File "C:\Users\Mario\Anaconda3\envs\DL4D\lib\site-packages\keras\callbacks.py", line 79, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "C:\Users\Mario\Anaconda3\envs\DL4D\lib\site-packages\keras\callbacks.py", line 338, in on_epoch_end
self.progbar.update(self.seen, self.log_values)
AttributeError: 'ProgbarLogger' object has no attribute 'log_values'
Saving chebi dictionary...!

[AndreLamurias]

@mjlorenzo305 from the error log you seem to be using anaconda, which I'm really not familiar with, so I can't help you with that, although it's probably something to do with the keras/tensorflow versions.

I have updated Dockerfile and Dockerfile_gpu to include all the dependencies. It will download a patched version of sst-light, the pubmed-w2v binary and all the other dependencies you mentioned, except the DDICorpus, as the authors require a form submission to download it.

In the future I may update the repo again to make the image smaller, because as it is now it's too big to upload to dockerhub (7GB). But I tested building the image and running the commands inside the container and it seemed to work. I think this is the best way to do it, otherwise you will have issues with the package versions. You probably will want to use the -v option to mount some directories such as results/ and models/. But do not mount src/ as it will overwrite what is already in the container.

@mjlorenzo305 @dearchill let me now if the new Dockerfile is working for you.

[AndreLamurias]

Also, I noticed the code expects certain directories to be precreated (temp, models, data, etc) or it will fail during preprocessing or training.

this usually shouldn't be an issue, but you can add some lines to create the dirs if they don't exist, the paths are at the top of the script files, should be simple and you can then do a pull request

[mjlorenzo305]

@mjlorenzo305 from the error log you seem to be using anaconda, which I'm really not familiar with, so I can't help you with that, although it's probably something to do with the keras/tensorflow versions.
I have updated Dockerfile and Dockerfile_gpu to include all the dependencies. It will download a patched version of sst-light, the pubmed-w2v binary and all the other dependencies you mentioned, except the DDICorpus, as the authors require a form submission to download it.
In the future I may update the repo again to make the image smaller, because as it is now it's too big to upload to dockerhub (7GB). But I tested building the image and running the commands inside the container and it seemed to work. I think this is the best way to do it, otherwise you will have issues with the package versions. You probably will want to use the -v option to mount some directories such as results/ and models/. But do not mount src/ as it will overwrite what is already in the container.
@mjlorenzo305 @dearchill let me now if the new Dockerfile is working for you.
Great, I'll try out the new image today and let you know how it goes. Thanks!

[mjlorenzo305]

I was able to get beyond the issue I reported above: "AttributeError: 'ProgbarLogger' object has no attribute 'log_values'" as well as a few others that followed. I'll mention a few things here in case others run into those issues.

1. The issue with AttributeError is very misleading but it has to do with attempting to train the model and sending a batch with no data. This means that the previous "preprocessing" step completed but didn't actually produce the correct numpy arrays.
2. I found that you need to use chebi.obo (not chebi_lite) because the lite version include 0 synonyms which are required by the preprocessing step. Additional complication is that the chebi.obo file is in UTF-8 format which will produce a "character map" / encoding issue when you attempt to load the file using the obonet parser. I ended up having to make a change to the obonet code to pass in the "encoding='utf-8'" since they did not expose in their API a way to change the encoding params used. Note that for some reason "chebi_lite.obo" is actually in standard python encoding so it did not hit this issue.
3. In addition to the chebi.obo file being loaded correctly, you also need the chebi.db semantic base file produced from the DiShIn code. Fortunately I noticed that Adres already provides the chebi.db semantic base file available here: http://labs.rd.ciencias.ulisboa.pt/dishin/chebi201907.db.gz
4. The SST_light must be compiled for your local machine/hardware ... I was on a windows machine initially so I used cygwin with the gcc + make tools installed but I ended up having to make several changes to the .cpp and .h files which I can elaborate on further but inshort a few changes included specifying explicitly "#include " since new versions of gcc no longer include cstring automatically. I also hit issues with the initializer list in the constructors where variables that started with "_" (underscore) were flagged with erros and simply renaming them without the underscore got around those issues.
5. In order for SST to work I also had to add the SST bin to the path env variable as well as the Cygwin/bin path to resolve cygwin1.dll among other files. I also had to uncompress/extract the model files within the sst "MODEL" directory (which are gzipped by default) but produces no file found errors if you don't extract the files.
6. Lastly I am now able to get much further into the preprocessing steps but currently having issue where the code forks off processes invoking the SST parser on some sentence{}.txt files and produces tagged files. Unfortunately it seems to hang after completing a few of these. I'll include a snippet of where I got stuck here in case @AndreLamurias has ideas.
   Note all of the above was NOT using the latest docker image posted by Andre this week but rather trying to standup the code manually just using git hub source.

---

Trace output:
tagger_light::bitag(
modelname_pos: /Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/sst-light-0.4/MODELS/WSJPOSc_base_20
tagsetname_pos: /Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/sst-light-0.4/DATA/WSJPOSc.TAGSET
modelname: /Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/sst-light-0.4/MODELS/SEM07_base_12
targetname: /Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/temp/sentences_1.txt
tagset: /Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/sst-light-0.4/DATA/WNSS_07.TAGSET
lowercase: 0
output: /Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/temp/sentences_1.txt.tags )
load_words_notag(/Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/temp/sentences_1.txt) |D| = 33 |ID| = 33
tagger_light.init(/Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/sst-light-0.4/MODELS/WSJPOSc_base_20,/Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/sst-light-0.4/DATA/WSJPOSc.TAGSET) |Y| = 45/45
LoadModel(/Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/sst-light-0.4/MODELS/WSJPOSc_base_20).... OK
tagger_light.init(/Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/sst-light-0.4/MODELS/SEM07_base_12,/Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/sst-light-0.4/DATA/WNSS_07.TAGSET) |Y| = 93/93
LoadModel(/Users/Mario/Desktop/BOLSTM-master/BOLSTM-master/sst-light-0.4/MODELS/SEM07_base_12).... OKtagging:
.