How can I train your model using my own dataset

[ztw33]

Hi, sorry if I bother you, I'm trying to train your model on my own java dataset, but I don't know how to preprocess my data into your format. I download your data file data.zip, I understand that {train/valid/test}.token.code and {train/valid/test}.token.nl are the source and target files, but don't know how can I get {train/valid/test}.token.guid and adjacency for every code data. It will be very kind if you can offer some help. Look forward to your reply!

[gingasan]

In our paper, we follow the original Java dataset including the parsing. If you need to parse Java on your own, could you try this https://github.com/tree-sitter/py-tree-sitter/? It is an easy-to-use parser. I use the Python version, which can also parse Java, C, JS, etc. using Python. What you need to do is to traverse and transfer into adjacency matrices. And for you tokenization, you can directly use the tokenizer in https://github.com/wasiahmad/NeuralCodeSum. For .guid files, they are the unique ids for all code samples. We use this to facilitate loading the adjacency matrices, which are space hungry. You just need to distribute each adjacency with one id and put into corresponding .guid file and do not need to modify our training code. Best, Hongqiu

…

----- 原始邮件 ----- 发件人: "TingWei Zhu" ***@***.***> 收件人: "gingasan/sit3" ***@***.***> 抄送: "Subscribed" ***@***.***> 发送时间: 星期三, 2021年 8 月 04日 下午 4:38:21 主题: [gingasan/sit3] How can I train your model use my own dataset (#2) Hi, sorry if I bother you, I'm trying to train your model on my own java dataset, but I don't know how to preprocess my data into your format. I download your data file `data.zip`, I understand that `{train/valid/test}.token.code` and `{train/valid/test}.token.nl` are the source and target files, but don't know how can I get `{train/valid/test}.token.guid` and `adjacency` for every code data. It will be very kind if you can offer some help. Look forward to your reply!

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #2