Task 9978, classif.rpart only has 17 runs

[jakob-r]

What happened there? Accroding to [1] Table 3 it should have 6005 runs.

@mboecker Can you compare the numbers to the runs that we have in our database? If our number is smaller something is strange.

[1] Kühn et. al, Automatic Exploration of Machine Learning Experiments on OpenML, https://arxiv.org/pdf/1806.10961.pdf

[mboecker]

Task 14966, kknn also has only 28 runs in our DB but 30 in the paper.

SELECT DISTINCT rid, run.setup, implementation.name FROM run JOIN input_setting ON run.setup = input_setting.setup JOIN input ON input.id = input_setting.input_id JOIN implementation ON implementation.id = input.implementation_id WHERE task_id = 9978 AND implementation.name = "mlr.classif.rpart";

+---------+---------+-------------------+
| rid     | setup   | name              |
+---------+---------+-------------------+
| 1875185 |   47685 | mlr.classif.rpart |
| 1875182 |   47689 | mlr.classif.rpart |
| 1875187 |   47690 | mlr.classif.rpart |
| 1875189 |   47691 | mlr.classif.rpart |
| 1875188 |   47692 | mlr.classif.rpart |
| 1875186 |   47693 | mlr.classif.rpart |
| 1875191 |   47694 | mlr.classif.rpart |
| 2041106 |  181776 | mlr.classif.rpart |
| 2041108 |  181778 | mlr.classif.rpart |
| 2041112 |  181782 | mlr.classif.rpart |
| 2041116 |  181785 | mlr.classif.rpart |
| 2041124 |  181786 | mlr.classif.rpart |
| 2041122 |  181788 | mlr.classif.rpart |
| 2041123 |  181789 | mlr.classif.rpart |
| 2041127 |  181792 | mlr.classif.rpart |
| 2041129 |  181794 | mlr.classif.rpart |
| 2041142 |  181799 | mlr.classif.rpart |
| 8869850 | 6833268 | mlr.classif.rpart |
+---------+---------+-------------------+
18 rows in set (14.32 sec)

With restriction to our bot, we find only 17 results. This means the database doesnt contain any runs for this learner that has at least one parameter set in the database, other than these 18.

MariaDB [openml_native]> SELECT DISTINCT rid, run.setup, implementation.name FROM run JOIN input_setting ON run.setup = input_setting.setup JOIN input ON input.id = input_setting.input_id JOIN implementation ON implementation.id = input.implementation_id WHERE task_id = 9978 AND implementation.name LIKE "%rpart%";

81 rows in set (4.84 sec)

When I lifted the learner name from "mlr.classif.rpart" to anything containing "rpart", I got 81 different runs. When dropping the DISTINCT (thus allowing runs with the exact same parameter settings to appear more often), that number goes up to 570 rows in set (2.60 sec).

I don't know how to find the number 6005. We aren't even near it. Maybe I'm missing something?

[mboecker]

Task 14966, kknn also has only 28 runs in our DB but 30 in the paper.

I resolved this by re-inserting these values from the native database. I think I used an outdated version of the reduced.sql.gz file. Maybe you can upload a recent one?

[mboecker]

task 219, ranger, we have only 6607 runs, in the paper there are 12517 runs.
task 43 and 49, ranger, we have 13099 resp. 10484 runs, in the paper there are 15139.

I checked for task 43 and there are 15222 rows in the native database.
When using our reformatting script, there are between 7401 and 14945 runs for the 6 parameters of ranger.
The problem appears to be that the openml_exporting database doesn't contain all the runs anymore. I don't know yet what causes this.

[mboecker]

I dont think this is resolved.