ERROR "std::bad_array_new_length"

[Alex-cdea]

Hi,

I am currently trying to run cafe on my contingency table of wasp gene families using this command:

$ cafe5 -i my_contingency_table.txt.large -t my_tree.treefile -l 0.042503863579858 -eerror_model.txt -o large_results -c 15 -I 100000

After a few hours, the program stops with this error:

terminate called after throwing an instance of 'std::bad_array_new_length'
    what():  std::bad_array_new_length}

To generate my_contingency_table.txt.large, I used an R script that I wrote based on what clade_and_size_filter.py does. The R script filters families based on:

• the number of genes (g) per family (if any, for a family, any species of my analyses have a g > 100, the family is removed)
• the number of species (s) having orthologous copies per family (if s in < 30, the family is removed)
• the deviation quantile (q) to which the family belongs (if q > 99, the family is removed)

My scripts generates two contingency tables, one filtered and one large (= initial contingency table - filtered).
CAFE works fine on the filtered one (I used this one to predict lambda and the error_model), but I would like to analyze the rest of my dataset.

I saw that this error could come from C++ implemented limits. I then guessed that my large table is too big (?) (the large one is 3.2M and has 22155 rows x 69 columns, the filtered one is 1.2M and has 7889 x 69 fields), or that, at some point, the program generates an object with too many characters (?).

Is there a way to bypass this issue so I can run the analysis on the entire dataset?
Many thanks.

here is a sample of my large table:

Desc	Orthogroup	species_1	species_2	species_3	species_4	species_5	species_6	species_7	species_8	species_9	species_10	species_11	species_12	species_13	species_14	species_15	species_16	species_17	species_18	species_19	species_20	species_21	species_22	species_23	species_24	species_25	species_26	species_27	species_28	species_29	species_30	species_31	species_32	species_33	species_34	species_35	species_36	species_37	species_38	species_39	species_40	species_41	species_42	species_43	species_44	species_45	species_46	species_47	species_48	species_49	species_50	species_51	species_52	species_53	species_54	species_55	species_56	species_57	species_58	species_59	species_60	species_61	species_62	species_63	species_64	species_65	species_66	species_67
(null)	OG0000000	8	9	98	1016	8	120	246	122	93	206	216	22	123	175	27	117	163	182	247	280	5	384	439	3	221	198	2769	539	7	444	105	513	151	404	161	111	152	173	55	5	398	110	154	97	8	1260	120	75	29	47	8	375	3	70	12	23	44	99	736	126	175	91	282	132	99	38	89
(null)	OG0000001	13	14	82	262	14	16	77	67	45	81	132	3	94	173	11	25	17	62	110	107	0	311	81	0	231	158	2329	220	19	79	69	94	28	762	75	45	75	78	9	0	131	42	69	8	17	1162	57	15	51	34	10	106	2	51	9	49	26	58	332	57	69	48	155	80	30	28	73
(null)	OG0000002	13	14	69	70	18	67	40	8	19	151	67	6	140	2	33	10	53	66	85	95	20	23	53	61	637	20	2420	199	11	23	35	31	10	173	34	9	15	34	32	7	39	28	113	133	15	208	16	18	18	27	11	39	28	26	6	17	16	11	86	32	5	2	33	20	26	11	35
(null)	OG0000003	0	0	102	216	0	62	62	14	28	157	17	1	12	23	24	17	42	22	152	30	0	35	52	0	82	40	1920	62	0	134	74	257	27	143	31	163	27	56	31	0	237	43	27	7	1	328	0	47	0	4	1	33	6	148	0	0	7	10	215	57	58	45	229	48	7	2	25
(null)	OG0000004	2	3	58	57	3	8	19	26	57	30	13	4	18	47	9	4	17	26	34	78	1	192	30	3	139	46	1211	169	2	108	5	38	19	240	51	31	28	47	5	2	61	45	37	8	1	1078	7	4	9	19	3	20	8	19	4	3	11	15	101	17	28	45	72	50	24	21	53
(null)	OG0000005	0	0	45	158	0	30	147	40	15	107	42	20	66	87	11	67	63	61	47	141	1	158	147	0	107	77	792	48	0	91	69	99	25	84	20	27	19	22	12	1	29	31	195	58	0	285	111	20	19	16	0	89	1	8	3	23	40	32	187	24	71	57	72	34	30	17	26
(null)	OG0000006	2	4	26	102	3	44	28	8	20	98	49	2	15	59	5	10	34	13	56	84	2	100	57	0	189	45	643	100	6	15	7	33	21	225	39	34	52	48	21	6	31	18	49	8	2	498	13	2	9	14	1	32	1	32	6	6	22	27	102	48	38	53	112	67	44	6	78
(null)	OG0000007	2	3	28	232	3	24	47	43	6	187	133	12	31	8	10	26	107	41	162	35	0	91	5	0	71	29	283	68	4	1	47	101	43	192	75	11	56	69	5	0	56	23	36	4	2	298	7	4	6	45	0	30	0	32	1	5	47	20	9	3	9	25	66	36	68	1	43
(null)	OG0000008	0	0	14	478	0	6	14	15	0	306	69	5	0	0	4	9	192	15	253	37	0	83	0	0	4	116	384	84	1	0	33	177	7	70	16	8	15	39	0	0	101	9	108	0	0	23	0	0	0	9	0	1	0	14	0	0	0	0	0	0	7	5	29	22	37	0	11

here of the filtered table:

Desc	Orthogroup	species_1	species_2	species_3	species_4	species_5	species_6	species_7	species_8	species_9	species_10	species_11	species_12	species_13	species_14	species_15	species_16	species_17	species_18	species_19	species_20	species_21	species_22	species_23	species_24	species_25	species_26	species_27	species_28	species_29	species_30	species_31	species_32	species_33	species_34	species_35	species_36	species_37	species_38	species_39	species_40	species_41	species_42	species_43	species_44	species_45	species_46	species_47	species_48	species_49	species_50	species_51	species_52	species_53	species_54	species_55	species_56	species_57	species_58	species_59	species_60	species_61	species_62	species_63	species_64	species_65	species_66	species_67
(null)	OG0000044	14	15	16	19	15	12	18	14	14	13	14	16	13	9	11	12	11	14	19	15	20	19	18	13	15	17	23	25	14	11	9	12	22	15	14	16	10	11	12	16	12	13	15	16	15	15	14	13	14	13	15	18	20	14	15	15	14	11	18	22	18	23	18	18	22	16	18
(null)	OG0000066	10	13	13	10	14	9	10	10	10	11	10	13	14	12	13	10	11	14	11	16	10	11	9	8	10	11	23	10	16	9	11	9	10	13	13	10	11	13	10	9	9	9	14	9	13	13	8	9	9	12	8	11	20	9	11	10	12	14	10	10	14	14	17	14	26	10	18
(null)	OG0000071	11	7	11	11	11	6	13	11	10	5	9	8	11	11	6	6	8	8	10	8	17	12	16	10	11	10	18	20	7	11	6	11	21	13	9	12	7	6	11	11	9	10	10	12	11	11	10	10	11	11	12	14	16	11	14	12	14	7	14	20	12	13	17	16	22	11	13
(null)	OG0000073	9	8	12	12	8	6	9	4	10	10	8	6	10	6	8	10	10	9	6	11	13	14	12	9	13	9	12	16	12	11	11	9	14	16	14	15	14	14	10	5	4	8	15	6	9	14	9	11	13	14	13	16	16	14	17	11	13	9	13	15	15	13	16	16	17	10	15
(null)	OG0000077	10	13	13	12	11	4	10	8	13	11	12	12	9	15	6	12	6	11	10	16	18	15	12	12	12	7	10	12	14	8	11	5	10	11	13	19	13	14	10	13	8	7	15	5	8	12	11	7	12	9	12	14	13	10	16	12	6	15	14	17	5	9	14	7	6	11	12
(null)	OG0000083	9	11	9	10	8	7	11	8	9	10	8	9	9	8	8	9	8	10	8	6	12	9	14	11	11	8	11	13	10	10	10	8	10	12	11	8	13	12	10	9	9	9	9	8	7	11	7	9	10	8	7	11	12	7	14	7	12	10	14	14	10	9	13	10	12	10	16
(null)	OG0000085	8	9	8	8	8	5	10	8	8	13	7	11	6	13	6	12	12	10	17	13	9	10	9	8	8	9	9	9	9	8	10	10	20	18	17	11	17	17	7	7	9	8	8	7	9	8	6	9	7	20	7	9	12	6	8	8	6	7	9	9	9	9	9	8	9	6	7
(null)	OG0000086	8	6	6	9	9	5	10	9	7	7	10	7	8	7	9	5	7	7	7	11	9	13	16	6	12	13	9	28	8	12	15	11	6	4	6	12	4	5	13	9	13	8	9	15	9	9	9	13	6	8	7	9	8	11	11	8	8	8	14	9	17	10	14	9	16	8	10
(null)	OG0000091	9	9	13	7	8	7	9	7	8	4	8	7	6	6	7	6	8	5	12	9	13	10	10	9	7	8	19	18	9	9	7	8	13	8	7	7	7	9	10	7	7	8	10	7	9	7	7	9	9	9	9	9	15	9	10	9	11	7	13	13	9	13	13	10	15	9	11

and my ultrametric tree:

(((((((species_12:0.07998626016,species_14:0.07998626016):0.4624314192,species_16:0.5424176793):0.5647638348,(species_18:0.689690748,species_31:0.689690748):0.4174907661):0.7911694434,((species_32:0.40202424,species_41:0.40202424):1.039183753,species_42:1.441207993):0.4571429643):0.3858169317,((species_22:0.01159417542,species_26:0.01159417542):1.961816636,species_27:1.973410812):0.3107570772):0.09022929699,((((((species_3:0.4425245055,species_28:0.4425245055):0.2075305276,species_36:0.650055033):0.05775391428,species_43:0.7078089471):0.09177996102,species_25:0.7995889083):0.9214960803,(((species_6:1.122330686,species_15:1.122330686):0.1478515293,((((((species_21:0.00762437382,species_24:0.00762437382):0.0541073628,species_40:0.06173173662):0.4363333674,(((((species_23:0.006103769931,species_48:0.006103769931):0.01558277119,species_39:0.02168654112):0.05445593886,species_59:0.07614247998):0.007981046856,species_30:0.08412352683):0.01825725381,species_44:0.1023807806):0.3956843232):0.09236598234,species_60:0.5904310863):0.1311257495,(species_61:0.2780961649,(species_62:0.2015952099,(species_63:0.05736204942,species_64:0.05736204942):0.1442331605):0.07650095496):0.4434606708):0.1377098491,species_46:0.8592666849):0.4109155302):0.1975977259,(species_8:0.1156454902,species_7:0.1156454902):1.352134451):0.2533050477):0.1370955896,(((species_11:0.2840388402,species_52:0.2840388402):0.2701977268,species_58:0.5542365669):0.2402756174,((((species_51:0.1244649365,species_55:0.1244649365):0.09871408716,species_54:0.2231790237):0.07202447364,species_57:0.2952034973):0.2182951293,(((species_9:0.07857259689,species_13:0.07857259689):0.1262910069,((species_47:0.008928888495,species_56:0.008928888495):0.03303176775,species_49:0.04196065623):0.1629029476):0.05875017462,((species_2:0.008997544947,species_5:0.008997544947):0.03263521971,(species_29:0.03532415673,(species_1:0.005644443516,species_45:0.005644443516):0.02967971321):0.006308607942):0.2219810138):0.2498848482):0.2810135578):1.063668394):0.5162166081):0.6256028139,((((species_10:0.17930573,species_17:0.17930573):0.1721043189,species_19:0.351410049):0.9287024865,(((species_34:0.01514369796,species_37:0.01514369796):0.0116520413,species_38:0.02679573926):0.004231996053,species_35:0.03102773532):1.2490848):0.2415841681,species_50:1.521696703):1.478303297);

[benfulton]

Generally the number of rows isn't a concern, although 22K is an awful lot. I would be more concerned about a peculiar number showing up in your data - a species count of -1, say, or a number in the billions. What is the largest family size in your large table?

[Alex-cdea]

Hi,
Sorry for the late reply, I did have some "odd" families (huge families for which all copies were present in only one or two species). I did not find negative numbers and the largest total family size was 11944 copies.

After filtering out these odd families (my largest family size is 5005), I still have the same issue. Should I filter out families whose family size is above a certain threshold? If so, what should this threshold be to prevent this issue?
Many thanks.