Dependency Issue

[amine759]

• ferret version: 0.4.1
• Python version: 3.10.12
• Running on : Google Colab

Description

I'm trying to run :

bench.show_table(explanations)

and I get the following error :

AttributeError                            Traceback (most recent call last)
[<ipython-input-9-39a3816ba913>](https://localhost:8080/#) in <cell line: 1>()
----> 1 bench.show_table(explanation2)

[/usr/local/lib/python3.10/dist-packages/ferret/benchmark.py](https://localhost:8080/#) in show_table(self, explanations, apply_style, remove_first_last)
    396             table.columns = pd.io.parsers.base_parser.ParserBase(
    397                 {"names": table.columns, "usecols": None}
--> 398             )._maybe_dedup_names(table.columns)
    399 
    400         return (

AttributeError: 'ParserBase' object has no attribute '_maybe_dedup_names'

This only happens when I have duplicate tokens in my text example which I'm willing to explain, apparently in The latest version of pandas (2.2.2) _maybe_dedup_names is deprecated, probably to _maybe_make_multi_index_columns.
I have tried to downgrade pandas yet new errors occurs, Can you provide me with the pandas version used here ?

Thanks.

[g8a9]

Hey, thank you for reaching out. Yes indeed, it seems related to the deprecation of that pandas method. I believe the best way here is to rename duplicated columns (tokens) ourselves (i.e., not relying on pandas for that) -- but I would include the change in a new library release.

Before doing that, can you share the google colab or snippet of code that crashes on your side?

[amine759]

Hi @g8a9, Thanks for your reply.
This code is just from the doc but text example has duplicate tokens :

from transformers import AutoModelForSequenceClassification, AutoTokenizer
from ferret import Benchmark

name = "cardiffnlp/twitter-xlm-roberta-base-sentiment"
model = AutoModelForSequenceClassification.from_pretrained(name)
tokenizer = AutoTokenizer.from_pretrained(name)

bench = Benchmark(model, tokenizer)

explanations = bench.explain("You look stunning stunning !", target=1)
evaluations = bench.evaluate_explanations(explanations, target=1)

bench.show_table(explanations)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-3-1ca62c9897f7>](https://localhost:8080/#) in <cell line: 1>()
----> 1 bench.show_table(explanations)

[/usr/local/lib/python3.10/dist-packages/ferret/benchmark.py](https://localhost:8080/#) in show_table(self, explanations, apply_style, remove_first_last)
    396             table.columns = pd.io.parsers.base_parser.ParserBase(
    397                 {"names": table.columns, "usecols": None}
--> 398             )._maybe_dedup_names(table.columns)
    399 
    400         return (

AttributeError: 'ParserBase' object has no attribute '_maybe_dedup_names'

[melanchthon19]

Hi @g8a9! I am having the same error. I am using:

• ferret-xai 0.4.2
• pandas 2.0.3
• python3.10
• macOS

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[8], line 2
      1 explanations = bench.explain(text, target=0)
----> 2 bench.show_table(explanations)

File [~/miniconda3/envs/ferret/lib/python3.10/site-packages/ferret/benchmark.py:398](http://localhost:8889/lab/tree/~/miniconda3/envs/ferret/lib/python3.10/site-packages/ferret/benchmark.py#line=397), in Benchmark.show_table(self, explanations, apply_style, remove_first_last)
    394 # Rename duplicate columns (tokens) by adding a suffix
    395 if sum(table.columns.duplicated().astype(int)) > 0:
    396     table.columns = pd.io.parsers.base_parser.ParserBase(
    397         {"names": table.columns, "usecols": None}
--> 398     )._maybe_dedup_names(table.columns)
    400 return (
    401     table.style.background_gradient(
    402         axis=1, cmap=SCORES_PALETTE, vmin=-1, vmax=1
   (...)
    405     else table.style.format("{:.2f}")
    406 )

AttributeError: 'ParserBase' object has no attribute '_maybe_dedup_names'

Is there any workaround?

Thank you!

[melanchthon19]

Quick solution changing benchmark.py around lines 398. Added try except block: it falls back to manual deduplication.

# Rename duplicate columns (tokens) by adding a suffix
if sum(table.columns.duplicated().astype(int)) > 0:
    try: # original source code (line ~398 benchmark.py)
        table.columns = pd.io.parsers.base_parser.ParserBase(
            {"names": table.columns, "usecols": None}
        )._maybe_dedup_names(table.columns)
    except Exception: # ugly solution
        from collections import defaultdict
        def _dedup_cols(cols):
            counts = defaultdict(int)
            out = []
            for c in cols:
                counts[c] += 1
                out.append(c if counts[c] == 1 else f"{c}.{counts[c]-1}")
            return out

        table.columns = _dedup_cols(list(table.columns))