AISdb marinetraffic scraper not working

[guillermodangelo]

The scraper of marinetraffic.com data by MMSI codes is not working, giving timeout error even when I can see that the MMSI exists in the Gecko driver, using the DEBUG variable set to "1".

Steps to reproduce the behavior:

import os
from aisdb.webdata.marinetraffic import VesselInfo
os.environ["DEBUG"] = "1"
output_db_filepath = 'PATH TO SQLITE DB FILE'

mmsi = [
    372351000,
    373416000,
    477003800,
    477282400,
    477519900,
    477593700,
    477831200,
    477864000,
    477905000,
    563626000,
    565715000
]

scrapper = VesselInfo(output_db_filepath)
scrapper.vessel_info_callback(mmsi)

• OS: Windows 11
• Gecko driver
• Using pip virtual environment and the last version of AISdb
• Python 3.12.3

[JayKumarr]

@guillermodangelo
Can you please provide which browsers are installed on your OS? The code uses firefox browser as default to sent request to marinetraffic.

[guillermodangelo]

@JayKumarr
I'm using Firefox 126. I also have Edge and Chrome 125.
I think the issue is related to the cookie compliance message or to the div with the add.

[gabrielspadon]

@Jinnkunn, can you please follow up on this and loop @tsuzzy into the problem?

[Jinnkunn]

@Jinnkunn, can you please follow up on this and loop @tsuzzy into the problem?

Sure, we are going to work on this

[gabrielspadon]

@guillermodangelo, it seems the issue is due to a change in the layout of the MarineTraffic that is being scrapped. @tsuzzy is working on a solution related to this and will be uploading a nightly building of AISdb to fix it.

[gabrielspadon]

Unfortunately, according to the last development iteration, MarineTraffic scraping won't be available anymore. The website now implements scraping blockers and changed its business model to sell its data on a subscription system. However, we will implement an API communication module that will interact with their new subscription system, allowing the person interested in their data to merge it with AIS data stored on AISdb.