diff --git a/README.md b/README.md
index e718c46..94c58da 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,5 @@
# Memrise Course Dump
-This **Google Chrome** extension downloads word lists from [**Memrise**](https://memrise.com/) courses as ".csv" spreadsheets along with all associated images, audio, and video files. It also supports [batch download](https://github.com/Eltaurus-Lt/CourseDump2022?tab=readme-ov-file#batch-download) of Memrise courses. The format of the downloaded data is suitable for subsequent import into [**Anki**](https://apps.ankiweb.net/).
+This **Chrome** and **Firefox** extension downloads word lists from [**Memrise**](https://memrise.com/) courses as ".csv" spreadsheets along with all associated images, audio, and video files. It also supports [batch download](https://github.com/Eltaurus-Lt/CourseDump2022?tab=readme-ov-file#batch-download) of Memrise courses. The format of the downloaded data is suitable for subsequent import into [**Anki**](https://apps.ankiweb.net/).
The extension *does not* download personal study data (although it is planned to be added in the future). It also *does not* download the words you have marked as "ignored" on Memrise. You might want to unignore them before downloading a course or make a separate clean Memrise account specifically for downloading purposes.
@@ -10,7 +10,7 @@ At the top of this page click `Code` and then `Download ZIP` (Note, that the `Co
-## Installation
+## Installation (Chrome)
1. [Download](https://github.com/Eltaurus-Lt/CourseDump2022/archive/refs/heads/main.zip) the ***CourseDump2022-main.zip*** archive and extract ***CourseDump2022-main*** folder from it. At this step, you can move the extension folder to any place in your filesystem.
2. In *Google Chrome* click the `Extensions` button and then (alternatively go to the Main menu in the top right corner and click `Extensions` → `Manage Extensions`)
@@ -20,6 +20,15 @@ At the top of this page click `Code` and then `Download ZIP` (Note, that the `Co
5. (_optional_) Click the `Extensions` button from step 2 again and pin the extension to the toolbar by clicking the pin button
+## Installation (Firefox)
+1. [Download](https://github.com/Eltaurus-Lt/CourseDump2022/archive/refs/heads/main.zip) the ***CourseDump2022-main.zip*** archive and extract ***CourseDump2022-main*** folder from it. At this step, you can move the extension folder to any place in your filesystem.
+2. Install the [`web-ext`](https://extensionworkshop.com/documentation/develop/getting-started-with-web-ext/#installation-section) command line tool.
+3. In the extension directory, run `web-ext build`. This outputs `web-ext-artifacts/memrise_course_dump-9.1.zip`.
+4. Go to `about:debugging` in your browser.
+5. Switch to "This Firefox" tab
+6. Click "Load Temporary Add-on..." and select the .zip file from step 3
+7. (_optional_) Click the extensions button in the toolbar and pin the extension to the toolbar using the gear menu
+
## 💡 Downloading a Memrise Course
>---
@@ -61,7 +70,7 @@ After a download is complete, you should see the progress bar turning green:
-The downloaded files should appear in your Chrome downloads directory, in a subfolder with the name comprised of the id, name, and author of that course:
+The downloaded files should appear in your downloads directory, in a subfolder with the name comprised of the id, name, and author of that course:
@@ -265,7 +274,7 @@ Note, that decks can be nested inside each other (via drag-and-drop) to group co
5. **Learnable IDs**: Appends an additional column to the course spreadsheet containing a unique ID for each item. Can be used to manage duplicates inside Anki (if imported into the sorting field), or to cross-reference against other Memrise data downloaded separately, such as [archived mems](https://github.com/Eltaurus-Lt/MemDump)
6. **Video files**: Allows excluding video files from a download: when turned off overwrites the `Download media` setting for video files while leaving images and audio unaffected (has no effect if the `Download media` toggle is turned off)
7. **Skip media download**: Allows skipping media files during the file download phase. In contrast to the `Download media` setting, does not remove the respective columns from the spreadsheet when turned off. It can be helpful if a course spreadsheet needs to be recompiled with different settings without downloading the whole media folder again
-8. **Course metadata**: Enables downloading three metadata files in addition to the basic spreadsheet and media: an `info.md` file containing the text description of a course, the course's thumbnail image, and the course author's avatar. When turned off, the ".csv" spreadsheet and respective media folder (when applicable) will be placed directly into the Chrome download folder, instead of being bundled together with meta files in a separate course folder
+8. **Course metadata**: Enables downloading three metadata files in addition to the basic spreadsheet and media: an `info.md` file containing the text description of a course, the course's thumbnail image, and the course author's avatar. When turned off, the ".csv" spreadsheet and respective media folder (when applicable) will be placed directly into the download folder, instead of being bundled together with meta files in a separate course folder
## Discussion
If you encounter errors, have further questions regarding the extension, or need any help with using the downloaded materials in Anki, please leave a comment in this thread: [An alternative to Memrise2Anki](https://forums.ankiweb.net/t/an-alternative-to-memrise2anki-support-thread/30084)
diff --git a/background.js b/background.js
index 07781f5..933421a 100644
--- a/background.js
+++ b/background.js
@@ -91,7 +91,7 @@ function ciddsParse(cidd_strings) {
}
}
-chrome.runtime.onMessage.addListener(async (arg, sender, sendResponse) => {
+chrome.runtime.onMessage.addListener((arg, sender, sendResponse) => {
//messages from menu
if (arg.type === "coursedump_checkOngoing") {
sendResponse({ "ongoing-status": !!ongoingTab });
@@ -177,34 +177,40 @@ chrome.runtime.onMessage.addListener(async (arg, sender, sendResponse) => {
let urls = [];
if (arg.type === "coursedump_downloadFiles") {
let terminated = false;
- if (arg.file_queue) urls = arg.file_queue;
- const todo = urls.length;
- if (arg.maxThreads) maxConnections = arg.maxThreads;
- console.log(`max threads set to : ${maxConnections}`);
- let done = 0;
- let pids = Array(maxConnections).fill().map((_, i) => i + 1);
- const results = await Promise.allSettled(pids.map(async pid => {
- while (ongoingTab && urls.length) {
- // const [url, filename] = ["some url", "some filename.ext"]; //emu
+ if (arg.file_queue) urls = arg.file_queue;
+ const todo = urls.length;
+ if (arg.maxThreads) maxConnections = arg.maxThreads;
+ console.log(`max threads set to : ${maxConnections}`);
+ let done = 0;
+ let pids = Array(maxConnections).fill().map((_, i) => i + 1);
+ Promise.allSettled(pids.map(async pid => {
+ while (ongoingTab && urls.length) {
+ // const [url, filename] = ["some url", "some filename.ext"]; //emu
// const emu = urls.shift();//emu
- const [url, filename] = urls.shift();
- await sleep(200);
- let did;
- try {
- did = await downloadFile({url, filename, conflictAction: "overwrite" });
+ let [url, filename] = urls.shift();
+ if (url instanceof Blob) {
+ url = URL.createObjectURL(url);
+ }
+ await sleep(200);
+ let did;
+ try {
+ did = await downloadFile({url, filename, conflictAction: "overwrite" });
//did = await sleep(Math.floor(Math.random() * 600 + 300)); //emulate download
- } catch (err) {
- console.error(filename, err);
+ } catch (err) {
+ console.error(filename, err);
chrome.tabs.sendMessage(tabId, {
type: "coursedump_error",
error: err.message,
url, filename
}).catch(err => {});
- }
- if (did !== undefined) {
- await chrome.downloads.erase({ id: did });
- }
- done++;
+ } finally {
+ if (new URL(url).protocol === "blob:")
+ URL.revokeObjectURL(url);
+ }
+ if (did !== undefined) {
+ await chrome.downloads.erase({ id: did });
+ }
+ done++;
chrome.tabs.sendMessage(tabId, {
type: "coursedump_progressMedia_upd",
done, todo
@@ -214,37 +220,37 @@ chrome.runtime.onMessage.addListener(async (arg, sender, sendResponse) => {
console.log('Downloading tab appears to be closed. terminating file download.');
terminated = true;
ongoingTab = null;
- }
- })
- }
- }));
-
- for (let i = 0; i < results.length; i++) {
- const r = results[i];
- if (r.status === "rejected") {
- console.error(`pid ${i + 1}: ${r.reason}`);
- }
- }
-
- if (ongoingTab) {
- chrome.tabs.sendMessage(tabId, {
- type: "coursedump_mediaFinished",
- status: "done"
- }).catch(err => {});
- ongoingTab = null;
- } else {
- if (!terminated) {
- console.log('Download stopped by user during file downloading phase');
+ } })
+ }
+ })).then((results) => {
+ for (let i = 0; i < results.length; i++) {
+ const r = results[i];
+ if (r.status === "rejected") {
+ console.error(`pid ${i + 1}: ${r.reason}`);
+ }
+ }
+
+ if (ongoingTab) {
+ chrome.tabs.sendMessage(tabId, {
+ type: "coursedump_mediaFinished",
+ status: "done"
+ }).catch(err => {});
+ ongoingTab = null;
} else {
- console.log('Downloading tab was closed during file downloading phase');
- menuAlert("Downloading tab was closed. Download terminated",
- "no open menus left, download termination alert was not sent");
+ if (!terminated) {
+ console.log('Download stopped by user during file downloading phase');
+ } else {
+ console.log('Downloading tab was closed during file downloading phase');
+ menuAlert("Downloading tab was closed. Download terminated",
+ "no open menus left, download termination alert was not sent");
+ }
+ chrome.tabs.sendMessage(tabId, {
+ type: "coursedump_mediaFinished",
+ status: "stopped"
+ }).catch(err => {});
}
- chrome.tabs.sendMessage(tabId, {
- type: "coursedump_mediaFinished",
- status: "stopped"
- }).catch(err => {});
- }
- updAllMenus();
+ updAllMenus();
+ });
+ return true;
}
});
diff --git a/coursescan.js b/coursescan.js
index 768cea0..2a752f4 100644
--- a/coursescan.js
+++ b/coursescan.js
@@ -66,21 +66,19 @@ async function fetchMeta(cidd) {
}
function meta2txt(meta) {
- let text = 'data:md/plain;charset=utf-8,' + encodeURIComponent(
+ let text =
`# **${meta['proper name'] || meta['url name']}**\n` +
`### by _${meta['author']}_\n` +
`### (${meta['number of items']} learnable items)\n` +
`\n` +
- meta['description']
- );
+ meta['description'];
if (!ANKI_HEADERS) {
- text = text + encodeURIComponent(
+ text = text +
`\n\n` +
`## Course Fields\n` +
- `| ${meta['course fields']} |`
- );
+ `| ${meta['course fields']} |`;
}
- return text
+ return new Blob([text], {type: 'text/markdown'});
}
//THE MAIN FUNCTION FOR SCANNING ALL LEVELS OF A COURSE
@@ -434,7 +432,6 @@ async function scanCourse(cidd, threadN) {
"#columns:" + course_fields.join(",") + "\n" +
csv_data;
}
- const csv_encoded = 'data:text/csv;charset=utf-8,%EF%BB%BF' + encodeURIComponent(csv_data);
//names for directory and spreadsheet
let course_filename, course_folder;
@@ -451,7 +448,7 @@ async function scanCourse(cidd, threadN) {
};
//add all files to global queue
- file_queue.unshift([csv_encoded, `${course_folder}${course_filename}_(${meta['number of items'].toString()}).csv`]);
+ file_queue.unshift([new Blob([csv_data], {type: 'text/csv'}), `${course_folder}${course_filename}_(${meta['number of items'].toString()}).csv`]);
if (settings["course_metadata"]) {
file_queue.unshift([meta2txt(meta), `${course_folder}info.md`]);
file_queue.unshift([meta['ava'], `${course_folder}${meta['author']}.${meta['ava'].split(".").slice(-1)}`]);
@@ -580,11 +577,11 @@ async function batchDownload() {
//global variables
-if (typeof cidds === 'undefined') {var cidds = []} //should be defined as an argument passed from menu.js through background.js
-if (typeof batch_done === 'undefined') {var batch_done = 0} //global progress counter
-if (typeof file_queue === 'undefined') {var file_queue = []} //global list of files
+if (typeof cidds === 'undefined') {cidds = []} //should be defined as an argument passed from menu.js through background.js
+if (typeof batch_done === 'undefined') {batch_done = 0} //global progress counter
+if (typeof file_queue === 'undefined') {file_queue = []} //global list of files
if (typeof threads !== 'undefined') {
alert('Script is already executing on this page. Reload to retry'); //should be impossible to trigger if the menu state is correct
} else {
- batchDownload();
-}
\ No newline at end of file
+ batchDownload().catch(console.error);
+}
diff --git a/manifest.json b/manifest.json
index fd9713c..bc203ba 100644
--- a/manifest.json
+++ b/manifest.json
@@ -20,7 +20,8 @@
"default_popup": "menu.html"
},
"background": {
- "service_worker": "background.js"
+ "service_worker": "background.js",
+ "scripts": ["background.js"]
},
"content_security_policy": {},
"web_accessible_resources": [{
-## Installation
+## Installation (Chrome)
1. [Download](https://github.com/Eltaurus-Lt/CourseDump2022/archive/refs/heads/main.zip) the ***CourseDump2022-main.zip*** archive and extract ***CourseDump2022-main*** folder from it. At this step, you can move the extension folder to any place in your filesystem.
2. In *Google Chrome* click the `Extensions` button