Ensure error and live_status columns are created from the start by deldesir · Pull Request #191 · iiab/calibre-web

deldesir · 2024-06-18T13:01:14Z

Makes identification of requested files more robust and logical for

Lines 91 to 98 in d8a2748

    
           # Abort if there is not a path 
        
           if not requested_file: 
        
               log.info("No path found in the database") 
        
               error = conn.execute("SELECT error, webpath FROM media WHERE error IS NOT NULL").fetchone() 
        
               if error: 
        
                   log.error("[xklb] An error occurred while trying to download %s: %s", error[1], error[0]) 
        
                   self.message = f"{error[1]} failed to download: {error[0]}" 
        
               return

holta · 2024-06-18T13:23:45Z

What YouTube playlist URL (or any such!) is this tested with?
Which issues is this PR most likely to help move forward?

Possibly related:

PR Prevent stuck videos from carrying over to next download session ["unavailable fragments" ?] #193
PR Adjust error message for stuck/slow videos [if elapsed_time >= timeout, e.g. due to unavailable fragments &/or xklb media_check] #223
PR Increase timeout to 120s [fragment_stuck_timeout = 120 # seconds] #229

deldesir · 2024-06-18T13:32:52Z

I can't find the dpaste link that prompted me to propose this. I've seen in one of the logs shared there were a case of missing error column. It was a failed video redownload attempt.

holta · 2024-06-18T13:49:14Z

What YouTube playlist URL (or any such!) is this tested with?

Clarify smoke test details when you can, Thanks!

deldesir · 2024-06-18T19:02:48Z

Premature creation of error column is not necessary anymore because videos failed without errors are removed and reprocessed (in next retry) per #193

holta · 2024-06-18T19:04:43Z

Premature creation of error column is not necessary anymore because videos failed without errors are removed and reprocessed (in next retry) per #193

Interesting, Ok!

deldesir · 2024-07-21T11:56:17Z

I reopened this after some reexamination... Error column is expected whenever a download fails.

holta · 2024-07-21T18:06:03Z

Is it possible the live_status column might also be needed in advance?

Metadata Fetch: [PLAYLIST URL] failed: An error occurred (no such column: live_status) while trying to connect to the database. #228

deldesir · 2024-07-21T21:08:05Z

Is it possible the live_status column might also be needed in advance?

Metadata Fetch: [PLAYLIST URL] failed: An error occurred (no such column: live_status) while trying to connect to the database. #228

Yes, when a playlist URL is submitted.

holta · 2024-07-21T23:28:55Z

Great news if this is tested + broadly safe, or close?

deldesir · 2024-07-22T07:47:32Z

Smoke tested on Ubuntu 24.04. Missing columns error doesn't happen anymore

screenshot: N/A

holta · 2024-07-22T13:45:45Z

Smoke tested on Ubuntu 24.04. Missing columns error doesn't happen anymore

Can you test all 3 original problem URL's to confirm... ?

(1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed #186
Formerly live https://youtu.be/4BL65HElOPg fails to download [same "no such column: error" issue as #186 ?] [unavailable fragments?] #215
Metadata Fetch: [PLAYLIST URL] failed: An error occurred (no such column: live_status) while trying to connect to the database. #228

Are there any relevant revisions or related clarifications that can now be added at... ?

Overall "Download to IIAB" workflow #190

EMG70 · 2024-07-22T21:23:28Z

SUDO IIAB-DIAGNOSTICS: https://dpaste.com/GYCDC4NQG

Tested on a new VM and ran PR#191 on it.
The following video urls were tested .
1.https://www.youtube.com/playlist?list=PLXpbdfI05JvRT07cgyZjwcfSYGiFgnbtM precious plastics
2. https://youtu.be/4BL65HElOPg mondiales( Nzolas's)
3. www.youtube.com/playlist?list=PLqxP5EuGxPncYWWc1cXlcuhfcoo3Wl3Ru The 5 Hardest GCSE Questions Series

RESULTS
-The first video failed to download as seen on task but did appear later in books as a successful download.This happened after triggering download of video 3 above and seems to join both playlist of 29+20 and appeared as downloading 49 videos?.
-The second video downloaded but had a recurring warning during download ( some fragments are taking longer than expected to download,please wait.The whole video eventually downloaded and warnings disappeared from tasks,see screenshot)

-The 3rd video also downloaded Ok as part of combined download albeit randomly (as part of 49 videos url 1 and url 2)
-Precious plastics videos did not create a shelf but split video authors into ( one army and precious plastics) 28/29 videos appear in books as one army and one as precious plastics.Total 29/29 downloaded OK though.This maybe due to owners naming of videos?

holta · 2024-07-22T21:45:48Z

@EMG70 I'm not seeing that you ran the git pull https://github.com/deldesir/calibre-web deldesir-patch-44 --no-rebase --no-edit command to install this PR — according to our usual "How do I test a pull request (PR) ?" instructions at: https://github.com/iiab/calibre-web/wiki#how-do-i-test-a-pull-request-pr-

Assuming you're trying to test this PR #191:

The branch deldesir-patch-44 should appear in your git log output, if you're in /usr/local/calibre-web-py3 — and also in your iiab-diagnostics git log output here: https://dpaste.com/GYCDC4NQG#line-1034

holta · 2024-07-23T02:41:29Z

@EMG70 I apologize:

Ansible was destructively removing all local commits, preventing testing of PR's !

I've now fixed that — so you can try again — testing PR #191 here if you have time:

PR 'git pull' if upgrading Calibre-Web, bypassing Ansible's incomplete git module iiab#3777

EMG70 · 2024-07-23T03:01:11Z

Ok will test later today.

EMG70 · 2024-07-23T15:02:48Z

A new VM was created and ran PR#191.
SUDO IIAB-DIAGNOSTICS - https://dpaste.com/FJR5AZQEW

All three video URLs downloaded successfully.
This video ( https://youtu.be/4BL65HElOPg mondiales ( Nzolas's)) of duration 1hr56mins resulted in real time repetitive messages in tasks (some fragments are taking longer than expected to download,please wait). These messages however disappeared upon successful download.Is one message for the whole video good enough or it comes per fragment?see screenshot.

Task display after video downloaded successfully.
( https://youtu.be/4BL65HElOPg mondiales ( Nzolas's))

deldesir · 2024-07-23T15:15:51Z

Smoke tested on Ubuntu 24.04. Missing columns error doesn't happen anymore

Can you test all 3 original problem URL's to confirm... ?

(1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type not in session, add operation along 'Authors.books' won't proceed #186

Formerly live https://youtu.be/4BL65HElOPg fails to download [same "no such column: error" issue as #186 ?] [unavailable fragments] #215

Metadata Fetch: [PLAYLIST URL] failed: An error occurred (no such column: live_status) while trying to connect to the database. #228

All passed (no errors), except editbooks.py: SAWarning: Object of type not in session, add operation along 'Authors.books' won't proceed #186](#186)

Are there any relevant revisions or related clarifications that can now be added at... ?

Overall "Download to IIAB" workflow #190

No, but emphasis should be put on playlists downloads because live_status addition introduced a regression due to missing column

deldesir · 2024-07-23T15:21:18Z

@EMG70, thanks for testing. The repetitive messages should have been gone.

holta · 2024-07-23T15:31:04Z

@deldesir I suggest this be merged now, and @codewiz review it quickly for possible improvements after merging, Ok?

deldesir · 2024-07-23T16:18:55Z

I agree.

holta · 2024-07-23T16:55:02Z

@codewiz can you review this PR quick — if possible?

(Many Thanks!!)

holta · 2024-07-23T17:01:53Z

Somewhat related to this PR converging quickly!

PR Adjust error message for stuck/slow videos [if elapsed_time >= timeout, e.g. due to unavailable fragments &/or xklb media_check] #223

Ensure error column is created from the start

fcd60c7

deldesir self-assigned this Jun 18, 2024

deldesir requested a review from holta June 18, 2024 13:07

deldesir mentioned this pull request Jun 18, 2024

Overall "Download to IIAB" workflow #190

Open

6 tasks

deldesir added bug Something isn't working enhancement New feature or request labels Jun 18, 2024

deldesir closed this Jun 18, 2024

deldesir deleted the deldesir-patch-44 branch July 1, 2024 21:40

deldesir restored the deldesir-patch-44 branch July 1, 2024 21:59

deldesir reopened this Jul 1, 2024

deldesir closed this Jul 1, 2024

holta mentioned this pull request Jul 1, 2024

(1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed #186

Open

deldesir reopened this Jul 21, 2024

deldesir added 2 commits July 21, 2024 07:51

Merge branch 'master' into deldesir-patch-44

420e992

Ensure error column is created

2546b4e

Add live_status if not exist

86fa47a

deldesir changed the title ~~Ensure error column is created from the start~~ Ensure error and live_status columns are created from the start Jul 21, 2024

holta mentioned this pull request Jul 23, 2024

'git pull' if upgrading Calibre-Web, bypassing Ansible's incomplete git module iiab/iiab#3777

Merged

deldesir mentioned this pull request Jul 23, 2024

Increase timeout to 120s [fragment_stuck_timeout = 120 # seconds] #229

Merged

holta merged commit 52285cf into iiab:master Jul 23, 2024

holta mentioned this pull request Jul 23, 2024

Adjust error message for stuck/slow videos [if elapsed_time >= timeout, e.g. due to unavailable fragments &/or xklb media_check] #223

Merged

holta mentioned this pull request Aug 28, 2024

Metadata Fetch: [PLAYLIST URL] failed: An error occurred (no such column: live_status) while trying to connect to the database. #228

Closed

holta mentioned this pull request Oct 10, 2024

Ensure xklb-metadata.db has live_status and error columns #265

Closed

1 task

	# Abort if there is not a path
	if not requested_file:
	log.info("No path found in the database")
	error = conn.execute("SELECT error, webpath FROM media WHERE error IS NOT NULL").fetchone()
	if error:
	log.error("[xklb] An error occurred while trying to download %s: %s", error[1], error[0])
	self.message = f"{error[1]} failed to download: {error[0]}"
	return

Uh oh!

Conversation

deldesir commented Jun 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holta commented Jun 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deldesir commented Jun 18, 2024

Uh oh!

holta commented Jun 18, 2024

Uh oh!

deldesir commented Jun 18, 2024

Uh oh!

holta commented Jun 18, 2024

Uh oh!

deldesir commented Jul 21, 2024

Uh oh!

holta commented Jul 21, 2024

Uh oh!

deldesir commented Jul 21, 2024

Uh oh!

holta commented Jul 21, 2024

Uh oh!

deldesir commented Jul 22, 2024

Uh oh!

holta commented Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EMG70 commented Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holta commented Jul 22, 2024

Uh oh!

holta commented Jul 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EMG70 commented Jul 23, 2024

Uh oh!

EMG70 commented Jul 23, 2024

Uh oh!

deldesir commented Jul 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deldesir commented Jul 23, 2024

Uh oh!

holta commented Jul 23, 2024

Uh oh!

deldesir commented Jul 23, 2024

Uh oh!

holta commented Jul 23, 2024

Uh oh!

holta commented Jul 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

deldesir commented Jun 18, 2024 •

edited

Loading

holta commented Jun 18, 2024 •

edited

Loading

holta commented Jul 22, 2024 •

edited

Loading

EMG70 commented Jul 22, 2024 •

edited

Loading

holta commented Jul 23, 2024 •

edited

Loading

deldesir commented Jul 23, 2024 •

edited

Loading