Site download

[Andersama]

I know that more than likely you're working towards this, so far it seems publishing works to whatever extent it currently does. However the button that hints at exporting the site https://github.com/palacms/palacms/blob/f98027d9315e5098e84ed73d593865000382a9d5/app/routes/dashboard/sites/%2Bpage.svelte#L63

Is still unimplemented, I'd suggest after having read / seen the issue about downloading the site with images that potentially two versions of this button are made. One which offloads images as before etc..., another which links images locally.

However for the moment, I'd suggest at least implementing one of the two so that sites can be exported.

[Andersama]

Looking at the code for pocketbase and previewing a site, it looks like something like this might be needed

function get_slugged(record, records) {
	let p = record.get('parent')
	let s = record.get('site')
	if (p != null) {
		for (var r of records) {
			if (p == r.get('id') && s = r.get('site')) {
				return get_slugged(r,records) + record.get('slug')
			}
		}
		return '/' + record.get('slug')
	}
	return '/'
}

routerAdd('GET', '/_download/{site}', (e) => {
	if (!e.request) {
		throw new Error('No request')
	}

	const siteId = e.request.pathValue('site')
	let site
	try {
		site = $app.findRecordById('sites', siteId)
	} catch {
		throw new NotFoundError('Site not found')
	}

	// Respond with compiled HTML
	//const fileKey = site.baseFilesPath() + '/' + site.get('preview')
	const sitename = site.get('name')
    //get all pages with site
	let pages
	try {
		pages = $app.findAllRecords("pages",
        	$dbx.exp("LOWER(site) = {:site}", {"site": site})) //urls do not care about casing should assume lowercased*
    ) catch {
		throw new NotFoundError('No pages found')
	}

	//get all symbols with site
	let symbols
	try {
		symbols = $app.findAllRecords("site_symbols",
			$dbx.exp("LOWER(site) = {:site}", {"site": site})) //urls do not care about casing should assume lowercased*
	) catch {
    	throw new NotFoundError('No pages found')
	}

	let site_uploads
	let site_upload
	try  {
		site_uploads = $app.findRecordsByFilter('site_uploads', `site_uploads.site = {:host} && file = {:file}`, 'id', 1, 0, { host, slug: finalSlug || null })
		site_upload = site_uploads[0]
	} catch {
		site_upload = null
	}
	
	let fsys, reader, content
	try {
		fsys = $app.newFilesystem()
		
		if (site_upload && site_uplaod.get('updated') > site.get('updated')) { //cached since
			const filekey = site_upload.baseFilesPath() + '/' + site_upload.get('file')
			reader = fsys.getReader(filekey)
			content = toString(reader)
			return e.blob(200, 'application/zip', content)
		} else {
			const archive = archiver('zip', {
			  zlib: { level: 9 } // Sets the compression level.
			});

			//const output = fs.createWriteStream(site.baseFilesPath() + '/' + '_zipped_' + sitename + '.zip')
			//let output = fsys.fileFromPath()
			archive.pipe(output)
			
			for (page in pages) {
				const fileKey = page.baseFilesPath() + '/' + page.get('compiled_html')
				let reader = fsys.getReader(fileKey)
				archive.append(reader, {name: get_slugged(page,pages)+'index.html'})
				//content = toString(reader)
			}
			
			for (symbol in symbols) {
				const fileKey = symbol.baseFilesPath() + '/' + symbol.get('compiled_js')
				let reader = fsys.getReader(fileKey)
				archive.append(reader, {name: '_symbols/' + symbol.get('id') + '.js'})
				//content = toString(reader)
			}
			
			$app.save(record)
			//let reader = fsys.getReader()
		}

		return e.blob(200, 'application/zip', content)
	} catch {
		return e.string(404, 'Preview not found')
	} finally {
		reader?.close()
		fsys?.close()
	}
})

[joas8211]

@Andersama, what do you need site exporting for exactly? Can you comment about your use case for it.

We will definitely implement publishing to external services, but it's not a priority for the release of v3. Also exporting the site in built form could be an option at some point. I'm just thinking about prioritization currently that we can get 3.0 over the finish line. We could remove the button for now, so that it won't cause confusion. @matthewmateo, what do you think?

Looking at the code for pocketbase and previewing a site, it looks like something like this might be needed

Something like this could work, but I'd rather have it in the frontend application just to reduce risks and bloat on the backend.

[Andersama]

@joas8211 for me it's the more universal approach for deployment. The way I see it, you're essentially taking the result of the svelte compilation and choosing where to send it. I would imagine setting up a system to just download a resulting .zip would be simplest to achieve first.

From what I can see, personally I like the approach of using pocketbase's pb_public for statically hosting source files to do the work of hosting on top of the cms, that's pretty clever. I'm actually more invested in the approach of just having the cms for editing purposes, so exporting is much more important than any platform integration. Long story short, I don't need a "publish button", although I imagine that'd be convenient for other people. Unfortunately something has gone horribly wrong with my previous instance that I was hosting which was running a much older version. If the download button functioned it'd at least be functional enough for me to work with.

Something like this could work, but I'd rather have it in the frontend application just to reduce risks and bloat on the backend.

I'm not sure how you'd accomplish this via the frontend? It seems like the current code has the svelte compilation tied with inputs being provided to pocketbase. I figured since the current code grabs relevant compiled_html and compiled_js the quickest way would be to walk all the pages and sources and recreate the same folder structure as the svelte compilation.

[joas8211]

@Andersama, do you mean that you would like to automate the deployment yourself by using an endpoint for downloading the site as a zip archive? Just making sure I'm following. Allowing integration to site publishing workflows is something I'd like to support for sure. It's still in consideration if they would be plugins for PalaCMS or better yet using services that are notified and provided APIs for to allow them to do their thing, like what you are basically referring to.

I would like to just generate the site as static files into filesystem used by Pocketbase (whether actual FS or S3). That would allow a bunch more integrations more easily (eg. shared Docker volumes, S3 hosting). But Pocketbase file API does not allow it directly, because it does manage the files more closely (all files belong to specific collections-record combination that is reflected by the file path and identified is append to the file name).

If the site structure would exist in the file system, would that satisfy your requirements for the way you want to serve / publish your sites?

[Andersama]

@joas8211 I was after specifically just being able to generate the static files and have them downloadable*

Bonus points for just having the underlying structure available in the filesystem, that in theory would be much faster.

What I was after, or doing on my own branch that I can't test...unfortunately, is having a url available such that I could literally hit it on the browser and get a .zip. Which would satisfy I think both of our goals.

Now the real trick, is that .zip in theory independently stores each file, so you should, in theory, if there's a .zip writer out there programmed in this crazy way, be able to update a .zip. So you could have the current code update pocketbase AND a resulting .zip simultaneously and then just serve that.

[joas8211]

@Andersama I don't still get what are you using the zip for? Can you still try to explain to me that why do you need the feature? This should help me understand if we want to implement it directly to PalaCMS and how to approach it.

[Andersama]

@joas8211 the reason for the zip is so that you can transfer the project as a whole with the same directory structure that works as a static site and to lower network traffic when downloaded. I'm extracting it and in my case deploying the static files elsewhere.

From what I can tell svelte outputs a project such that javascript files are found under _symbols, and then for every page a index.html under directories as needed.

So my example source code roughly covers that, however I have no idea how to create and keep a .zip available to refer to later, I found an npm package called archiver and hopefully that would work. In my branch what I was planning was to iterate over all the pages, determine the relative slug path, use that to control where the .zip places the index.html files. Then iterate over all the symbols (javascript files) have them under _symbols the id appears to be the javascript's filename. The last part would be making images local, for that you'd walk through the newly added site_uploads database. I'm not sure where those should get placed, but in theory if the compiled html refers to the images locally, you'd have to make sure to place them as expected so you'd need their relative url or how to construct it.

You technically don't have to add this as a hook to pocketbase, I just did that because was the most immediate example that I could follow to make sense of how to work with the project. You could work with the database from svelte, but the ultimate underlying steps would be the same.

Now to explain get_slugged, to avoid talking to the database over and over again, I recursively call this function which takes the current record and a whole array of records. Best is to filter down to records with a matching site, then you could ignore the r.get('site') check. I just brute force and look through linearly for the parent page if there is one for the current page. Technically that loop should always find a record*.

function get_slugged(record, records) {
	let p = record.get('parent')
	let s = record.get('site')
	if (p != null) {
		for (var r of records) {
			if (p == r.get('id') && s = r.get('site')) {
				return get_slugged(r,records) + record.get('slug')
			}
		}
		return '/' + record.get('slug')
	}
	return '/'
}

A better approach would be to have the slug path stored in the database*, then it becomes linear.

[Andersama]

@joas8211 I essentially have a dockerized static site deployment, I unzip the archive, push with git and the site gets redeployed with the new sources, that's my use case. I could also test locally with static resources...tldr; I like having a .zip deployment option for the static site, works like a charm.

[Andersama]

@joas8211 I saw you added a utility to essentially do the following:

function get_slugged(record, records) {
	let p = record.get('parent')
	let s = record.get('site')
	if (p != null) {
		for (var r of records) {
			if (p == r.get('id') && s = r.get('site')) {
				return get_slugged(r,records) + record.get('slug')
			}
		}
		return '/' + record.get('slug')
	}
	return '/'
}

It occurred to me that the best approach would be to incorporate an index for the depth of the url or record. EG: if the records are sorted by that depth, then by travelling linearly through the records you're guaranteed to create the full url in a single pass.

page: home, slug: /, parent: none (depth: 0)
page: about, slug: about, parent: home (depth: 1)
page: travel, slug: travel, parent: home (depth: 1)
page: germany, slug: germany, parent: travel (depth: 2) //Note to create the relative url we could loop linearly through these records

The depth field is implicit given the parent, so it can be set when the record is created. That way you can query the page sorted by depth.

When updating a page however that means you'll need a second query to the database to update all of the children as needed.

[joas8211]

The depth field is implicit given the parent, so it can be set when the record is created. That way you can query the page sorted by depth.

When updating a page however that means you'll need a second query to the database to update all of the children as needed.

I think this would be a hurdle for maintainability we don't want to mess around quite right now. I might look into this with other possible optimizations later on.

[joas8211]

But talking about the site download. I'm quite certain we want to support site downloads alongside other deployment options. I will start doing the groundwork for those soon. The groundwork will mostly consist of internally generating the correct file structure from the get-go for PalaCMS generated sites that quite about any file based HTTP server can work with. I'll try to take inspiration from the conversation we have had here, including following points:

• Progressive site generation could be taken into account when doing the ground work, but most probably not to be implemented at this time. Effective implementation would probably require some sort of dependency analysis based on PalaCMS database structure.

• By using Pocketbase filesystem abstraction, support for ZIP filesystem could be later established by implementing a blob service via blob.Driver interface. This might require some changes to Pocketbase to properly support custom filesystems and the actual implemetation for the blob driver. I doubt the the benefit from creating such a thing would be great enough for the trouble, but I'm leaving it here as a possibility for the future.

• As the possibility for customizing Pocketbase more deeply has now come up, and there doesn't seem to be a good reason to use the precompiled Pocketbase binaries anymore, we could compile our own build of Pocketbase. That way we can convert Pocketbase hooks to use the Go API directly instead of using JSVM. That should improve hook execution performance. Also this allows to create lighter build and distribute PalaCMS also in other formats than Docker image (eg. self-contained binary).

[Andersama]

@joas8211 what's the progress like so far?

[joas8211]

@Andersama I have implemented the groundwork by generating site to the file system. That was implemented with a complete rewrite of the backend code using Golang. The changes are not yet released, as we are tackling a bunch of bugs at the same time.

Would it be usable enough for if you'd use docker cp to download the site files? That would be possibile with the next beta release. The command would be something like docker cp palacms-container:/app/pb_data/storage/sites/example.com/ ./ (replace palacms-container with container name or id, and example.com with your sites hostname). This site needs to be "published" so that the files are generated.

Otherwise download button could be added in the some release after the one we are now preparing.

[Andersama]

@joas8211 that's an idea, I'll try it out.