Add --exclude-snapshots option, which makes exclude pg_logical/snapshots by harukat · Pull Request #272 · ossc-db/pg_rman

harukat · 2024-12-18T10:20:51Z

If there are many large file size temporary files or logical replication snapshot files in the database cluster directory, it will take longer to get a full backup. These files are not likely to be used after the restore. Therefore, I have added an option to exclude these files when obtaining full backups.

Since pg_basebackup always excludes pgsql_tmp, it is possible to always exclude pgsql_tmp and make only snapshots optional. Please let us know what you think.

…ots dirs. If there are many large file size temporary files or logical replication snapshot files in the database cluster directory, it will take longer to get a full backup. These files are not likely to be used after the restore. Therefore, I have added an option to exclude these files when obtaining full backups.

harukat · 2024-12-20T02:38:08Z

It was pointed out to me that pgsql_tmp is already excluded in the current code, so I'll fixed it.

In the most recent commit, I added a --exclude-tmpdir option that also excludes snapshots and pgsql_tmp directory, but since pgsql_tmp was already always excluded, I limited it to only excluding snapshots.

shinyaaa

Thanks for the PR. I have some comments.

I'd like you to add option test, like below.

pg_rman/sql/option.sh

Lines 179 to 184 in d1b5c57

    
           echo '###### COMMAND OPTION TEST-0019 ######' 
        
           echo '###### invalid value in pg_rman.ini ######' 
        
           init_catalog 
        
           echo "HARD_COPY=FOO" >> ${BACKUP_PATH}/pg_rman.ini 
        
           pg_rman backup -B ${BACKUP_PATH} -A ${ARCLOG_PATH} -b full -p ${TEST_PGPORT};echo $? 
        
           echo ''

And env is need to be initialized.

pg_rman/sql/common.sh

Lines 9 to 30 in d1b5c57

    
           # Unset environment variables usable by both Postgres and pg_rman 
        
           unset PGUSER 
        
           unset PGPORT 
        
           unset PGDATABASE 
        
           unset COMPRESS_DATA 
        
           unset BACKUP_MODE 
        
           unset ARCLOG_PATH 
        
           unset BACKUP_PATH 
        
           unset SRVLOG_PATH 
        
           unset WITH_SERVLOG 
        
           unset SMOOTH_CHECKPOINT 
        
           unset KEEP_DATA_GENERATIONS 
        
           unset KEEP_DATA_DAYS 
        
           unset KEEP_ARCLOG_FILES 
        
           unset KEEP_ARCLOG_DAYS 
        
           unset KEEP_SRVLOG_FILES 
        
           unset KEEP_SRVLOG_DAYS 
        
           unset RECOVERY_TARGET_TIME 
        
           unset RECOVERY_TARGET_XID 
        
           unset RECOVERY_TARGET_INCLUSIVE 
        
           unset RECOVERY_TARGET_TIMELINE 
        
           unset HARD_COPY

shinyaaa · 2025-01-24T09:00:06Z

pg_rman.c

+	if (exclude_snapshots)
+	{
+		pgdata_exclude[i++] = SNAPSHOTS_DIR;
+	}


Suggested change

if (exclude_snapshots)

{

pgdata_exclude[i++] = SNAPSHOTS_DIR;

}

if (exclude_snapshots)

pgdata_exclude[i++] = SNAPSHOTS_DIR;

shinyaaa · 2025-01-24T09:36:58Z

docs/index-ja.html

+<li><strong><code>--exclude-snapshots</code></strong>
+
+<ul>
+<li>一時ファイル用に使われる <code>pgsql_tmp</code> ディレクトリ内と、ロジカルレプリケーションで使われる <code>snapshots</code> ディレクトリ内をバックアップ対象から除外します。</li>


Suggested change

<li>一時ファイル用に使われる <code>pgsql_tmp</code> ディレクトリ内と、ロジカルレプリケーションで使われる <code>snapshots</code> ディレクトリ内をバックアップ対象から除外します。</li>

<li>ロジカルレプリケーションで使われる <code>$PGDATA/pg_logical/snapshots</code> ディレクトリ内をバックアップ対象から除外します。</li>

shinyaaa · 2025-01-24T09:41:33Z

docs/index-ja.html

+<td>&ndash;exclude-snapshots</td>
+<td>EXCLUDE_TMPDIR</td>
+<td>指定可</td>
+<td>ロジカルスナップショットをバックアップから除外</td>


I'm not sure "ロジカルスナップショット" is general representation.

Suggested change

<td>ロジカルスナップショットをバックアップから除外</td>

<td>ロジカルレプリケーションのスナップショットをバックアップから除外</td>

shinyaaa · 2025-01-24T09:42:14Z

docs/index.html

+<li><strong><code>--exclude-snapshots</code></strong>
+
+<ul>
+<li>Exclude <code>snapshots</code> directory, which is used for logical replication.</li>


Suggested change

<li>Exclude <code>snapshots</code> directory, which is used for logical replication.</li>

<li>Exclude <code>$PGDATA/pg_logical/snapshots</code> directory which is used for logical replication.</li>

shinyaaa · 2025-01-24T09:43:05Z

docs/index-ja.html

+<tr>
+<td></td>
+<td>&ndash;exclude-snapshots</td>
+<td>EXCLUDE_TMPDIR</td>


This env name is confuse us.

Suggested change

<td>EXCLUDE_TMPDIR</td>

<td>EXCLUDE_SNAPSHOT_DIR</td>

shinyaaa · 2025-01-24T09:48:54Z

expected/option.out

  --keep-srvlog-days=DAY    keep serverlog modified in DAY days
  --standby-host=HOSTNAME   standby host when taking backup from standby
  --standby-port=PORT       standby port when taking backup from standby
+  --exclude-snapshots       exclude pg_logical/snapshots


Suggested change

--exclude-snapshots exclude pg_logical/snapshots

--exclude-snapshots exclude $PGDATA/pg_logical/snapshots

shinyaaa · 2025-01-24T09:50:28Z

pg_rman.c

 const char *PROGRAM_VERSION	= "1.3.16";
 const char *PROGRAM_URL		= "http://github.com/ossc-db/pg_rman";
 const char *PROGRAM_ISSUES	= "http://github.com/ossc-db/pg_rman/issues";
+const char *SNAPSHOTS_DIR   = "snapshots";


I think it should be below.

pg_rman/pg_rman.h

Lines 27 to 45 in d1b5c57

/* Directory/File names */

#define DATABASE_DIR "database"

#define ARCLOG_DIR "arclog"

#define SRVLOG_DIR "srvlog"

#define RESTORE_WORK_DIR "backup"

#define PG_XLOG_DIR "pg_wal"

#define PG_TBLSPC_DIR "pg_tblspc"

#define TIMELINE_HISTORY_DIR "timeline_history"

#define BACKUP_INI_FILE "backup.ini"

#define PG_RMAN_INI_FILE "pg_rman.ini"

#define SYSTEM_IDENTIFIER_FILE "system_identifier"

#define MKDIRS_SH_FILE "mkdirs.sh"

#define DATABASE_FILE_LIST "file_database.txt"

#define ARCLOG_FILE_LIST "file_arclog.txt"

#define SRVLOG_FILE_LIST "file_srvlog.txt"

#define SNAPSHOT_SCRIPT_FILE "snapshot_script"

#define PG_BACKUP_LABEL_FILE "backup_label"

#define PG_TBLSPC_MAP_FILE "tablespace_map"

#define PG_BLACK_LIST "black_list"

shinyaaa · 2025-01-24T09:55:12Z

pg_rman.c


 /* directory configuration */
 pgBackup	current;
+static bool exclude_snapshots = false;


I think this is backup configuration. For example, directory options are -D, -A, -B, -S, and -G.

shinyaaa · 2025-01-24T09:55:32Z

pg_rman.c

 	{ 's', 10, "recovery-target-timeline"	, &target_tli_string, SOURCE_ENV },
 	{ 's', 11, "recovery-target-action"		, &target_action	, SOURCE_ENV },
 	{ 'b', 12, "hard-copy"	, &is_hard_copy		, SOURCE_ENV },
+	{ 'b', 15, "exclude-snapshots", &exclude_snapshots, SOURCE_ENV },


I think this is backup configuration.

shinyaaa · 2025-01-24T09:56:21Z

pg_rman.c

 	printf(_("  --keep-srvlog-days=DAY    keep serverlog modified in DAY days\n"));
 	printf(_("  --standby-host=HOSTNAME   standby host when taking backup from standby\n"));
 	printf(_("  --standby-port=PORT       standby port when taking backup from standby\n"));
+	printf(_("  --exclude-snapshots       exclude pg_logical/snapshots\n"));


Suggested change

printf(_(" --exclude-snapshots exclude pg_logical/snapshots\n"));

printf(_(" --exclude-snapshots exclude $PGDATA/pg_logical/snapshots\n"));

harukat mentioned this pull request Dec 18, 2024

Add --exclude-tmpdir option, which makes exclude pgsql_tmp and snapshots dirs #271

Closed

Add --exclude-snapshots in place of --exclude-tmpdir.

4b11460

In the most recent commit, I added a --exclude-tmpdir option that also excludes snapshots and pgsql_tmp directory, but since pgsql_tmp was already always excluded, I limited it to only excluding snapshots.

harukat changed the title ~~Add --exclude-tmpdir option, which makes exclude pgsql_tmp and snapshots dirs~~ Add --exclude-snapshots option, which makes exclude pg_logical/snapshots dirs Dec 20, 2024

harukat changed the title ~~Add --exclude-snapshots option, which makes exclude pg_logical/snapshots dirs~~ Add --exclude-snapshots option, which makes exclude pg_logical/snapshots Dec 20, 2024

shinyaaa self-requested a review January 24, 2025 08:23

shinyaaa reviewed Jan 24, 2025

View reviewed changes

shinyaaa force-pushed the master branch from 8baa5ce to 0ab83ca Compare January 28, 2025 05:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add --exclude-snapshots option, which makes exclude pg_logical/snapshots #272

Add --exclude-snapshots option, which makes exclude pg_logical/snapshots #272
harukat wants to merge 2 commits intoossc-db:masterfrom
harukat:exclude_tmpdir

harukat commented Dec 18, 2024

Uh oh!

harukat commented Dec 20, 2024

Uh oh!

shinyaaa left a comment •

edited

Loading

Uh oh!

shinyaaa Jan 24, 2025

Uh oh!

shinyaaa Jan 24, 2025

Uh oh!

shinyaaa Jan 24, 2025

Uh oh!

shinyaaa Jan 24, 2025

Uh oh!

shinyaaa Jan 24, 2025

Uh oh!

shinyaaa Jan 24, 2025

Uh oh!

shinyaaa Jan 24, 2025

Uh oh!

shinyaaa Jan 24, 2025

Uh oh!

shinyaaa Jan 24, 2025

Uh oh!

shinyaaa Jan 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	echo '###### COMMAND OPTION TEST-0019 ######'
	echo '###### invalid value in pg_rman.ini ######'
	init_catalog
	echo "HARD_COPY=FOO" >> ${BACKUP_PATH}/pg_rman.ini
	pg_rman backup -B ${BACKUP_PATH} -A ${ARCLOG_PATH} -b full -p ${TEST_PGPORT};echo $?
	echo ''

	# Unset environment variables usable by both Postgres and pg_rman
	unset PGUSER
	unset PGPORT
	unset PGDATABASE
	unset COMPRESS_DATA
	unset BACKUP_MODE
	unset ARCLOG_PATH
	unset BACKUP_PATH
	unset SRVLOG_PATH
	unset WITH_SERVLOG
	unset SMOOTH_CHECKPOINT
	unset KEEP_DATA_GENERATIONS
	unset KEEP_DATA_DAYS
	unset KEEP_ARCLOG_FILES
	unset KEEP_ARCLOG_DAYS
	unset KEEP_SRVLOG_FILES
	unset KEEP_SRVLOG_DAYS
	unset RECOVERY_TARGET_TIME
	unset RECOVERY_TARGET_XID
	unset RECOVERY_TARGET_INCLUSIVE
	unset RECOVERY_TARGET_TIMELINE
	unset HARD_COPY

	<li>一時ファイル用に使われる <code>pgsql_tmp</code> ディレクトリ内と、ロジカルレプリケーションで使われる <code>snapshots</code> ディレクトリ内をバックアップ対象から除外します。</li>
	<li>ロジカルレプリケーションで使われる <code>$PGDATA/pg_logical/snapshots</code> ディレクトリ内をバックアップ対象から除外します。</li>

	<td>ロジカルスナップショットをバックアップから除外</td>
	<td>ロジカルレプリケーションのスナップショットをバックアップから除外</td>

	<li>Exclude <code>snapshots</code> directory, which is used for logical replication.</li>
	<li>Exclude <code>$PGDATA/pg_logical/snapshots</code> directory which is used for logical replication.</li>

	--exclude-snapshots exclude pg_logical/snapshots
	--exclude-snapshots exclude $PGDATA/pg_logical/snapshots

	/* Directory/File names */
	#define DATABASE_DIR "database"
	#define ARCLOG_DIR "arclog"
	#define SRVLOG_DIR "srvlog"
	#define RESTORE_WORK_DIR "backup"
	#define PG_XLOG_DIR "pg_wal"
	#define PG_TBLSPC_DIR "pg_tblspc"
	#define TIMELINE_HISTORY_DIR "timeline_history"
	#define BACKUP_INI_FILE "backup.ini"
	#define PG_RMAN_INI_FILE "pg_rman.ini"
	#define SYSTEM_IDENTIFIER_FILE "system_identifier"
	#define MKDIRS_SH_FILE "mkdirs.sh"
	#define DATABASE_FILE_LIST "file_database.txt"
	#define ARCLOG_FILE_LIST "file_arclog.txt"
	#define SRVLOG_FILE_LIST "file_srvlog.txt"
	#define SNAPSHOT_SCRIPT_FILE "snapshot_script"
	#define PG_BACKUP_LABEL_FILE "backup_label"
	#define PG_TBLSPC_MAP_FILE "tablespace_map"
	#define PG_BLACK_LIST "black_list"

	printf(_(" --exclude-snapshots exclude pg_logical/snapshots\n"));
	printf(_(" --exclude-snapshots exclude $PGDATA/pg_logical/snapshots\n"));

Conversation

harukat commented Dec 18, 2024

Uh oh!

harukat commented Dec 20, 2024

Uh oh!

shinyaaa left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shinyaaa left a comment •

edited

Loading