Skip to content

[FLINK-38777][history] HistoryServer supports application archives #27430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eemario wants to merge 1 commit into
base: master
Choose a base branch
from
+1,313 −97
Open

[FLINK-38777][history] HistoryServer supports application archives #27430

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions docs/layouts/shortcodes/generated/history_server_configuration.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,17 @@
</tr>
</thead>
<tbody>
<tr>
<td><h5>historyserver.archive.clean-expired-applications</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Boolean</td>
<td>Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. </td>
</tr>
<tr>
<td><h5>historyserver.archive.clean-expired-jobs</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Boolean</td>
<td>Whether HistoryServer should cleanup jobs that are no longer present `historyserver.archive.fs.dir`.</td>
<td>Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. <br />Note: This option applies only to legacy job archives created before the introduction of application archiving (FLINK-38761).</td>
</tr>
<tr>
<td><h5>historyserver.archive.fs.dir</h5></td>
Expand All @@ -26,17 +32,23 @@
<td>Duration</td>
<td>Interval for refreshing the archived job directories.</td>
</tr>
<tr>
<td><h5>historyserver.archive.retained-applications</h5></td>
<td style="word-wrap: break-word;">-1</td>
<td>Integer</td>
<td>The maximum number of applications to retain in each archive directory defined by org.apache.flink.configuration.description.TextElement@ae3540e. This option works together with the TTL (see <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>). Archived entities will be removed if their TTL has expired or the retention count limit has been reached. <br />If set to `-1`(default), there is no limit to the number of archives. If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consistent across them. </li></ul></td>
Copy link
Contributor

@davidradl davidradl Jan 16, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is org.apache.flink.configuration.description.TextElement@ae3540e - is this supposed to be a link?
what is <br />. This seems to be an end br tag with a space in. what does this mean?

Copy link
Contributor Author

@eemario eemario Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was caused by a mistake in the HistoryServerOptions description. I will fix it and regenerate the docs. Thanks for pointing it out!

</tr>
<tr>
<td><h5>historyserver.archive.retained-jobs</h5></td>
<td style="word-wrap: break-word;">-1</td>
<td>Integer</td>
<td>The maximum number of jobs to retain in each archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. <ul><li>If the option is not specified as a positive number without specifying <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>, all of the jobs archives will be retained. </li><li>If the option is specified as a positive number without specifying a value of <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>, the jobs archive whose order index based modification time is equals to or less than the value will be retained. </li><li>If this option is specified as a positive number together with the specified <code class="highlighter-rouge">historyserver.archive.retained-ttl</code> option, the job archive will be removed if its TTL has expired or the retained job count has been reached. </li></ul>If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consistent across them. </li></ul></td>
<td>The maximum number of jobs to retain in each archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. <ul><li>If the option is not specified as a positive number without specifying <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>, all of the jobs archives will be retained. </li><li>If the option is specified as a positive number without specifying a value of <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>, the jobs archive whose order index based modification time is equals to or less than the value will be retained. </li><li>If this option is specified as a positive number together with the specified <code class="highlighter-rouge">historyserver.archive.retained-ttl</code> option, the job archive will be removed if its TTL has expired or the retained job count has been reached. </li></ul>If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consistent across them. </li></ul><br />Note: This option applies only to legacy job archives created before the introduction of application archiving (FLINK-38761).</td>
</tr>
<tr>
<td><h5>historyserver.archive.retained-ttl</h5></td>
<td style="word-wrap: break-word;">(none)</td>
<td>Duration</td>
<td>The time-to-live duration to retain the jobs archived in each archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. <ul><li>If the option is not specified without specifying <code class="highlighter-rouge">historyserver.archive.retained-jobs</code>, all of the jobs archives will be retained. </li><li>If the option is specified without specifying <code class="highlighter-rouge">historyserver.archive.retained-jobs</code>, the jobs archive whose modification time in the time-to-live duration will be retained. </li><li>If this option is specified as a positive time duration together with the <code class="highlighter-rouge">historyserver.archive.retained-jobs</code> option, the job archive will be removed if its TTL has expired or the retained job count has been reached. </li></ul>If set to equal to or less than <code class="highlighter-rouge">0</code> milliseconds, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consistent across them. </li></ul></td>
<td>The time-to-live duration to retain the archived entities (jobs and applications) in each archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. This option works together with the retention count limits (see <code class="highlighter-rouge">historyserver.archive.retained-applications</code> and <code class="highlighter-rouge">historyserver.archive.retained-jobs</code>). Archived entities will be removed if their TTL has expired or the retention count limit has been reached. <br />If set to equal to or less than <code class="highlighter-rouge">0</code> milliseconds, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consistent across them. </li></ul></td>
</tr>
<tr>
<td><h5>historyserver.log.jobmanager.url-pattern</h5></td>
Expand Down
93 changes: 71 additions & 22 deletions flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,17 +51,6 @@ public class HistoryServerOptions {
+ " monitor these directories for archived jobs. You can configure the JobManager to archive jobs to a"
+ " directory via `jobmanager.archive.fs.dir`.");

/** If this option is enabled then deleted job archives are also deleted from HistoryServer. */
public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_JOBS =
key("historyserver.archive.clean-expired-jobs")
.booleanType()
.defaultValue(false)
.withDescription(
String.format(
"Whether HistoryServer should cleanup jobs"
+ " that are no longer present `%s`.",
HISTORY_SERVER_ARCHIVE_DIRS.key()));

/**
* Pattern of the log URL of TaskManager. The HistoryServer will generate actual URLs from it.
*/
Expand Down Expand Up @@ -137,6 +126,24 @@ public class HistoryServerOptions {
"Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, ";
private static final String CONFIGURE_CONSISTENT =
"Or you can keep the value of this configuration consistent across them. ";
private static final String LEGACY_NOTE_MESSAGE =
"Note: This option applies only to legacy job archives created before the introduction of application archiving (FLINK-38761).";
private static final String RETAINED_STRATEGY_MESSAGE =
"Archived entities will be removed if their TTL has expired or the retention count limit has been reached. ";

/** If this option is enabled then deleted job archives are also deleted from HistoryServer. */
public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_JOBS =
key("historyserver.archive.clean-expired-jobs")
.booleanType()
.defaultValue(false)
.withDescription(
Description.builder()
.text(
"Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by %s. ",
code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
.linebreak()
.text(LEGACY_NOTE_MESSAGE)
Copy link
Contributor

@davidradl davidradl Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should deprecate the options that only apply to the legacy case. I assume there is an intention to remove them in the next version change

.build());

public static final ConfigOption<Integer> HISTORY_SERVER_RETAINED_JOBS =
key(HISTORY_SERVER_RETAINED_JOBS_KEY)
Expand Down Expand Up @@ -164,6 +171,52 @@ public class HistoryServerOptions {
code("IllegalConfigurationException"))
.linebreak()
.text(NOTE_MESSAGE)
.list(
text(CONFIGURE_SINGLE_INSTANCE),
text(CONFIGURE_CONSISTENT))
.linebreak()
.text(LEGACY_NOTE_MESSAGE)
.build());

/**
* If this option is enabled then deleted application archives are also deleted from
* HistoryServer.
*/
public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_APPLICATIONS =
key("historyserver.archive.clean-expired-applications")
.booleanType()
.defaultValue(false)
.withDescription(
Description.builder()
.text(
"Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by %s. ",
code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
.build());

public static final ConfigOption<Integer> HISTORY_SERVER_RETAINED_APPLICATIONS =
key("historyserver.archive.retained-applications")
.intType()
.defaultValue(-1)
.withDescription(
Description.builder()
.text(
String.format(
"The maximum number of applications to retain in each archive directory defined by %s. ",
code(HISTORY_SERVER_ARCHIVE_DIRS.key())))
.text(
"This option works together with the TTL (see %s). ",
code(HISTORY_SERVER_RETAINED_TTL_KEY))
.text(RETAINED_STRATEGY_MESSAGE)
.linebreak()
.text(
"If set to `-1`(default), there is no limit to the number of archives. ")
.text(
"If set to %s or less than %s, HistoryServer will throw an %s. ",
code("0"),
code("-1"),
code("IllegalConfigurationException"))
.linebreak()
.text(NOTE_MESSAGE)
.list(
text(CONFIGURE_SINGLE_INSTANCE),
text(CONFIGURE_CONSISTENT))
Expand All @@ -176,18 +229,14 @@ public class HistoryServerOptions {
.withDescription(
Description.builder()
.text(
"The time-to-live duration to retain the jobs archived in each archive directory defined by %s. ",
"The time-to-live duration to retain the archived entities (jobs and applications) in each archive directory defined by %s. ",
code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
.list(
text(
"If the option is not specified without specifying %s, all of the jobs archives will be retained. ",
code(HISTORY_SERVER_RETAINED_JOBS_KEY)),
text(
"If the option is specified without specifying %s, the jobs archive whose modification time in the time-to-live duration will be retained. ",
code(HISTORY_SERVER_RETAINED_JOBS_KEY)),
text(
"If this option is specified as a positive time duration together with the %s option, the job archive will be removed if its TTL has expired or the retained job count has been reached. ",
code(HISTORY_SERVER_RETAINED_JOBS_KEY)))
.text(
"This option works together with the retention count limits (see %s and %s). ",
code(HISTORY_SERVER_RETAINED_APPLICATIONS.key()),
code(HISTORY_SERVER_RETAINED_JOBS_KEY))
.text(RETAINED_STRATEGY_MESSAGE)
.linebreak()
.text(
"If set to equal to or less than %s milliseconds, HistoryServer will throw an %s. ",
code("0"), code("IllegalConfigurationException"))
Expand Down
Loading