Allow archive_catalog entries for non-required fields

[braingram]

For future changes it could be beneficial to include new archive_catalog information for non-required fields. From conversation with @schlafly and @cjarnold it would be good to test how this works for:

• manifest generation: where the expectation is that a non-required field that is present and has archive_catalog information will be included in the manifest
• archive ingest: where the expectation is that for a field missing from the manifest, the archive would fill in a NULL value

On the RAD/schema side we will need to update tests to allow for this:

rad/tests/test_schemas.py

Line 199 in 7cf6d0d

def test_metadata_force_required(self, schema_uri, schema, request):

and consider adding additional romancal unit tests to cover the expected behavior (field populated for new files, etc...).

[PaulHuwe]

On the RAD side, I propose that we add a keyword to flag if a db entry is required or not (maybe a bool entitled dbrequired or something). That way we could have both optional and required keywords, while enforcing the latter exist for validation.

[schlafly]

I feel like we already have one notion of required. One could add the notion of archive_required or something, but if we do have a new keyword that we want the archive to require we should probably just make the keyword required overall.

[braingram]

How about instead of dbrequired we could default to entries with archive_catalog information being required then add exclusions in the updated test?

[PaulHuwe]

I am concerned about 1) an 'exclusion list' getting large in the test and 2) having that information divorced from the related schema. Maybe Eddie's suggestion of keying off of required may work? The question would then be - would we have keywords that we want to be required but the archive doesn't?

[schlafly]

To be clear, I'm suggesting no changes except removing tests that require 'required' for fields with archive catalogs. I think it was right to discourage that in the past, making sure that things get into the archive, but is kind of fine going forward. We probably should make it easy to periodically review whether we want to add pre-existing keywords to the 'required' list on some slow cadence (annual?).

[karlhillx]

@braingram @schlafly @cjarnold

Before I update test_metadata_force_required, I wanted to check.... should sdf continue to force fields into required, or should the relaxation apply to it as well? That determines whether METADATA_FORCING_REQUIRED becomes ("sdf",) or ().