Policy for DQ bit handling?

[schlafly]

I wanted to lay out what I thought the 'right' DQ bit handling was to make sure that we can be on the same page for Roman and Webb. Here's what I think we should do:

• routines should flag problematic pixels with a user-specified bit following a dictionary
• routines should not flag DO_NOT_USE; downstream pipeline should decide DO_NOT_USE behavior
  • I think I would except ramp_fitting here and let it flag DO_NOT_USE for sufficiently bad ramp fits (e.g., 0 good resultants)
• ramp fitting should flag results with the or of the groupdq flags

I think this is mostly already what is done? #515 is an exception that we've already discussed, and where we think that moving closer to the above would be the right move. Another case where we are not doing the above is in ramp fitting, where currently only jump and sat propagate to the final DQ image; the above would advocate for treating those equivalently to other bits. For Roman that would mean also propagating things like DROPOUT (some issue reading out some groups; not expected to occur frequently) and AD_FLOOR (likewise not expected to occur frequently). I don't immediately see why we should not propagate those in ramp fitting, though I agree that jump and saturation are more common and important.

@drlaw1558 , @tapastro , do you see this differently?

[drlaw1558]

I believe more generally what's done by various steps is to set both the process-impacting DO_NOT_USE bit and a information-only bit (e.g., HOT) describing why the DO_NOT_USE was set.

This isn't so much a code-choices issue, but more commonly how reference files are applied. The flatfield reference files for instance set SCI=ERR=NaN and DQ=DO_NOT_USE+NO_FLAT_FIELD in regions that don't see light from the sky. Likewise with the MIRI RSCD correction, in which reference files flag various groups with DO_NOT_USE.

I don't think it's practical to change that given all of the affected files/steps, but even if we were designing the system anew I'm not sure that it makes sense to always decide DO_NOT_USE downstream as that splits the logical flow. If you know at the flatfield step, for instance, that certain pixels shouldn't be used for anything else later on (and indeed are NaN valued after flatfield) why require logic to set DO_NOT_USE for them in the next step (photom) or potentially all later steps that need to not use them?

It's worth mentioning another case currently under discussion; see JP-4000 and spacetelescope/jwst#10091 for background.

Brief synopsis is that we're adding the ability to deal with persistence in detector1; the persistence step is being modified to look for anywhere SATURATION was set and to set the PERSISTENCE DQ bit for all groups likely to be affected in both integrations and exposures later on. (Exposure flagging for offline use only, not in DMS). Ordinarily this bit is inform-only, but absent a true correction for persistence with trap decay times that we don't have, the plan is to allow users to also choose to set DO_NOT_USE for such groups if they want to mask out affected data.

At the moment the switch to set both PERSISTENCE and DO_NOT_USE is in the persistence step, as this is where a user would most likely look to customize how persistence is being handled in the pipeline. Arguably the ramp fitting step could be modified to be told to interpret PERSISTENCE as DO_NOT_USE, but that would mean users would need to adjust parameters for two steps rather than one (and would need to know to look at the ramp_fit step rather than the persistence step to change how persistence is handled).

As to propagating information from group to pixel stage through ramp fitting though I agree it seems reasonable to pass more information through. Most of the existing flags pertain really to the overall state of a given pixel rather than to something group-specific.

[schlafly]

This isn't so much a code-choices issue, but more commonly how reference files are applied. The flatfield reference files for instance set SCI=ERR=NaN and DQ=DO_NOT_USE+NO_FLAT_FIELD in regions that don't see light from the sky. Likewise with the MIRI RSCD correction, in which reference files flag various groups with DO_NOT_USE.

Thanks. I was actually focused here on the pre-ramp fitting stages. Roman also for pixel-level bad flags always ORs in the flags (though ~always because they're explicitly marked DO_NOT_USE in the reference file rather than because the pipeline code upgrades e.g. NO_FLAT_FIELD to DO_NOT_USE).

For the groupdq flags, we don't now, for example set DO_NOT_USE for saturated pixels, but we do for AD_FLOOR, and I'm actually unaware of how JUMP works in this regard. I think it would be good to have a more consistent policy there. I would also be happy with always setting both an informational bit and DO_NOT_USE, and having ramp_fitting respect only DO_NOT_USE; that would make the groupdq bit treatment more parallel to the pixeldq treatment.