diff --git a/changes/562.jump.rst b/changes/562.jump.rst new file mode 100644 index 000000000..c063e9f9b --- /dev/null +++ b/changes/562.jump.rst @@ -0,0 +1 @@ +Updated the jump step documentation and fixed a small bug in computing ``total_sigclip_groups``. diff --git a/docs/stcal/jump/description.rst b/docs/stcal/jump/description.rst index bd0bbe7f9..6dd404fa1 100644 --- a/docs/stcal/jump/description.rst +++ b/docs/stcal/jump/description.rst @@ -9,15 +9,45 @@ any pixels that have non-positive or NaN values in the gain reference file will have DQ flags "NO_GAIN_VALUE" and "DO_NOT_USE" set in the output PIXELDQ array. The SCI array of the input data is not modified. -Jumps can be detected in two different ways. The primary way is the two-point -difference method described below. The other way is by selecting ``only_use_ints`` -as ``True`` and if there are enough integrations, then ``sigma_clip`` from the -``astropy.stats`` package will be used to detect jumps. The ``sigma_clip`` method -will also be used if the total number of usable groups (number of groups per -integration multiplied by the number of integrations) is above a minimum threshold. +Jumps in the ramps of a given pixel are detected using statistics of the +two-point differences between successive groups. Those statistics either use +sigma clipping or the method described in +`Anderson & Gordon (2011) `_. -The current implementation uses the two-point difference method described -in `Anderson & Gordon (2011) `_. +The flow control of the jump step is based on two conditions: + +A. If ``only_use_ints`` is ``True``, then the number of integrations needs to be + greater than ``minimum_sigclip_groups``, which has a default of 100. + +B. If ``only_use_ints`` is ``False``, then the total number of available groups + needs to be greater than ``minimum_sigclip_groups``. + +If both A and B are false and there are not enough usable groups in each integration, +``minimum_groups`` defaults to 3, then the jump step is skipped. + +If A is ``True``, then sigma clip across integrations for each group difference +in the ramp (e.g., sigma clip groups 3-2 for all integrations, then sigma clip +groups 4-3 for all integrations, etc). + +If B is ``True``, then sigma clip across all group differences and all integrations +simultaneously (e.g., treat all group differences within an integration and in other +integrations equally). + +If neither A, nor B, are ``True``, then ``astropy.stats.sigma_clip`` is not used. +The algorithm detailed below is used on groups within each integration or is used +over all groups in all integrations. + +If there are at least ``min_diffs_single_pass`` in each integration, then use the +two-point difference detailed below over the first group differences in each +integration. This will do a single pass over the first group differences in each +integration. + +Otherwise, use the two-point difference detailed below looping over all groups +across all integrations. This will be an iterative approach that loops over all +first group differences, :math:`(ngroups-1) * nints`, where :math:`ngroups` is +the number of groups in each integration (the :math:`-1` is used because the +operations are on the first differences) and :math:`nints` is the number of +integrations. Two-Point Difference Method ^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -25,9 +55,9 @@ The two-point difference method is applied to each integration as follows: #. Compute the first differences for each pixel (the difference between adjacent groups) -#. Compute the clipped median (dropping the largest difference) of the first differences for each pixel. - If there are only three first difference values (four groups), no clipping is - performed when computing the median. +#. Compute the clipped median (dropping the largest difference) of the first + differences for each pixel. If there are only three first difference values + (four groups), no clipping is performed when computing the median. #. Use the median to estimate the Poisson noise for each group and combine it with the read noise to arrive at an estimate of the total expected noise for each difference. @@ -40,8 +70,9 @@ The two-point difference method is applied to each integration as follows: that still exceed the rejection threshold. #. Stop iterating on a given pixel when no new jumps are found or only one difference remains. -#. If there are only two differences (three groups), the smallest one is compared to the larger - one and if the larger one is above a threshold, it is flagged as a jump. +#. If there are only two differences (three groups), the smallest one is compared + to the larger one and if the larger one is above a threshold, it is flagged + as a jump. #. If flagging of the 4 neighbors is requested, then the 4 adjacent pixels will have ramp jumps flagged in the same group as the central pixel as long as it has a jump between the min and max requested levels for this option. diff --git a/src/stcal/jump/twopoint_difference.py b/src/stcal/jump/twopoint_difference.py index 952cb4191..6e26f88db 100644 --- a/src/stcal/jump/twopoint_difference.py +++ b/src/stcal/jump/twopoint_difference.py @@ -663,7 +663,7 @@ def groups_all_set_dnu(nints, ngroups, gdq, twopt_p): if twopt_p.only_use_ints: total_sigclip_groups = nints else: - total_sigclip_groups = nints * ngroups - num_flagged_grps + total_sigclip_groups = nints * ngroups - total_flagged_grps min_usable_groups = ngroups - max_flagged_grps total_groups = nints * ngroups - total_flagged_grps