Examine alignment results

[cwenge]

Hello there,

we've been extensiveley using your tool on our dataset. It is very helpful! Now we're trying to examine the results. Best indicator for now is looking manually at the deformation fields, which show us the areas where the image has been transformed. Irregularities at the "grid" show us deformations of the area, whereas the color and brightness indicate the direction of the displacement and its' magnitude according to the repositories' readme.

Are there any further informations on the color code? How are they mapped to the direction?

I'm also interested in evaluating the data summary, but I'm not quite sure how to interprete these values. Is there any way to calculate, whether images had to be warped intensively or not? We only want to keep those image datasets, where the alignment didn't need to alter the original images too much.

I'm looking forward to some insights and pleace correct me, if I misunderstood things.

Best regards,

Chris

[cdgatenbee]

Hi @cwenge,
So glad to hear you're finding valis useful! I put together a function that will summarize the transformations for each image, to help you gauge how much each image was warped. For the non-rigid transformations, it provides the minimum, mean, and maximum displacements, but you could edit the code to do a more detailed analysis. One thing is sometimes the really large displacements are outside the area you're interested in, so the image may not be as distorted as the summary values suggest.

from valis import registration, warp_tools, viz


def transform_M(M, transformation_src_shape_rc, transformation_dst_shape_rc, src_shape_rc, out_shape_rc):
    """Transform M to work on scaled image

    """
    img_corners_xy = warp_tools.get_corners_of_image(src_shape_rc)[::-1]
    warped_corners = warp_tools.warp_xy(img_corners_xy, M=M,
                            transformation_src_shape_rc=transformation_src_shape_rc,
                            transformation_dst_shape_rc=transformation_dst_shape_rc,
                            src_shape_rc=src_shape_rc,
                            dst_shape_rc=out_shape_rc)
    M_tform = transform.ProjectiveTransform()
    M_tform.estimate(warped_corners, img_corners_xy)
    warp_M = M_tform.params

    return warp_M



def summarize_transforms(registrar):
    """
    Summarize image transformations

    Parameters
    ----------
    registrar : registration.Valis
        Valis object that has registered images

    Returns
    -------
    transformations_df : pd.Dataframe
        pandas dataframe containing information about how each image is warped:

        "image": name of the image
        "tx": left/right translation (shift)
        "ty": up/down translation (shift)
        "sx": image scaling (resizing) along the x-axis
        "sy": image scaling (resizing) along the y-axis
        "rotation": counter-clockwise rotation in radians
        "min_dx": minimum non-rigid displacement along the x-axis
        "mean_dx": average non-rigid displacement along the x-axis
        "max_dx": maximum non-rigid displacement along the x-axis
        "min_dy": minimum non-rigid displacement along the y-axis
        "mean_dy": average non-rigid displacement along the y-axis
        "max_dy": maximum non-rigid displacement along the y-axis

    """

    img_names = [None] * registrar.size

    # Record translations (shifts up/down, left right)
    all_tx = [None] * registrar.size
    all_ty = [None] * registrar.size

    # Recording scalings
    all_sx = [None] * registrar.size
    all_sy = [None] * registrar.size

    # Record rotation, counter-clockwise direction as radians
    all_rotations = [None] * registrar.size

    # Summarize non-rigid displcaments
    all_min_dx = [None] * registrar.size
    all_mean_dx = [None] * registrar.size
    all_max_dx = [None] * registrar.size

    all_min_dy = [None] * registrar.size
    all_mean_dy = [None] * registrar.size
    all_max_dy = [None] * registrar.size

    aligned_img_shape_rc = registrar.get_aligned_slide_shape(0)
    for slide_obj in registrar.slide_dict.values():
        img_names[slide_obj.stack_idx] = slide_obj.name

        # Extract rigid transformation parameters
        # Registration performed on smaller images, so need to transform M before extracting rigid parameters
        img_M = transform_M(slide_obj.M,
                            transformation_src_shape_rc=slide_obj.processed_img_shape_rc,
                            transformation_dst_shape_rc=slide_obj.reg_img_shape_rc,
                            src_shape_rc=slide_obj.slide_dimensions_wh[0][::-1],
                            out_shape_rc=aligned_img_shape_rc)


        A = transform.AffineTransform(img_M)
        all_tx[slide_obj.stack_idx] = A.translation[0] # for uncropped image
        all_ty[slide_obj.stack_idx] = A.translation[1] # for uncropped image
        all_sx[slide_obj.stack_idx] = A.scale[0]
        all_sy[slide_obj.stack_idx] = A.scale[1]
        all_rotations[slide_obj.stack_idx] = A.rotation

        # Account for cropping after warping
        if slide_obj.crop is not False:
            if slide_obj.crop == registration.CROP_REF:
                ref_slide = registrar.get_ref_slide()
                to_calc_crop_shape_rc = ref_slide.slide_dimensions_wh[0][::-1]
            elif slide_obj.crop == registration.CROP_OVERLAP:
                to_calc_crop_shape_rc = aligned_img_shape_rc

            crop_bbox_xywh, _ = slide_obj.get_crop_xywh(crop=slide_obj.crop,
                                                        out_shape_rc=to_calc_crop_shape_rc)
            all_tx[slide_obj.stack_idx] += crop_bbox_xywh[0]
            all_ty[slide_obj.stack_idx] += crop_bbox_xywh[1]

        # Summarize non-rigid transformations stored in slide_obj.bk_dxdy. Scale to determine effect on full resolution image
        if isinstance(slide_obj.bk_dxdy, pyvips.Image):
            dispalcement_shape_rc = np.array([slide_obj.bk_dxdy.height, slide_obj.bk_dxdy.width])
            avg_fxn_str = "avg"
        else:
            dispalcement_shape_rc = np.array(slide_obj.bk_dxdy[0].shape)
            avg_fxn_str = "mean"

        displacement_sxy = (aligned_img_shape_rc/dispalcement_shape_rc)[::-1]

        all_min_dx[slide_obj.stack_idx] = displacement_sxy[0]*slide_obj.bk_dxdy[0].min()
        all_mean_dx[slide_obj.stack_idx] = displacement_sxy[0]*getattr(slide_obj.bk_dxdy[0], avg_fxn_str)()
        all_max_dx[slide_obj.stack_idx] = displacement_sxy[0]*slide_obj.bk_dxdy[0].max()

        all_min_dy[slide_obj.stack_idx] = displacement_sxy[1]*slide_obj.bk_dxdy[1].min()
        all_mean_dy[slide_obj.stack_idx] = displacement_sxy[1]*getattr(slide_obj.bk_dxdy[1], avg_fxn_str)()
        all_max_dy[slide_obj.stack_idx] = displacement_sxy[1]*slide_obj.bk_dxdy[1].max()

    transformations_df = pd.DataFrame({
        "image": img_names,
        "tx": all_tx,
        "ty": all_ty,
        "sx": all_sx,
        "sy": all_sy,
        "rotation": all_rotations,
        "min_dx": all_min_dx,
        "mean_dx": all_mean_dx,
        "max_dx": all_max_dx,
        "min_dy": all_min_dy,
        "mean_dy": all_mean_dy,
        "max_dy": all_max_dy
    })

    return transformations_df



registrar = registration.load_registrar(path_to_registrar)
transformations_df = summarize_transforms(registrar)

To interpret the colors in the grid image, you can create this displacement legend using the following code:

viz.draw_displacement_legend()
plt.show()

it should show this qualitative legend. So pixels that were pushed up and to the right are purple, those pushed down to the left are green, and so on. The colors in each grid image based on the values in that image though, so comparing the colors in two different displacement grid images may not be too helpful.

In this summary csv file, the "mean_rigid_D" and "mean_non_rigid_D" values indicate the average distance between automatically detected image features in the ''from" image (the one being warped) and the corresponding ones in the "to" image (the one the "from" image is being aligned to) after rigid and non-rigid warping, respectively. Something to consider here too is that these features were detected in much smaller images, and so may not be that precise of a measurement for the full resolution image. In fact, in the recent ACROBAT Grand Challenge, I found that the error estimated using these feature points was about three times worse than the actual error calculated using hand annotated tissue landmarks. In other words, the registration actually measured three times better than the error estimates suggested. I think they can still be helpful though, especially if trying to identify images that had especially good/bad alignments.

I hope the above helps identify those alignments that didn't require altering the image very much. If you have any more questions, please don't hesitate to ask.

Best,
-Chandler