Question about repruduce on Immersive

[ccxi1008]

I tried to get the same structure like in README ,but still have some questions:

1. Can you provide the specific code for processing the data since my undistortion seems to have some problem.
2. When I use the structure and train, this happens:

Why it becomes Nonetype?

[amritamaz]

We unfortunately don't have plans to release our Immersive dataset preprocessing at this time.

However, you can refer to the implementation from SpaceTime Gaussians which is very similar, and then reorganize the results in the QUEEN data format specified in the readme.

Re: the NoneType issue, which config file are you using to train? Can you share the full training command?

[ccxi1008]

We unfortunately don't have plans to release our Immersive dataset preprocessing at this time.

However, you can refer to the implementation from SpaceTime Gaussians which is very similar, and then reorganize the results in the QUEEN data format specified in the readme.

Re: the NoneType issue, which config file are you using to train? Can you share the full training command?

I use command

python train.py \
        --config $CONFIG_FILE \
        -s "${DATA_ROOT}/${scene}" \
        -m "${OUTPUT_ROOT}/${scene}" \
        --save_iterations -1 \
        --log_compressed \
        --log_images

and the scene config you priveded such as immersive_cave.yaml or immersive.yaml
And i think this problem is not relative to camera paramters？
BTW, the result from the only succeed firts frame is very bad. Do you have any ideas?

[amritamaz]

if your undistortion has some problems, then I would suspect that is the cause of poor first frame reconstruction. Can you sanity check by training with the base 3dgs or gsplat repo's?

[ccxi1008]

if your undistortion has some problems, then I would suspect that is the cause of poor first frame reconstruction. Can you sanity check by training with the base 3dgs or gsplat repo's?

I think I have 2 problems ,and these two happen with all other data (which I organized similar to Immersive with specific data_reader) except for dynerf-N3V:
1.The result is bad (1st frame). This may be caused by poor undistortion,but I use the same undistortion results on other proceded data, which preform well on works like 3dgstream.
2.Procedure stops after 1st frame, as shown in picture:

This should have no relationship with problem 1,right? Was something others wrong?

[amritamaz]

Item 1: poor first frame result -- can you check the training parameters in the config compared to parameters that worked for other methods like 3DGStream? It does look like the loss reported (0.111) is quite high, I suspect something is wrong with the data layout or cameras.

Item 2: I have not experienced this error before. Does the training stop after one frame even for dynerf/N3DV scenes? The other thing you can test is not using --log-compressed when training.

[ccxi1008]

Item 1: poor first frame result -- can you check the training parameters in the config compared to parameters that worked for other methods like 3DGStream? It does look like the loss reported (0.111) is quite high, I suspect something is wrong with the data layout or cameras.

Item 2: I have not experienced this error before. Does the training stop after one frame even for dynerf/N3DV scenes? The other thing you can test is not using --log-compressed when training.

Thanks and I'll have a try.
Actually I do use the STG process code. But I want to ensure if I should use pre_immersive_undistorted.py instead of pre_immersive_distorted.py (which is the link you provided) since in your datareader:

        if intr.model=="SIMPLE_PINHOLE":
            focal_length_x = intr.params[0]/scale
            FovY = focal2fov(focal_length_x, height)
            FovX = focal2fov(focal_length_x, width)
        elif intr.model=="PINHOLE":
            focal_length_x = intr.params[0]/scale
            focal_length_y = intr.params[1]/scale
            FovY = focal2fov(focal_length_y, height)
            FovX = focal2fov(focal_length_x, width)
        else:
            assert False, "Colmap camera model not handled: only datasets"+\
                   "(PINHOLE or SIMPLE_PINHOLE cameras) supported!"

And another question is can you remember the settings of scale and focal_scale to get image_scaled_2? Especially in these 2 functions in STG:

def imageundistort_no_mapper(video, offsetlist=[0],focalscale=1.0, fixfocal=None):
    with open(video / "models.json", "r") as f:
        meta = json.load(f)

    for idx , camera in enumerate(tqdm.tqdm(meta, desc="undistort")):
        folder = camera['name'] # camera_0001
        view = camera
        intrinsics = np.array([[view['focal_length'], 0.0, view['principal_point'][0]],
                            [0.0, view['focal_length'], view['principal_point'][1]],
                            [0.0, 0.0, 1.0]])
        dis_cef = np.zeros((4))

        dis_cef[:2] = np.array(view['radial_distortion'])[:2]

        map1, map2 = None, None
        for offset in offsetlist:
            imagepath = video / folder / f"{offset}.png"
            imagesavepath = video / f"colmap_{offset}" / "input" / f"{folder}.png"

            inputimagefolder = video / f"colmap_{offset}" / "input"
            inputimagefolder.mkdir(exist_ok=True, parents=True)
            assert imagepath.exists()
            
            if not imagesavepath.exists():
                try:
                    image = cv2.imread(str(imagepath)).astype(np.float32) #/ 255.0
                except:
                    print("failed to read image", imagepath)
                    quit()
                    
                h, w = image.shape[:2]

                image_size = (w, h)
                knew = np.zeros((3, 3), dtype=np.float32)

    
                knew[0,0] = focalscale * intrinsics[0,0]
                knew[1,1] = focalscale * intrinsics[1,1]
                knew[0,2] =  view['principal_point'][0] # cx fixed half of the width
                knew[1,2] =  view['principal_point'][1] #
                knew[2,2] =  1.0


                
                
                map1, map2 = cv2.fisheye.initUndistortRectifyMap(intrinsics, dis_cef, R=None, P=knew, size=(w, h), m1type=cv2.CV_32FC1)

                undistorted_image = cv2.remap(image, map1, map2, interpolation=cv2.INTER_CUBIC, borderMode=cv2.BORDER_CONSTANT)
                undistorted_image = undistorted_image.clip(0,255.0).astype(np.uint8)

                cv2.imwrite(str(imagesavepath), undistorted_image)
            else:
                print("already exists")

and when writting to sparse file

def convertmodel2dbfiles(path, offset=0, scale=1.0):
    with (path / "models.json").open("r") as f:
        meta = json.load(f)

    cameras =[]

    for idx , camera in enumerate(meta):
        focolength = camera['focal_length'] 
        R = Rotation.from_rotvec(camera['orientation']).as_matrix()
        t = np.array(camera['position'])[:, np.newaxis]
        w2c = np.concatenate((R, -np.dot(R, t)), axis=1)
        
        colmapR = w2c[:3, :3]
        T = w2c[:3, 3]
        colmapQ = rotmat2qvec(colmapR)

        cameras.append({
            'id': str(idx+1),
            'filename': camera['name']+'.png',
            'w': camera['width'],
            'h': camera['height'],
            'fx': focolength * float(scale),
            'fy': focolength * float(scale),
            'cx': camera['principal_point'][0],
            'cy': camera['principal_point'][1],
            'q': colmapQ,
            't': T,
        })

    write_colmap(path, cameras, offset)

I'm confused about whether to set all 'scale' to 0.5 or not. Since I downscale the pictures by 2 in 'def extractframes(videopath: Path, startframe=0, endframe=300, downscale=1, save_subdir = '', ext='png'):', directly set the 'focal_scale' to 0.5 will result undistortion pictures with 3/4 black areas. How should I set to get correct undistorted 'image_sclaed_2'? And whether should I change the scale (fx,fy,cx,cy,h,w ) when 'write_colmap'? I think these are the the reason caused my bad results.

Thanks again.