Hi, I was searching for "spatiotemporal" when I came across this project. I would like to know if mae_st can be used for modeling spatiotemporal transcriptomics. Today I just read the stVCR paper, which focuses on reconstructing spatiotemporal dynamics. I feel that its task aligns with what mae_st aims to achieve.
I asked Hy3 preview model for answer: "facebookresearch/mae_st cannot be used as-is for a temporal transcriptomics reconstruction task — it is a video MAE codebase whose data assumptions are tightly coupled to spatiotemporal cubes (B, 3, T, H, W) (RGB video frames), with a 3D patch embedding that splits (tubelet, h, w) and positional embeddings tied to (T, H, W) grid dimensions. Single-cell or bulk temporal transcriptomics, by contrast, is typically a matrix (samples/cells, genes) with a time axis but no spatial H/W or RGB channels, so the tubelet embedder, decoder head, and mask scheduling would all need replacing. That said, the architectural backbone is reusable: you can keep the ViT encoder/decoder skeleton, the masking logic, and the reconstruction loss, but swap in a 1D gene-level linear projection (or treat each time point × cell as a token), redesign positional embeddings to encode (time, cell/group) or just (gene) + (time) separately, and drop the 3D convolutions. If your "temporal transcriptomics" data can be reshaped into pseudo-images (e.g., genes binned into 2D grids per time point), then the repo is closer to usable; otherwise a vanilla MAE (e.g., facebookresearch/mae) adapted to 1D is usually a simpler starting point than mae_st." I would like to know if this answer is feasible.
Hi, I was searching for "spatiotemporal" when I came across this project. I would like to know if mae_st can be used for modeling spatiotemporal transcriptomics. Today I just read the stVCR paper, which focuses on reconstructing spatiotemporal dynamics. I feel that its task aligns with what mae_st aims to achieve.
I asked Hy3 preview model for answer: "facebookresearch/mae_st cannot be used as-is for a temporal transcriptomics reconstruction task — it is a video MAE codebase whose data assumptions are tightly coupled to spatiotemporal cubes (B, 3, T, H, W) (RGB video frames), with a 3D patch embedding that splits (tubelet, h, w) and positional embeddings tied to (T, H, W) grid dimensions. Single-cell or bulk temporal transcriptomics, by contrast, is typically a matrix (samples/cells, genes) with a time axis but no spatial H/W or RGB channels, so the tubelet embedder, decoder head, and mask scheduling would all need replacing. That said, the architectural backbone is reusable: you can keep the ViT encoder/decoder skeleton, the masking logic, and the reconstruction loss, but swap in a 1D gene-level linear projection (or treat each time point × cell as a token), redesign positional embeddings to encode (time, cell/group) or just (gene) + (time) separately, and drop the 3D convolutions. If your "temporal transcriptomics" data can be reshaped into pseudo-images (e.g., genes binned into 2D grids per time point), then the repo is closer to usable; otherwise a vanilla MAE (e.g., facebookresearch/mae) adapted to 1D is usually a simpler starting point than mae_st." I would like to know if this answer is feasible.