Time independent skulltulas by mracsys · Pull Request #2332 · OoTRandomizer/OoT-Randomizer

mracsys · 2024-11-18T03:03:01Z

Implements #2356.

Two parts to this PR. There is the skulltula time of day feature, and a new framework to enable arbitrary changes to any of the scene or room files parsed to linked python objects. The framework automatically shifts all pointers within the files to handle new file locations and additions/deletions within the file. It will have multiple applications besides the skulltula feature:

Detailed cutscene parsing instead of writing raw bytes (WIP). Also more easily allows writing completely new cutscenes.
Adding any actor/object to any room, such as a sign outside Ganon’s castle for the bridge condition or anywhere else a hint might be helpful
Entrance/exit additions, such as promoting grottos to the same framework as other entrances, or new loading planes for mixed pool door rando
Space reclamation. Almost a MB is wasted in padding between scene and room files.
More user friendly interface for MQ patching, and opens the door for other alternate scene options for anyone crazy enough to attempt it.
Collision changes for problems like Reduce boss doors side range to prevent voiding out in Forest temple #2331 and for my vision of door rando without scene reloads between doors.

More details on the design of the parser is included in Notes/scene-and-room-patching.md.

GUI Impacts for Web

TL;DR, Creating Patch File is added as a progress bar milestone between Patching ROM and Starting compression.

Due to the increased time spent patching the rom caused by the significantly larger amount of changed bytes compared to the current randomizer, an additional status message milestone is added to the GUI progress bar: Creating Patch File. This was previously included in the Patching ROM milestone. Additionally, as these milestones alternate for each player in multiworld generation, the progress percentages for these two milestones were modified from a static number to one that scales based on the currently processing world. This requires resetting currentWorld after successful patching for subsequent generations to work when Generation Count is greater than 1.

Python Virtual Environment Addition

This PR makes use of the numpy package to more efficiently process changed bytes for patch generation compared to base python lists and dictionaries. While the randomizer has avoided external dependencies thus far, using numpy reduces the performance penalty substantially (times are from single world runs on an AMD 5900X with the S8 Tournament preset):

Task	Penalty with numpy	Penalty without numpy
ROM Generation	25% (1.6s)	66% (4.3s)
ZPF Generation	168% (8.4s)	278% (13.9s)

A virtual environment is automatically created and activated by the major entry points to the randomizer:

GUI - GUI.py
Console - OoTRandomizer.py
CI - CI.py

Each of these entry points independently check if they are running in a virtual environment located in subfolder .venv at the local path. If the environment doesn't exist, it is created automatically. There is also check if all dependencies in requirements.txt are fulfilled via pip. If not fulfilled, they are installed automatically in the virtual environment. If running in a release build, any pip errors are captured and passed to the regular error popup. Otherwise, running via the scripts logs errors to the console before exiting. If venv and pip checks are successful, but either the current python environment is not the virtual environment, or new dependencies were added, the script relaunches itself via the subprocess module in the virtual environment with all arguments intact. This behavior can be added to additional entry points or custom scripts by importing the ensure_venv() function from Utils.py and adding it as the first line wherever it's needed. See gui_main(), start(), and the top level of CI.py for example usage.

numpy is pinned between the last version supported by python 3.8 (1.24.4) and the version in use on RealRob's custom_voice_packs branch (2.2.1). Both versions have been tested to work. For compatibility with RealRob's work, I strongly recommend increasing the minimum python version for the randomizer to 3.10, the earliest supported by numpy 2.2.1. 3.8 is EOL, and 3.9 will be EOL in October 2025. This PR does not change the minimum python version.

Unrelated Bugfixes

Fix a typo in an exception for wad file generation (see

OoT-Randomizer/Main.py

Line 267 in a821705

raise FileNotFoundError(f'Invalid path to Base WAD: "{input_file}"')

)
Fix Rom.copy() handling of the original property. On copy, the new object's original would refer to itself instead of the unmodified rom buffer. This was causing problems re-using the patched rom before cosmetics to output a rom and zpf at the same time. This isn't a problem with the current randomizer as it always fully repatches the rom for every output. The same issue also applied to the overlay_table, but that had no ill effects.

Todo

Test compressed roms. ~~DMA changes for compressor?~~
Update pointers outside scene files to globals in them, mainly cutscenes.
If 2 doesn’t fix problems moving file start addresses, figure that out
Integrate framework with rando patching pipeline
Move cutscene patches to framework
Move MQ patching to framework
Move any other scene changes to framework
Add setting for skulltulas ignoring time of day. Add actors/objects where needed and patch all actor parameters.

Rando Patches to Test with New System

r0bd0g · 2024-11-18T07:07:52Z

I guess for logic you'd go around to each skulltula you affected that checks for night and change at_night to (at_night or [setting]). Something to look out for is the watchtower, which will no longer be climbable in child day without a means of killing the skulltula. In closed forest, the time of day access check is skipped, so there's special logic on the night skull there to confirm you can actually get it to night somehow, and you'd also have to add the new setting as an alternative for that.

mzxrules · 2025-03-11T05:58:12Z

+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[0].headers[0].actor_list.actors[4],
+        # GTG Stalfos Room
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[1].headers[0].actor_list.actors[2],
+        # GTG Flame Wall Maze
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[2].headers[0].actor_list.actors[15],
+        # GTG Pushblock Room
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[3].headers[0].actor_list.actors[10],
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[3].headers[0].actor_list.actors[11],
+        # GTG Rotating Statue Room
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[4].headers[0].actor_list.actors[3],
+        # GTG Megaton Statue Room
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[5].headers[0].actor_list.actors[18],
+        # GTG Lava Room
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[6].headers[0].actor_list.actors[17],
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[6].headers[0].actor_list.actors[18],
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[6].headers[0].actor_list.actors[19],
+        # GTG Dinolfos Room
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[7].headers[0].actor_list.actors[14],
+        # GTG Ice Arrow Room
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[8].headers[0].actor_list.actors[4],
+        # GTG Shellblade Room
+        scenes[SceneIDs.GERUDO_TRAINING_GROUND].rooms[9].headers[0].actor_list.actors[22],
+        # Death Mountain Crater
+        scenes[SceneIDs.DEATH_MOUNTAIN_CRATER].rooms[1].headers[2].actor_list.actors[44],
+        # Thieves' Hideout Green Cell Room 3 torches
+        scenes[SceneIDs.THIEVES_HIDEOUT].rooms[1].headers[0].actor_list.actors[9],
+        # Thieves' Hideout Red Cell Room 1 torch
+        scenes[SceneIDs.THIEVES_HIDEOUT].rooms[2].headers[0].actor_list.actors[9],
+        # Thieves' Hideout Green Cell Room 4 torches
+        scenes[SceneIDs.THIEVES_HIDEOUT].rooms[4].headers[0].actor_list.actors[11],
+        # Thieves' Hideout Blue Cell Room 2 torches
+        scenes[SceneIDs.THIEVES_HIDEOUT].rooms[5].headers[0].actor_list.actors[14],
    ]
-    for address in wonder_talk2_y_coordinates:
-        rom.write_byte(address, 0xFB)
+
+    # Pre-scene-framework hack replaced the first byte with 0xFB
+    bit_mask1 = int.from_bytes(b'\xFB\x00', 'big', signed=True)
+    bit_mask2 = int.from_bytes(b'\xFB\xFF', 'big', signed=True)
+    for actor in wonder_talk2_actor_entries:
+        actor.pos.y = (actor.pos.y | bit_mask1) & bit_mask2

    if 'frogs2' in settings.misc_hints:
        # Prevent setting the replaced textbox flag so that the hint is easily repeatible by walking over the spot again.
        # And move the hint spot down the log so that it doesn't pop every time a song is played, and let some room to do ocarina item glitch.
-        rom.write_int16s(0x2059412, [0x03C0, 0x00E2, 0xFAA6]) # Move coordinates. Original value : 1000, 205, -1202. New value : 960, 226, -1370.
-        rom.write_byte(0x205941F, 0xBF) # Never set the flag.
+        actor = scenes[SceneIDs.ZORAS_RIVER].rooms[0].headers[0].actor_list.actors[58]
+        actor.pos = Vec3s(960, 226, -1370) # original 1000, -205, -1202
+        actor.params = 0x4BBF # original 0x4BBB
+
+
+# Gaps in IDs are intentional
+# https://github.com/zeldaret/oot/blob/7235af2249843fb68740111b70089bad827a4730/include/z64cutscene.h#L35-L165
+class CutsceneCommandID(IntEnum):


man, rando sure is a hot mess when it comes to organization.

I find it kind of odd to just stuff these important type definitions after a bunch of function patching routines split out of patches.py. I think we need to consider moving this type access stuff into a new folder like "ztypes" or something

You're right, I should have put more effort into structure. Some of the other utility classes like Vec3s and Vec3i that didn't fit well got put into Rom.py despite not fitting there either. I can refactor into the same file/folder structure as decomp for structs lifted from there, with a top level folder to keep it neat.

r0bd0g · 2025-03-12T23:19:22Z

I warned you about special logic that would be needed for KF GS Know It All House! As you have it currently, in closed forest, if you start at daytime, under this new setting the skulltula will be there but you won't have logic to collect it before beating deku or finding sun's song. That skull has an extra check on it to make sure it's actually present since closed forest breaks the assumption that you will always have access to moving time of day from spawn, so you need to add the new setting as an alternative way to have the time of day to access it.

(This isn't really related to your PR, you shouldn't change anything about this, but I wonder if all of those skulltulas that check for ToD in areas where time moves already, should just have the time of day check removed. It was nice to look at before but with the new setting it's become kind of bloated. I made a choice along those lines with Peahats when reviewing enemy drop shuffle b/c separate night and day logic for them was pretty convoluted for no real benefit.)

I don't know enough about glitched to say whether or not the logic is wrong absolutely for sure or not, but at a glance I don't trust it. For example, man on roof, you check for a weapon to kill the skulltula, but you always are checking it regardless of if the skulltula is potentially there at daytime. You should get a glitched logic expert to review it, or make no glitched logic changes and mark the settings as incompatible.

(Unrelated to your PR but I was glancing over the night skulls in glitched logic and noticed some logic for buying the Giant's Knife that is assuming that the Knife isn't shuffled. This shows the dangers of marking new settings as glitched logic compatible without getting a specialist's review!)

fenhl · 2025-03-23T17:25:02Z

Should this setting be enabled in the Easy Mode preset?

mracsys · 2025-03-23T17:47:09Z

It would make sense. Would that need to go through some sort of vote or poll?

fenhl · 2025-03-23T17:59:38Z

Current dev team call is in favor so I'd say you can just add it.

shirosoluna · 2025-03-24T19:01:20Z

@r0bd0g >

I don't know enough about glitched to say whether or not the logic is wrong absolutely for sure or not, but at a glance I don't trust it. For example, man on roof, you check for a weapon to kill the skulltula, but you always are checking it regardless of if the skulltula is potentially there at daytime. You should get a glitched logic expert to review it, or make no glitched logic changes and mark the settings as incompatible.

(Unrelated to your PR but I was glancing over the night skulls in glitched logic and noticed some logic for buying the Giant's Knife that is assuming that the Knife isn't shuffled. This shows the dangers of marking new settings as glitched logic compatible without getting a specialist's review!)

I wouldnt worry too much about glitched logic right now, because it's being updated entirely to have all settings included. There are way more important stuff that isnt compatible at all currently and in released versions. If this is going to be slated for 9.0 as well then we can make it incorporated. I dont know the timing for this PR.
Just offering support to not be concerned about glitched logic too too much atm

fenhl · 2025-03-24T19:09:58Z

There's not really any timing for this PR, it will be merged once it's been code reviewed and sufficiently tested. Since it's a rather large change, that might take a while, but so is glitched logic 3.0, so it's hard to say which is more likely to be merged first. I'd recommend not including the changes for this in glitched logic 3.0 for now or perhaps maintaining them separately.

shirosoluna · 2025-03-27T21:25:49Z

There's not really any timing for this PR, it will be merged once it's been code reviewed and sufficiently tested. Since it's a rather large change, that might take a while, but so is glitched logic 3.0, so it's hard to say which is more likely to be merged first. I'd recommend not including the changes for this in glitched logic 3.0 for now or perhaps maintaining them separately.

wasn't meaning that it would wait for it but that I'd incorporate it into logic when the time comes. Even if it's after being pulled.
They are maintained separately. But if new glitch logic comes later then still dont bother worry about current glitched logic. It's too incompatible with everything currently and one more setting wouldnt make a difference lol.

fenhl · 2025-06-03T00:03:07Z

I'll see if we can get a web generator for this branch set up to encourage more testing.

mracsys · 2025-06-05T01:07:13Z

FYI, I don't have much free time through at least fall, possibly into next year. I'm still around, but changes might be delayed.

cjohnson57

No real blockers, just a couple small things I noticed. Though, I feel a little impotent in my ability to review this properly, given the scope and the fact that there's thousands of lines of transliterated game data that I can't really have an opinion on.

I kind of wonder if it's worth removing the nighttime skulls expect sun song option now... It feels a bit vestigial. Doesn't do the same thing obviously, but fulfills the role of "I don't want to mess with waiting for night for skulls)

Also, I see this is still tagged as needs testing, is this actually still the case?

cjohnson57 · 2026-08-10T17:42:01Z


    # Jabu Jabu swallowing Link
-    patch_cutscene_destination_and_length(rom, 0xCA0784, 1)
+    jabu_swallow_cutscene = Cutscene.decode(rom, 0xC9FC84)


Not something that needs to be fixed really but it would be nice if these "magic numbers" could be stored in a table too, like how many of the others have been eliminated by these changes

Agreed it could be better documented. Cutscenes are stored all over the place in the codebase with no definitive centralized list unfortunately. This one is in the Jabu actor file for example and only called by it.

The cutscene objects try to minimize the magic VROM locations to just the start of the cutscene. Those are easy to line up with the decomp ntsc-1.0 map file for the cutscene variable address. In the future, maybe there could be a writeup at the top of the file on how to do that.

cjohnson57 · 2026-08-10T17:59:23Z

+    def read_s8(self, address: Optional[int] = None) -> int:
+        if address is None:
+            address = self.last_address
+        return int.from_bytes(self.read_bytes(address, 1), 'big', signed=True)
+
+    def read_s16(self, address: Optional[int] = None) -> int:
+        if address is None:
+            address = self.last_address
+        return int.from_bytes(self.read_bytes(address, 2), 'big', signed=True)
+
+    def read_s32(self, address: Optional[int] = None) -> int:
+        if address is None:
+            address = self.last_address
+        return int.from_bytes(self.read_bytes(address, 4), 'big', signed=True)


Is there a particular reason not to make these one function with the size as a param? (Though that being said, I think I might have done the same thing in my model code)

This is pretty standard when working with binary data. Having to specify the size as a parameter just makes each call harder to read because you have to think about which one is the size parameter and may have to refer to the function definition, but by having separate calls for each size makes it obvious. Especially with these being for signed bytes I think it is very important to make it absolutely obvious what the function is doing.

Basically what flag said. With how often these are called in the parsing functions, it's more ergonomic to do rom.read_s16(address) than something like rom.read_bytes_to_int(address, length, signed=True). Also makes it more obvious in the calling code what the C type of the data is supposed to be.

fenhl · 2026-08-10T21:53:08Z

Also, I see this is still tagged as needs testing, is this actually still the case?

No, I consider this to be sufficiently tested after the recent full-game test. I guess I forgot to remove the label.

mracsys · 2026-08-10T22:51:26Z

Also, I see this is still tagged as needs testing, is this actually still the case?

No, I consider this to be sufficiently tested after the recent full-game test. I guess I forgot to remove the label.

To be clear, I've had multiple full-game tests of this on console and emulator for a while in addition to all the individual cases checked off in the PR description. Always preferable to have independent testing though.

fenhl · 2026-08-10T23:14:01Z

…I was not aware of that.

mracsys · 2026-08-10T23:19:32Z

I'm not sure I ever said that to be fair, looking back at the description.

fenhl · 2026-08-10T23:21:12Z

Oh well, like you said, doesn't hurt to have independent confirmation too.

fenhl mentioned this pull request Nov 18, 2024

Make grotto entrances act more like normal entrances #2023

Open

fenhl linked an issue Mar 1, 2025 that may be closed by this pull request

Time Dependant Skulls Always Appear #2356

Open

mracsys force-pushed the time_independent_skulltulas branch from 40b8106 to 4c1fc19 Compare March 9, 2025 02:22

mracsys marked this pull request as ready for review March 11, 2025 01:13

mzxrules reviewed Mar 11, 2025

View reviewed changes

fenhl removed the Status: Under Consideration Developers are considering whether to accept or decline the feature described label Mar 23, 2025

fenhl mentioned this pull request Jun 2, 2025

Boulder Shuffle #2422

Open

fenhl requested changes Jun 2, 2025

View reviewed changes

Comment thread ASM/src/hacks.asm Outdated

Comment thread Patches.py Outdated

Comment thread ProcessActors.py Outdated

fenhl added the Status: Waiting for Author Changes or response requested label Jun 3, 2025

fenhl approved these changes Jun 5, 2025

View reviewed changes

fenhl added Status: Waiting for Maintainers and removed Status: Waiting for Author Changes or response requested Status: Needs Review Someone should be looking at it labels Jun 5, 2025

mracsys added 20 commits June 6, 2026 13:07

add auto venv creation to handle numpy dependency

ac246c1

fix try catch exception on numpy import

61f32a9

fix missed scene ID refactor from merge

849f899

always check requirements.txt on launch

455d743

add venv python file extension for windows

c2be976

fix unit test unknown argument from venv system

516d858

fix unknown cmd line args error

3c826da

only patch functional rom changes once with separate cosmetics

9493091

fix rom copy bugs, add zpf GUI status milestone

ea35a76

add player number to patch file GUI status message

7aa51d7

consolidate GUI milestones and handle multiworld status

4520e45

relax numpy version requirements for python 3.8

5c7746a

add terminator bytes to normal and cullable room shapes

2e83524

align file end in DMA, not just start

2aa80e4

check venv earlier to preserve stdin for settings

07dba88

fix copy/paste error on cullable mesh header end address

e5c5cd5

fix error detection for pip

0626c57

move venv launch function to utils

70aa783

convert prints to logs

a8e755f

rebuild payload after rebase

1af8bc0

mracsys force-pushed the time_independent_skulltulas branch from 30876af to 1af8bc0 Compare June 6, 2026 17:16

flagrama mentioned this pull request Aug 1, 2026

Add daytime Gold Skulltulas option #2612

Closed

cjohnson57 approved these changes Aug 10, 2026

View reviewed changes

fenhl removed the Status: Needs Testing Probably should be tested label Aug 10, 2026

fenhl added the Status: Waiting for Maintainers label Aug 10, 2026

Uh oh!

Conversation

mracsys commented Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GUI Impacts for Web

Python Virtual Environment Addition

Unrelated Bugfixes

Todo

Rando Patches to Test with New System

Uh oh!

r0bd0g commented Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mzxrules Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mracsys Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

r0bd0g commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fenhl commented Mar 23, 2025

Uh oh!

mracsys commented Mar 23, 2025

Uh oh!

fenhl commented Mar 23, 2025

Uh oh!

shirosoluna commented Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fenhl commented Mar 24, 2025

Uh oh!

shirosoluna commented Mar 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fenhl commented Jun 3, 2025

Uh oh!

mracsys commented Jun 5, 2025

Uh oh!

cjohnson57 left a comment

Choose a reason for hiding this comment

Uh oh!

cjohnson57 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

mracsys Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

cjohnson57 Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

flagrama Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

mracsys Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

fenhl commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mracsys commented Aug 10, 2026

Uh oh!

fenhl commented Aug 10, 2026

Uh oh!

mracsys commented Aug 10, 2026

Uh oh!

fenhl commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

mracsys commented Nov 18, 2024 •

edited

Loading

r0bd0g commented Nov 18, 2024 •

edited

Loading

mzxrules Mar 11, 2025 •

edited

Loading

r0bd0g commented Mar 12, 2025 •

edited

Loading

shirosoluna commented Mar 24, 2025 •

edited

Loading

fenhl commented Aug 10, 2026 •

edited

Loading