RE: The setting to change the Language of the game to Any language

[JackTriton]

Hi, it's been a long time
This PR is redo of all of the functions I made almost 5 years ago

Description

With this PR, you can play OOTR with different languages or event texts / custom texts as well

---

Addition

Files

• language (folder): Consists every language you can play with
  Also, you can add another language file just by drag and drop the file with same structure
  For how to create and what file is included, checkout language / README.md
• Language.py: allows to manage language files (json, bin, ia4)

---

Additional Control Codes for wide characters

• All of the additional control codes are placed the code + 0x8700 (for example, silver rupee count is placed 0xF0 + 0x8700 = 0x87F0)
  For the character to use, check out Messages.py

---

Text alignment

• You can change the alignment of the text to Left, Center and Right

---

How to Change the language?

You can change the language by changing Main Rules >> Language >> Language Selection

---

Tips for language creation

If you want to add language with characters not included in vanilla game (for example Spanish has ñ), do the same thing as other images: decompile the original game, replace some files with new one and recompile / get diff with get_diff.py

---

Notes

Currently, besides English which is already implemented from beginning, it allows you to play with Japanese as well
Why Japanese? well, it's because Japanese is the other language that NTSC rom has and one of language that uses wide characters instead and also because I'm Japanese

Just because I didn't implemented other languages such as German, French or Chinese (which is PAL and iQue version has), that doesn't mean you can't play with those languages or other languages that only got the translated version by using mod / patch or even that never got the version

Not only that, you can create version for events, competition or even speedruns as well (like shortening all of the texts, and speed up everything)

Screen shots

[fenhl]

Preliminary review. I've already mentioned some of this on Discord but am repeating it here for reference.

• The hint area type being language-dependent is going to be an issue for parts of the codebase that don't directly interact with text and therefore don't care about the language. Currently, if I understand the code correctly, these just use an English-language version of the hint area type. This will cause issues if such a hint area value does end up in language-specific code and causes text to be generated in English instead of the correct language. I'd separate the language information from the hint area by passing it as a parameter to methods that generate text.
• Some of the variable names and magic strings (e.g. magic strings called o, wiw, and cbf; single-letter variable names in Language.format_from_text and Patches.create_fake_name) are difficult to understand at a first glance. I would recommend replacing them with longer, more descriptive names.
• There seem to be some pieces of logic that check if the language is English and use the logic for Japanese in the else case. The level of Japanese proficiency among maintainers and contributors is much lower than that of English, so for maintainability, English should be the fallback case.
• The PR adds .DS_Store files to the repo. These should be removed, and I recommend adding them to your .git/info/exclude to prevent accidentally adding them back in the future.
• The JSON file uses fixed-length (tuple-like) arrays in some places. To make it easier to read, I suggest replacing these with objects ({}) with descriptive property names.

I will take a closer look at specific parts of the code and go into more detail in a later review, once CI is green.

[JackTriton]

Most of the system reworked, deleted .DS_Store, add some descriptions to the property_build.py and changed some of the values to have clearer infos

[JackTriton]

Sorry there was some typos in japanese/property.json

[cjohnson57]

Some pretty major (but easily fixable) issues with the setting definition itself, adding this separately before I review the rest of it.

[cjohnson57]

Got different errors when trying to generate in English and Japanese.

English:

<Dungeon.Dungeon object at 0x000001EEE96AD480>
Traceback (most recent call last):
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\OoTRandomizer.py", line 57, in start
    main(settings)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 57, in main
    patch_and_output(settings, spoiler, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 375, in patch_and_output
    patch_cosmetics_log = prepare_rom(spoiler, world, rom, settings, rng_state, restore_rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 214, in prepare_rom
    patch_rom(spoiler, world, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Patches.py", line 1849, in patch_rom
    "dungeon_reward": world.language.hintTable[dungeon][1],
KeyError: <Dungeon.Dungeon object at 0x000001EEE96AD480>

Seems to be because the dungeon name has a control code in it. Also, pretty sure the the should be dropped and it should just be Deku Tree...

Japanese:

from_bytes() missing required argument 'byteorder' (pos 2)
Traceback (most recent call last):
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\OoTRandomizer.py", line 57, in start
    main(settings)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 57, in main
    patch_and_output(settings, spoiler, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 375, in patch_and_output
    patch_cosmetics_log = prepare_rom(spoiler, world, rom, settings, rng_state, restore_rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 214, in prepare_rom
    patch_rom(spoiler, world, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Patches.py", line 1176, in patch_rom
    update_message_by_id(messages, 0x305C, lang.format_from_id("PATCH_TEXTS.claim"), lang)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Messages.py", line 748, in update_message_by_id
    text = line_wrap(text, lang.base, align=lang.lang_property["align_text"])
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\TextBox.py", line 51, in line_wrap
    text_codes = Messages.parse_control_codes(text, lang)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Messages.py", line 345, in parse_control_codes
    text_bytes = encode_text_string(text) if lang else encode_text_string_jp(text)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Messages.py", line 295, in encode_text_string_jp
    result.append(int.from_bytes(h.encode("cp932")))
TypeError: from_bytes() missing required argument 'byteorder' (pos 2)

Adding byteorder="big" to the function call fixed it, but I'm not actually sure why that was necessary when the function definition already sets it to big by default? result.append(int.from_bytes(h.encode("cp932"), byteorder="big"))

def from_bytes(
            cls,
            bytes: Iterable[SupportsIndex] | SupportsBytes | ReadableBuffer,
            byteorder: Literal["little", "big"] = "big",
            *,
            signed: bool = False,
        )

@flagrama any ideas here?

Anyway, fixing that issue ran into the same issue English did with the dungeon name.

[JackTriton]

Got different errors when trying to generate in English and Japanese.

English:

<Dungeon.Dungeon object at 0x000001EEE96AD480>
Traceback (most recent call last):
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\OoTRandomizer.py", line 57, in start
    main(settings)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 57, in main
    patch_and_output(settings, spoiler, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 375, in patch_and_output
    patch_cosmetics_log = prepare_rom(spoiler, world, rom, settings, rng_state, restore_rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Main.py", line 214, in prepare_rom
    patch_rom(spoiler, world, rom)
  File "C:\Users\Caleb\Repos\OoT-Randomizer-Tests\Patches.py", line 1849, in patch_rom
    "dungeon_reward": world.language.hintTable[dungeon][1],
KeyError: <Dungeon.Dungeon object at 0x000001EEE96AD480>

LangDev here, this is probably because of the given value is dungeon instead of dungeon_reward

For the requested changes, moving the language_selection in SettingsList.py to below logic_rules would be great?

[cjohnson57]

Here's my review of the rest of the code. Admittedly mostly nitpicks, but a few actual issues.

Thanks so much for all your work on this!

[cjohnson57]

Thanks for your quick fixes on my first two messages! Can confirm they're resolved. I could launch the ROM and talk to NPCs and read hints in both English and Japanese.

[JackTriton]

Thanks for your quick fixes on my first two messages! Can confirm they're resolved. I could launch the ROM and talk to NPCs and read hints in both English and Japanese.

Regarding the last request, I'll edit them right now!

[cjohnson57]

One question, what exactly is the purpose of the property_build.py file? It seems to mainly contain the English messages that are now formatted in the English folder's property.json. Is it to generate the property.json?

[flagrama]

Adding byteorder="big" to the function call fixed it, but I'm not actually sure why that was necessary when the function definition already sets it to big by default? result.append(int.from_bytes(h.encode("cp932"), byteorder="big"))

FYI, the default value is just a change made in python 3.11. If you are using 3.11 or newer, this error won't appear. If you are using anything older this error comes up. If you are using 3.11 or newer but you have your dev environment set up to ensure everything works as far back as 3.8 it may be doing something additional to cause that error to show up.

[cjohnson57]

Interesting, ty for the info!

[JackTriton]

One question, what exactly is the purpose of the property_build.py file? It seems to mainly contain the English messages that are now formatted in the English folder's property.json. Is it to generate the property.json?

property_build.py creates property.json for the language
The default value would be the ones on English for compare / know what string to use

Also, the PLANE_TEXTS would be the ones for other texts to implement in game when the language is not on NTSC or any kind of variety

[cjohnson57]

Gotcha. Should that be "plain" texts?

[JackTriton]

Gotcha. Should that be "plain" texts?

Oh, sorry yes PLAIN_TEXTS

[cjohnson57]

Thanks so much for all your changes! It's good now as far as I can tell.

Before merging, we should also get approval from Fenhl, and probably more testing of full JP seeds.

[JackTriton]

@cjohnson57 Thanks for the detailed reviews!

[JackTriton]

Add UnitTest that checks language file / properties

Below this is template for the errors

If the language file misses property.json
{lang_name}: property.json is not included

If some properties are missing / wrong in property.json
{lang_name}: some properties are wrong in property.json

If some bin files are not included in the bin_patch.json
{lang_name}: some non-wanted bin files are included

[JackTriton]

Unless the conflicts revolve with in-game messages or custom texts, all conflicts will be resolved after merging

[fenhl]

Might make sense to document that in a new file in the Notes directory.

[JackTriton]

Currently I'm working on the Note file and some adjustment for word gender properties
The next update should be the last one besides the adjustment for the Sharing system

[JackTriton]

@fenhl Added Notes for both texture gathering and syntax
I also added some extra syntaxes for Language for future Language updates
I also finished with the first version of the German language for both textures and texts

[fenhl]

I would recommend keeping this PR focused on Japanese for now, and handling any changes necessary to support more languages later in separate PRs. The bulk of this PR has already been reviewed and tested, and making changes to support other languages here as well risks getting it stuck in the review process.

[JackTriton]

@fenhl Unfortunately there is one small problem within the last 2 updates (revolving area in update_map_messages)
I'll fix them right away

[JackTriton]

@fenhl Ok, now I can confirm that all of the issues are resolved
Sorry for the constant updates while reviewing / pre-reviewing process

[JackTriton]

@fenhl How's the test going?
It's been 2 months since the last messages
I hope this repo to be implemented some day

Currently, I'm doing some implementations for other languages such as German, French and Spanish
Texture files for German and French is completed but needs some further adjustments revolving texts in order to make it as official for future project (after JP implementation)

For French, I found that some characters aren't implemented in the official game such as œ and î so I replaced some german characters to them
This allows additional text refinements

Also, I found that in the Messages.py around L94, there's one character that is listed wrongly
î isn't supposed to be listed and this should be Á instead
I don't know whether this is intentional or not but if we were to implement French in the future, this should be fixed

[fenhl]

Sorry, between multiple maintainers being on vacation and a critical issue coming up that needed to be handled before the upcoming 9.1 release, there hasn't been much time spent on this. As far as testing is concerned, this PR is ready to go, and the racing impact has been deemed acceptable, but we haven't reached a consensus on how it should interact with the settings string or how it should be presented in the GUI.

[JackTriton]

@fenhl I have one solution regarding the racing impact and that is to implement SHA256 (or similar hashing tech) for the language
This allows not only to confirm what language is being used but also confirm whether they are using right language or not

Calculate SHA256 over the language file (for example English's property.json), and get the string out of it
We compare this string with the list that has all of the language that is legal to use in racing

For placement of GUI, I think under or beside of the Enable Plandomizer / Cosmetic Plandomizer switch is great

[fenhl]

Given that the racing impact has been deemed acceptable, I don't think we should implement any sort of verification mechanism. It would only add to the overhead for racers and race mods.

[cjohnson57]

Also, I found that in the Messages.py around L94, there's one character that is listed wrongly
î isn't supposed to be listed and this should be Á instead
I don't know whether this is intentional or not but if we were to implement French in the future, this should be fixed

I remember running into this before, and unfortunately as far as I remember this is actually how it is implemented in the game. You can see on cloudmodding: https://wiki.cloudmodding.com/oot/Text_Format

[flagrama]

Weird that there is no reference to this in decomp from what I can see. However it does seem like rando has it correct for the way it was implemented in game to me too.

[cjohnson57]

Weird that there is no reference to this in decomp from what I can see. However it does seem like rando has it correct for the way it was implemented in game to me too.

I don't think the actual accented characters would be listed anywhere in the codebase, just the character IDs corresponding to textures, which wouldn't be present in decomp

[JackTriton]

Weird that there is no reference to this in decomp from what I can see. However it does seem like rando has it correct for the way it was implemented in game to me too.

I don't think the actual accented characters would be listed anywhere in the codebase, just the character IDs corresponding to textures, which wouldn't be present in decomp

@cjohnson57
I think I know what's happening
I searched through the decompiled file for both NTSC and GC EU MQ Debug and found out that for gMsgChar81LatinCapitalLetterIWithCircumflexTex.i4, Á is listed as its texture on NTSC rom but for GC EU MQ Debug rom however, the listed character was Î instead so I think they already patched the problem for NTSC roms

[flagrama]

Weird that there is no reference to this in decomp from what I can see. However it does seem like rando has it correct for the way it was implemented in game to me too.

I don't think the actual accented characters would be listed anywhere in the codebase, just the character IDs corresponding to textures, which wouldn't be present in decomp

I believe assets/text/charmap.nes.txt should have it documented, and technically there is a bug with the chamap being used for both N64 and GameCube builds, but the game just happens to not use it, or happily just displays the wrong texture for some other reason, so it doesn't cause a mismatch when compiled.

[cjohnson57]

Okay we talked about it a bit. We think that the language setting should not affect settings string, so it acts sort of like a cosmetic setting. Along with this it should not change item placement either to be clear.

[JackTriton]

@cjohnson57 @fenhl Should I include these changes as well?

By the way, for the newest Messages.py ID Adding these will resolve them:

Messages.py (No changes compared with the current ReLanguage one): (L212) - (0x9097, "\x08\x13\x2EYou got a \x05\x41Chicken, \x05\x40one\x01of Anju's prized hens! It fits \x01in your pocket."),

data/lang/property_build.py:

+ {
+        "id": 0x9097,
+        "text": "\x08\x13\x2EYou got a \x05\x41Chicken, \x05\x40one\x01of Anju's prized hens! It fits \x01in your pocket.",
+    },

- # 0x9097 and 0x9098 unused
+ # 0x9098 unused

data/lang/English/property.json:

+ {
+        "id": 37015,
+        "text": "\b\u0013.You got a \u0005AChicken, \u0005@one\u0001of Anju's prized hens! It fits \u0001in your pocket."
+    },

data/lang/Japanese/property.json:

+ {
+        "id": 37015,
+        "text": "♂★2E#01コッコ#00を見つけた！&アンジュさんのコッコだ！&手の平に収まる。"
+    },

[cjohnson57]

@cjohnson57 @fenhl Should I include these changes as well?

Why are these changes needed?

[JackTriton]

@cjohnson57 This is the message which is added in the newest version of Messages.py
In order to implement this branch into the main stream, this must be included

[cjohnson57]

@JackTriton Oh sorry I misunderstood why you were posting that. Yes, please do include it.

[JackTriton]

@fenhl @cjohnson57 Both cosmetic settings and 0x9097 entry are applied