Platform Development • Re: I created a Python 3 port of UXP out of boredom...

I tried to make a build with the latest version of the patch "Follow-up: Fix 32-bit Windows builds" but it also ended with errors.

[1775203281.165765, "build_output", {"line": "\u040f\u0430\u0401\u00ac\u0490\u0437\u00a0\u00ad\u0401\u0490: \u045e\u0404\u00ab\u043e\u0437\u0490\u00ad\u0401\u0490 \u0434\u00a0\u00a9\u00ab\u00a0: c:\\pm_src\\platform\\media\\ffvpx\\libavutil\\tx_template.c"}] [1775203281.165765, "build_output", {"line": "mozavutil.dll"}] [1775203281.649366, "build_output", {"line": "Traceback (most recent call last):"}] [1775203281.649366, "build_output", {"line": " File "c:/pm_src/platform/config/expandlibs_exec.py", line 355, in "}] [1775203281.649366, "build_output", {"line": " exit(main(sys.argv[1:]))"}] [1775203281.649366, "build_output", {"line": " File "c:/pm_src/platform/config/expandlibs_exec.py", line 348, in main"}] [1775203281.649366, "build_output", {"line": " sys.stderr.write(stdout.decode(encoding='utf-8'))"}] [1775203281.649366, "build_output", {"line": "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x91 in position 3: invalid start byte"}] [1775203281.680566, "build_output", {"line": "mozmake.EXE[5]: *** [c:/pm_src/platform/config/rules.mk;765: mozavutil.dll] Error 1"}] [1775203281.680566, "build_output", {"line": "mozmake.EXE[5]: *** Deleting file 'mozavutil.dll'"}] [1775203281.680566, "build_output", {"line": "mozmake.EXE[4]: *** [c:/pm_src/platform/config/recurse.mk;71: media/ffvpx/libavutil/target] Error 2"}] [1775203283.92697, "build_output", {"line": "Traceback (most recent call last):"}] [1775203283.94257, "build_output", {"line": " File "c:/pm_src/platform/config/expandlibs_exec.py", line 355, in "}] [1775203283.94257, "build_output", {"line": " exit(main(sys.argv[1:]))"}] [1775203283.94257, "build_output", {"line": " File "c:/pm_src/platform/config/expandlibs_exec.py", line 348, in main"}] [1775203283.94257, "build_output", {"line": " sys.stderr.write(stdout.decode(encoding='utf-8'))"}] [1775203283.94257, "build_output", {"line": "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x91 in position 3: invalid start byte"}] [1775203283.9581702, "build_output", {"line": "mozmake.EXE[5]: *** [c:/pm_src/platform/config/rules.mk;765: icu78.dll] Error 1"}] [1775203283.9581702, "build_output", {"line": "mozmake.EXE[5]: *** Deleting file 'icu78.dll'"}] [1775203283.9581702, "build_output", {"line": "mozmake.EXE[4]: *** [c:/pm_src/platform/config/recurse.mk;71: config/external/icu/target] Error 2"}] [1775203283.9581702, "build_output", {"line": "mozmake.EXE[3]: *** [c:/pm_src/platform/config/recurse.mk;33: compile] Error 2"}] [1775203283.9581702, "build_output", {"line": "mozmake.EXE[2]: *** [c:/pm_src/platform/config/rules.mk;493: default] Error 2"}] [1775203283.9581702, "build_output", {"line": "mozmake.EXE[1]: *** [c:/pm_src/client.mk;406: realbuild] Error 2"}] [1775203283.97377, "build_output", {"line": "mozmake.EXE: *** [client.mk;164: build] Error 2"}] [1775203284.00497, "warning_summary", {"count": 142}]

Yes, what you're finding is pretty similar to what I'm finding. This is purely an encoding mismatch with Russian Windows I believe... and yeah, the way I tried to deal with this initially was to detect the system locale in the places where I know for sure it writes to the console, but that's proving not to be very reliable because it really seems like either a lot of people who speak other languages have their system's codepage set to CP1252 regardless and just have their native language as a langpack for MSVC itself, or else every check for the system codepage fails inside MSYS1 and it reports CP1252 regardless of what you have set.

About the only way I was able to get the right character display so far was to force the console into 1251 mode with chcp, and then hunt down every instance of "getpreferredencoding," and just hardcode 1251 instead... obviously that wouldn't be a good upstream fix, but until I think of something better, all I can really suggest is that people building in non-English languages might have to locally patch the build system's encoding detection to be hardcoded to their language's ANSI codepage (1252 for Russian, maybe 932 for Japanese, etc) and use chcp 1251 in MozillaBuild's .bat files on top of that to try and get the display in their own output language. Basically, this doesn't support Unicode (but does support ANSI codepages), and Python 3 assumes Unicode... that's the mismatch.

The intermediate solution... would be to detect the console's encoding rather than the system-wide Windows encoding, and use that. But that would still rely on the user having set that properly in the .bat file that runs MozillaBuild, because it probably isn't set correctly by default. I know for me, system console encoding was set to good-old DOS-based CP437 by default (which is what gave me a false positive on a test suggesting we had a pure ASCII environment earlier) even though the system itself was set to CP1252 by default.