Problem building 3325 on windows machines

[Phylanx]

Hi!

We tried automating our CEF build (version 3325) with jenkins and hang on sporadic build fails that all look like this (containing this "WindowsError: [Error 145] " with different temp folder names):

Code: Select all

00:10:53.889 [2253/33941] C:/Jenkins/workspace/Chromium_3325_CEF/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../mojo/public/tools/bindings/mojom_bindings_generator.py --use_bundled_pylibs generate ../../device/geolocation/public/interfaces/geoposition.mojom -d ../../ -I ../../ -o gen --bytecode_path gen/mojo/public/tools/bindings --scrambled_message_id_salt_path ../../chrome/VERSION --allow_native_structs -g c++,java --depfile gen/device/geolocation/public/interfaces/interfaces__generator_geoposition.d --depfile_target gen/device/geolocation/public/interfaces/geoposition.mojom.cc --typemap gen/device/geolocation/public/interfaces/interfaces__type_mappings --use_once_callback
00:10:53.889 FAILED: gen/device/geolocation/public/interfaces/geoposition.mojom.cc gen/device/geolocation/public/interfaces/geoposition.mojom.h gen/device/geolocation/public/interfaces/geoposition.mojom.srcjar
00:10:53.889 C:/Jenkins/workspace/Chromium_3325_CEF/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../mojo/public/tools/bindings/mojom_bindings_generator.py --use_bundled_pylibs generate ../../device/geolocation/public/interfaces/geoposition.mojom -d ../../ -I ../../ -o gen --bytecode_path gen/mojo/public/tools/bindings --scrambled_message_id_salt_path ../../chrome/VERSION --allow_native_structs -g c++,java --depfile gen/device/geolocation/public/interfaces/interfaces__generator_geoposition.d --depfile_target gen/device/geolocation/public/interfaces/geoposition.mojom.cc --typemap gen/device/geolocation/public/interfaces/interfaces__type_mappings --use_once_callback
00:10:53.889 Traceback (most recent call last):
00:10:53.889   File "../../mojo/public/tools/bindings/mojom_bindings_generator.py", line 397, in <module>
00:10:53.889     sys.exit(main())
00:10:53.889   File "../../mojo/public/tools/bindings/mojom_bindings_generator.py", line 393, in main
00:10:53.889     return args.func(args, remaining_args)
00:10:53.889   File "../../mojo/public/tools/bindings/mojom_bindings_generator.py", line 281, in _Generate
00:10:53.889     processor.ProcessFile(args, remaining_args, generator_modules, filename)
00:10:53.889   File "../../mojo/public/tools/bindings/mojom_bindings_generator.py", line 170, in ProcessFile
00:10:53.889     RelativePath(filename, args.depth))
00:10:53.889   File "../../mojo/public/tools/bindings/mojom_bindings_generator.py", line 221, in _GenerateModule
00:10:53.889     generator.GenerateFiles(filtered_args)
00:10:53.889   File "C:\jenkins\workspace\Chromium_3325_CEF\target\chromium\src\mojo\public\tools\bindings\generators\mojom_java_generator.py", line 533, in GenerateFiles
00:10:53.889     build_utils.ZipDir(zip_filename, temp_java_root)
00:10:53.889   File "C:\Jenkins\workspace\Chromium_3325_CEF\depot_tools\win_tools-2_7_6_bin\python\bin\lib\contextlib.py", line 24, in __exit__
00:10:53.889     self.gen.next()
00:10:53.889   File "C:\jenkins\workspace\Chromium_3325_CEF\target\chromium\src\mojo\public\tools\bindings\generators\mojom_java_generator.py", line 406, in TempDir
00:10:53.889     shutil.rmtree(dirname)
00:10:53.889   File "C:\Jenkins\workspace\Chromium_3325_CEF\depot_tools\win_tools-2_7_6_bin\python\bin\lib\shutil.py", line 256, in rmtree
00:10:53.889     onerror(os.rmdir, path, sys.exc_info())
00:10:53.889   File "C:\Jenkins\workspace\Chromium_3325_CEF\depot_tools\win_tools-2_7_6_bin\python\bin\lib\shutil.py", line 254, in rmtree
00:10:53.889     os.rmdir(path)
00:10:53.889 WindowsError: [Error 145] Das Verzeichnis ist nicht leer: 'c:\\users\\johann~1.sch\\appdata\\local\\temp\\tmpa7rqq6'
00:10:53.909 [2254/33941] C:/Jenkins/workspace/Chromium_3325_CEF/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../mojo/public/tools/bindings/mojom_bindings_generator.py --use_bundled_pylibs generate ../../device/geolocation/public/interfaces/geoposition.mojom -d ../../ -I ../../ -o gen --bytecode_path gen/mojo/public/tools/bindings --scrambled_message_id_salt_path ../../chrome/VERSION --allow_native_structs -g javascript
00:10:53.937 [2255/33941] C:/Jenkins/workspace/Chromium_3325_CEF/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../mojo/public/tools/bindings/mojom_bindings_generator.py --use_bundled_pylibs generate ../../device/geolocation/public/interfaces/geolocation.mojom -d ../../ -I ../../ -o gen --bytecode_path gen/mojo/public/tools/bindings --scrambled_message_id_salt_path ../../chrome/VERSION --allow_native_structs --generate_non_variant_code -g c++ --depfile gen/device/geolocation/public/interfaces/interfaces_shared__generator_geolocation.d --depfile_target gen/device/geolocation/public/interfaces/geolocation.mojom-shared-internal.h
00:10:54.035 [2256/33941] C:/Jenkins/workspace/Chromium_3325_CEF/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../build/toolchain/win/tool_wrapper.py stamp obj/device/screen_orientation/public/interfaces/interfaces__generator.inputdeps.stamp


The build script runs in multiple steps:
[*] initialize environment variables (such as paths, versions,...)
[*] sync of cef sources (with automate-git.py --download-dir=c:\my\path\to\cef --depot-tools-dir=c:\my\path\to\depot_tools --branch=3325 --checkout=7c74b17 --no-debug-build --no-release-build --no-distrib)
[*] apply our patches (some xcopy calls)
[*] delete chromium/src/out folder if it exists (rmdir statement)
[*] compile sources (with automate-git.py --download-dir=c:\my\path\to\cef --depot-tools-dir=c:\my\path\to\depot_tools --branch=3325 --checkout=7c74b17 --force-distrib --no-update --no-depot-tools-update --verbose-build --no-debug-build --force-build)
[*] publish results (7z them and put them on a network share)

The scripts sometimes also finish successfully and the binaries are as expected.
But - as said - sometimes the build fails with the described WindowsError.

All that I've found for now is to manually delete this folder, but that is no option for an automatic jenkins build.
Is this a known problem?
I never had this problem with version 2704, can it have something to do with the "geolocation" feature where mojo stuff has been generated before and after (mojom_bindings_generator.py)?

What are these MOJOMs used for? Just for test execution? Can I skip the mojom_bindings_generator invokation?

[Czarek]

The "mojom_java_generator.py" script makes calls to shutil.rmtree which is not reliable on Windows, as it is vulnerable to race conditions. The solution is to pass "ignore_errors=True" second argument to that function. See this comment:

https://bitbucket.org/chromiumembedded/ ... t-32231089

Source:
https://cs.chromium.org/chromium/src/mo ... &g=0&l=414

[Phylanx]

Thank you Czarek for your fast reply.
Our own analysis went in the same direction, unfortunately I did only a search for this error in the forums, not in the issues.

Our current state:
We suspect our AntiVirus software to have a lock on the tmp directory, so we will not add this "ignore_errors=True".
Our current attempt: Changing the TEMP folder for this Jenkins Build from the default windows Temp folder (%user%/appdata/....) to the jenkins workspace folder that will not be scanned by the AntiVirus.
Currently we are waiting for the results of this build, we'll keep you on track.

[Czarek]

shutil.rmtree is a general OS issue. I've been encountering this issue many times on a computer that has no AV installed and when using any directories.

[Phylanx]

ok, we'll keep that in mind.
thx again for your help, it is highly appreciated!

[ndesktop]

We suspect our AntiVirus software to have a lock on the tmp directory, so we will not add this "ignore_errors=True".

Under any circumstances, do *not* enable antivirus on build directories and processes.
Especially, if you can control features such as "on access", "realtime protection" and such, at least exclude locations and processes involved.
Since the build touched %TEMP%, which is a prime target for dropping malware, the AV jumped probably ten meters up.
Define another %TEMP% for the build and add it to the excluded directories from scanning, realtime etc.

[Phylanx]

We suspect our AntiVirus software to have a lock on the tmp directory, so we will not add this "ignore_errors=True".

Under any circumstances, do *not* enable antivirus on build directories and processes.
Especially, if you can control features such as "on access", "realtime protection" and such, at least exclude locations and processes involved.
Since the build touched %TEMP%, which is a prime target for dropping malware, the AV jumped probably ten meters up.
Define another %TEMP% for the build and add it to the excluded directories from scanning, realtime etc.

Thank you for your advice, we've already done it
For the processes: It's a bit rough in our AV Management to add (you have to add every used *.exe manually twice (exceptions for "behavior monitoring" and additional "trusted programs")), but it works.
For the directory: The directly used directory where CEF sources are synced and compiled, was already excluded, so I just changed the TEMP/TMP/TEMPDIR folder to a temp subfolder in the working directory.

Seems to work