Proxy settings cause unbounded memory growth

Having problems with building or using CEF's C/C++ APIs? This forum is here to help. Please do not post bug reports or feature requests here.

Proxy settings cause unbounded memory growth

Postby gagere » Thu Dec 05, 2019 3:56 pm

While trying to diagnose why WEBM's can't be played from a custom scheme handler I ran into a problem with CEF. At least I think it's CEF, since the similar version of chromium (from https://chromium.woolyss.com/) works okay. Both our application that uses CEF and the CEF Client sample app when loaded inside our corporate environment hit an issue with our proxy settings. It seems this causes unbounded memory growth. I see the processes private byte usage go up to 4gb and then crash without any dumps or logging.

Logs:
Code: Select all
[1205/155204.408:VERBOSE2:thread_state.cc(529)] [state:14C43448] ScheduleGCIfNeeded
[1205/155204.408:VERBOSE2:thread_state.cc(529)] [state:14C43448] ScheduleGCIfNeeded
[1205/155204.410:ERROR:paint_controller.cc(548)] PaintController::FinishCycle() completed
[1205/155204.495:VERBOSE1:proxy_resolver_v8_tracing.cc(909)] PAC-error: line: 24: Uncaught SyntaxError: Unexpected end of input
[1205/155204.497:VERBOSE1:proxy_resolver_v8_tracing.cc(909)] PAC-error: line: 24: Uncaught SyntaxError: Unexpected end of input
[1205/155204.497:VERBOSE1:proxy_resolution_service.cc(1340)] Failed configuring with PAC script, falling-back to manual proxy servers.
[1205/155204.497:VERBOSE1:proxy_resolution_service.cc(1340)] Failed configuring with PAC script, falling-back to manual proxy servers.
[1205/155212.514:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://wpad.example.com/wpad-example.dat
[1205/155212.514:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://wpad.example.com/wpad-example.dat
[1205/155212.515:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://wpad.example.com/wpad-example.dat
[1205/155212.515:VERBOSE1:network_delegate.cc(32)] NetworkDelegate::NotifyBeforeURLRequest: http://wpad.example.com/wpad-example.dat


This did not happen previously on CEF 3683/73.

Looking at the contents of http://wpad.example.com/wpad-example.dat it appears to be badly formed (although perhaps still parsable in previous versions? missing end bracket), not sure if this is related:

Code: Select all
function FindProxyForURL(url, host)
    {
      // If URL has no dots in host name, send traffic direct.
      
      if (isPlainHostName(host))
         return "DIRECT";

      // If IP address is internal or hostname resolves to internal IP, send direct.
      
         var resolved_ip = dnsResolve(host);

      if (isInNet(resolved_ip, "10.0.0.0", "255.0.0.0") || isInNet(resolved_ip, "172.16.0.0", "255.240.0.0") || isInNet(resolved_ip, "192.168.0.0", "255.255.0.0") || isInNet(resolved_ip, "127.0.0.0", "255.0.0.0"))
         return "DIRECT";

      // DNS domain exceptions for direct access

      if (dnsDomainIs(host, "localhost") || dnsDomainIs(host, ".ad.example.com"))
         return "DIRECT";


                        else


                 return "PROXY test.example.com:8080";


Environment is Windows 10 1809 and I've been testing with CefClient from 3094/78.

Disable the proxy settings in windows or running CefClient with --no-proxy-server works around the issue.

Let me know if I can provide any other information, or debugging steps.
gagere
Newbie
 
Posts: 6
Joined: Tue Oct 08, 2013 8:30 pm

Re: Proxy settings cause unbounded memory growth

Postby Czarek » Mon Dec 16, 2019 9:13 am

Does the issue reproduce with cefsimple/cefclient sample applications?
Maintainer of the CEF Python, PHP Desktop and CEF C API projects. My LinkedIn.
User avatar
Czarek
Virtuoso
 
Posts: 1927
Joined: Sun Nov 06, 2011 2:12 am

Re: Proxy settings cause unbounded memory growth

Postby JozefK » Mon Dec 14, 2020 3:43 pm

We are facing the same problem. (CEF ver. 85.3.9+gb045a6e+chromium-85.0.4183.102)

Yes this problem can be reproduced with cefclient and cefsimple application. But you have to have prepared testing environment correctly. (Im using Kali and repsonder as a host for broken wpad.pac file)
If the wpad.pac contain syntax error, the problem will arise.
Code: Select all
extensions::ProxyEventRouter::OnPACScriptError(...)


Can't say if this is a problem of CEF or chromium, but Chrome itself is OK. Spotify / Adobe Creative Cloud is facing same issue.

The problematic place is:
src-chromium-85.0.4183.102\chrome\browser\extensions\event_router_forwarder.cc line 107

Code: Select all
  DCHECK_GT(profiles_to_dispatch_to.size(), 0u)
      << "There should always be at least one profile!";

  std::vector<std::unique_ptr<base::ListValue>> per_profile_args;
  per_profile_args.reserve(profiles_to_dispatch_to.size());
  for (size_t i = 0; i < profiles_to_dispatch_to.size() - 1; ++i)
    per_profile_args.emplace_back(event_args->DeepCopy());
  per_profile_args.emplace_back(std::move(event_args));


If the container profiles_to_dispatch_to is empty -> there is cycle to 0xFFFFFFFFFFFFFFFF -> so massive attack to memory with DeepCopy()

Of course there is a debug check - container should not be empty, but in some situation it is.
Seems there are at least 2 instances of ProxyConfigMonitor, first is OK, and second instance of ProxyConfigMonitor has profile_ == nullptr.

This is a really serious issue, especially when your clients are companies. (used to use WPAD protocol).

Callstack
Code: Select all
>   libcef.dll!extensions::EventRouterForwarder::HandleEvent(const std::__1::basic_string<char,std::__1::char_traits<char>,std::__1::allocator<char>> & extension_id={...}, extensions::events::HistogramValue histogram_value=PROXY_ON_PROXY_ERROR, const std::__1::basic_string<char,std::__1::char_traits<char>,std::__1::allocator<char>> & event_name={...}, std::__1::unique_ptr<base::ListValue,std::__1::default_delete<base::ListValue>> event_args={...}, void * profile_ptr=0x00000000, bool use_profile_to_restrict_events=true, const GURL & event_url={...}, bool dispatch_to_off_the_record_profiles=false) Line 107   C++   Symbols loaded.
    libcef.dll!extensions::EventRouterForwarder::BroadcastEventToRenderers(extensions::events::HistogramValue histogram_value=PROXY_ON_PROXY_ERROR, const std::__1::basic_string<char,std::__1::char_traits<char>,std::__1::allocator<char>> & event_name={...}, std::__1::unique_ptr<base::ListValue,std::__1::default_delete<base::ListValue>> event_args={...}, const GURL & event_url={...}, bool dispatch_to_off_the_record_profiles=false) Line 37   C++   Symbols loaded.
    libcef.dll!extensions::ProxyEventRouter::OnPACScriptError(extensions::EventRouterForwarder * event_router=0x05e499c8, void * profile=0x00000000, int line_number=1, const std::__1::basic_string<wchar_t,std::__1::char_traits<wchar_t>,std::__1::allocator<wchar_t>> & error={...}) Line 88   C++   Symbols loaded.
    libcef.dll!ProxyConfigMonitor::OnPACScriptError(int line_number=1, const std::__1::basic_string<char,std::__1::char_traits<char>,std::__1::allocator<char>> & details={...}) Line 137   C++   Symbols loaded.
    libcef.dll!network::mojom::ProxyErrorClientStubDispatch::Accept(network::mojom::ProxyErrorClient * impl=0x0111a9fc, mojo::Message * message=0x00eff148) Line 630   C++   Symbols loaded.
    libcef.dll!network::mojom::ProxyErrorClientStub<mojo::RawPtrImplRefTraits<network::mojom::ProxyErrorClient>>::Accept(mojo::Message * message=0x00eff148) Line 320   C++   Symbols loaded.
    libcef.dll!mojo::InterfaceEndpointClient::HandleValidatedMessage(mojo::Message * message=0x00eff148) Line 554   C++   Symbols loaded.
    libcef.dll!mojo::MessageDispatcher::Accept(mojo::Message * message=0x00eff148) Line 46   C++   Symbols loaded.
    libcef.dll!mojo::internal::MultiplexRouter::ProcessIncomingMessage(mojo::internal::MultiplexRouter::MessageWrapper * message_wrapper=0x00eff148, mojo::internal::MultiplexRouter::ClientCallBehavior client_call_behavior=99028832, base::SequencedTaskRunner * current_task_runner=0x05e59ec8) Line 954   C++   Symbols loaded.
    libcef.dll!mojo::internal::MultiplexRouter::Accept(mojo::Message * message=0x00eff148) Line 624   C++   Symbols loaded.

JozefK
Techie
 
Posts: 20
Joined: Mon Feb 27, 2017 3:24 pm

Re: Proxy settings cause unbounded memory growth

Postby magreenblatt » Mon Dec 14, 2020 4:08 pm

@JozefK Please file a bug with Chromium at https://crbug.com/new.
magreenblatt
Site Admin
 
Posts: 12383
Joined: Fri May 29, 2009 6:57 pm

Re: Proxy settings cause unbounded memory growth

Postby JozefK » Fri Jan 15, 2021 9:05 am

JozefK
Techie
 
Posts: 20
Joined: Mon Feb 27, 2017 3:24 pm

Re: Proxy settings cause unbounded memory growth

Postby digory » Tue Jan 26, 2021 9:39 am

digory
Expert
 
Posts: 118
Joined: Wed Oct 26, 2016 3:13 am

Re: Proxy settings cause unbounded memory growth

Postby JozefK » Mon Mar 15, 2021 2:55 am

It depends on the author/reporter of the ticket @Bfx. I was trying to find out.
But based on the Tobias Hübner and Eric Astor answers, definitely yes.
JozefK
Techie
 
Posts: 20
Joined: Mon Feb 27, 2017 3:24 pm


Return to Support Forum

Who is online

Users browsing this forum: Google [Bot] and 49 guests