CefBrowserHost::SendKeyEvent not working on text fields

Having problems with building or using CEF's C/C++ APIs? This forum is here to help. Please do not post bug reports or feature requests here.

CefBrowserHost::SendKeyEvent not working on text fields

Postby lax1dude » Mon Sep 06, 2021 6:14 pm

I am using CEF in my game to render UI, in windowless rendering mode, with a custom CefRenderHandler implementation updating an OpenGL texture within the callback function. This means the browser is not linked to an HWND and does not recieve any keyboard or mouse events directly. I have to send keyboard events to the browser using the SendKeyEvent function in CefBrowserHost, but this function appears to just create a KeyboardEvent and submit it in javascript with window.dispatchEvent. This means the SendKeyEvent function cannot be used to type text into a text field or textarea or process any shortcuts like CTRL+C. The reason is stated on MDN:

Note: Manually firing an event does not generate the default action associated with that event. For example, manually firing a key event does not cause that letter to appear in a focused text input. In the case of UI events, this is important for security reasons, as it prevents scripts from simulating user actions that interact with the browser itself.


How do I submit native KeyboardEvents to a CefBrowserHost that can be used with text fields and textareas?
lax1dude
Newbie
 
Posts: 4
Joined: Mon Sep 06, 2021 5:54 pm

Re: CefBrowserHost::SendKeyEvent not working on text fields

Postby magreenblatt » Tue Sep 07, 2021 2:34 am

I have to send keyboard events to the browser using the SendKeyEvent function in CefBrowserHost, but this function appears to just create a KeyboardEvent and submit it in javascript with window.dispatchEvent.

Where do you see it doing that in the code?
magreenblatt
Site Admin
 
Posts: 12383
Joined: Fri May 29, 2009 6:57 pm

Re: CefBrowserHost::SendKeyEvent not working on text fields

Postby lax1dude » Tue Sep 07, 2021 3:58 pm

magreenblatt wrote:
I have to send keyboard events to the browser using the SendKeyEvent function in CefBrowserHost, but this function appears to just create a KeyboardEvent and submit it in javascript with window.dispatchEvent.

Where do you see it doing that in the code?


I don't, I made an assumption based on the behavior I observed. I observed that CefBrowserHost::SendKeyEvent triggers DOM event listeners registered with window.addEventListener('keypress', ...) but it does not trigger keystrokes on text fields and CTRL+C and CTRL+V do not work. I assumed this behavior is the result of the security feature mentioned on MDN because it is the exact behavior described on MDN for that security feature. I do not have the CEF source code to reference because I am using the binaries available at https://cef-builds.spotifycdn.com/index.html, I am not compiling CEF manually because I don't see a reason to when the binaries are already available at the link

So I'm hoping there is a way to submit keyboard events on a level lower than CefBrowserHost::SendKeyEvent where keystrokes are processed correctly for text fields and for shortcuts like CTRL+C and CTRL+V
lax1dude
Newbie
 
Posts: 4
Joined: Mon Sep 06, 2021 5:54 pm

Re: CefBrowserHost::SendKeyEvent not working on text fields

Postby magreenblatt » Wed Sep 08, 2021 2:43 am

What OS and CEF version? Please test with "cefclient --off-screen-rendering-enabled".
magreenblatt
Site Admin
 
Posts: 12383
Joined: Fri May 29, 2009 6:57 pm

Re: CefBrowserHost::SendKeyEvent not working on text fields

Postby lax1dude » Wed Sep 08, 2021 3:47 pm

CEFClient never worked on my PC, it opens for a few seconds and then closes again without and flags and with --off-screen-rendering-enabled it behaves the same. It appears to be a problem with the configuration of the GPU process because this is what's in debug.log:

Code: Select all
[0908/132937.799:ERROR:gpu_process_host.cc(955)] GPU process launch failed: error_code=18
[0908/132937.799:WARNING:gpu_process_host.cc(1274)] The GPU process has crashed 1 time(s)
[0908/132937.810:ERROR:gpu_process_host.cc(955)] GPU process launch failed: error_code=18
[0908/132937.811:WARNING:gpu_process_host.cc(1274)] The GPU process has crashed 2 time(s)
[0908/132937.815:ERROR:gpu_process_host.cc(955)] GPU process launch failed: error_code=18
[0908/132937.815:WARNING:gpu_process_host.cc(1274)] The GPU process has crashed 3 time(s)
[0908/132937.819:ERROR:gpu_process_host.cc(955)] GPU process launch failed: error_code=18
[0908/132937.819:WARNING:gpu_process_host.cc(1274)] The GPU process has crashed 4 time(s)
[0908/132937.824:ERROR:gpu_process_host.cc(955)] GPU process launch failed: error_code=18
[0908/132937.824:WARNING:gpu_process_host.cc(1274)] The GPU process has crashed 5 time(s)
[0908/132937.827:ERROR:gpu_process_host.cc(955)] GPU process launch failed: error_code=18
[0908/132937.828:WARNING:gpu_process_host.cc(1274)] The GPU process has crashed 6 time(s)
[0908/132937.833:ERROR:gpu_process_host.cc(955)] GPU process launch failed: error_code=18
[0908/132937.834:WARNING:gpu_process_host.cc(1274)] The GPU process has crashed 7 time(s)
[0908/132937.950:ERROR:gpu_process_host.cc(955)] GPU process launch failed: error_code=18
[0908/132937.950:WARNING:gpu_process_host.cc(1274)] The GPU process has crashed 8 time(s)
[0908/132938.125:ERROR:gpu_process_host.cc(955)] GPU process launch failed: error_code=18
[0908/132938.126:WARNING:gpu_process_host.cc(1274)] The GPU process has crashed 9 time(s)
[0908/132938.126:FATAL:gpu_data_manager_impl_private.cc(414)] GPU process isn't usable. Goodbye.


I'm using the latest build available at https://cef-builds.spotifycdn.com/index.html#windows64, which appears to be cef_binary_93.1.11+g9e254fa+chromium-93.0.4577.63_windows64

I feel like you're gonna be tempted to blow me off until I get cefclient to work on my PC, but let's be real, this problem with the GPU process is related to rendering the page it has nothing to do with processing keyboard events, this crash with cefclient proves nothing related to the problem I'm having, so let's stay on topic. I think the question we should be asking is 'how are native keyboard events from the windows event loop sent to a browser created with CefWindowInfo::SetAsChild' because this appears to simply be a matter of bypassing the security feature mentioned on MDN, which is very likely the culprit of this problem from the behavior I have observed, and creating a browser with CefWindowInfo::SetAsChild doesn't block text field input and browser shortcuts so the solution to this problem is as simple as finding how keyboard events are handled on a browser created with CefWindowInfo::SetAsChild and using that mechanism instead of CefBrowserHost::SendKeyEvent.
lax1dude
Newbie
 
Posts: 4
Joined: Mon Sep 06, 2021 5:54 pm

Re: CefBrowserHost::SendKeyEvent not working on text fields

Postby magreenblatt » Thu Sep 09, 2021 2:19 am

I agree that the crash isn’t relevant. Unfortunately it keeps you from testing the SendKeyEvent behavior in cefclient. If it works as expected in cefclient then your SendKeyEvent usage is probably wrong, and comparing the usage would then be helpful to you. I suggest finding a computer (or VM) where you can test cefclient.
magreenblatt
Site Admin
 
Posts: 12383
Joined: Fri May 29, 2009 6:57 pm

Re: CefBrowserHost::SendKeyEvent not working on text fields

Postby lax1dude » Thu Sep 09, 2021 4:43 pm

I got cefclient to work via the --single-process flag, keyboard input to the google search text field does appear to work even with --off-screen-rendering-enabled. I ended up finding cefclient/browser/osr_window_win.cc after getting the source and I found the WM_KEYDOWN hook it uses to send keyboard events to the offscreen browser like I probably should have before posting here, and it appears to use CefBrowserHost::SendKeyEvent like my code already was

Code: Select all
void OsrWindowWin::OnKeyEvent(UINT message, WPARAM wParam, LPARAM lParam) {
  if (!browser_)
    return;

  CefKeyEvent event;
  event.windows_key_code = wParam;
  event.native_key_code = lParam;
  event.is_system_key = message == WM_SYSCHAR || message == WM_SYSKEYDOWN ||
                        message == WM_SYSKEYUP;

  if (message == WM_KEYDOWN || message == WM_SYSKEYDOWN)
    event.type = KEYEVENT_RAWKEYDOWN;
  else if (message == WM_KEYUP || message == WM_SYSKEYUP)
    event.type = KEYEVENT_KEYUP;
  else
    event.type = KEYEVENT_CHAR;
  event.modifiers = GetCefKeyboardModifiers(wParam, lParam);

  // mimic alt-gr check behaviour from
  // src/ui/events/win/events_win_utils.cc: GetModifiersFromKeyState
  if ((event.type == KEYEVENT_CHAR) && IsKeyDown(VK_RMENU)) {
    // reverse AltGr detection taken from PlatformKeyMap::UsesAltGraph
    // instead of checking all combination for ctrl-alt, just check current char
    HKL current_layout = ::GetKeyboardLayout(0);

    // https://docs.microsoft.com/en-gb/windows/win32/api/winuser/nf-winuser-vkkeyscanexw
    // ... high-order byte contains the shift state,
    // which can be a combination of the following flag bits.
    // 1 Either SHIFT key is pressed.
    // 2 Either CTRL key is pressed.
    // 4 Either ALT key is pressed.
    SHORT scan_res = ::VkKeyScanExW(wParam, current_layout);
    constexpr auto ctrlAlt = (2 | 4);
    if (((scan_res >> 8) & ctrlAlt) == ctrlAlt) {  // ctrl-alt pressed
      event.modifiers &= ~(EVENTFLAG_CONTROL_DOWN | EVENTFLAG_ALT_DOWN);
      event.modifiers |= EVENTFLAG_ALTGR_DOWN;
    }
  }

  browser_->GetHost()->SendKeyEvent(event);
}


there appears to be a KEYEVENT_RAWKEYDOWN and KEYEVENT_CHAR type to the cef_key_event_type_t enum I never picked up on

so up to this point I've been using GLFW in my application to handle retrieving key events from the window, assuming the GLFW key codes are the windows key codes adapted to also work on other platforms. I never did much research to back this up I assumed I'd just learn the hard way based on what keys work and which don't which was a stupid plan now that I'm thinking about it. Now that I'm seeing this code for properly creating a KeyEvent shows it requires the raw wParam and lParam passed from the windows event loop, so I went and modified GLFW to additionally pass the raw wParam and lParam in the keyboard callbacks alongside the default parameters and I used these as the windows_key_code and native_key_code in both the KEYEVENT_RAWKEYDOWN and KEYEVENT_KEYUP and additionally in KEYEVENT_CHAR and only when I did all of that did it finally work. So I have it working now thanks
lax1dude
Newbie
 
Posts: 4
Joined: Mon Sep 06, 2021 5:54 pm

Re: CefBrowserHost::SendKeyEvent not working on text fields

Postby ne8277 » Wed Nov 10, 2021 9:38 am

Hey, lax1dude, you said
"I went and modified GLFW to additionally pass the raw wParam and lParam in the keyboard callbacks alongside the default parameters and I used these as the windows_key_code and native_key_code in both the KEYEVENT_RAWKEYDOWN and KEYEVENT_KEYUP and additionally in KEYEVENT_CHAR and only when I did all of that did it finally work".
I met with the same problem with you, and my environment is the same as you too. And I do want to know how you modify GLFW to get wParam and lParam, can you show the source code or just some thoughts on it. I really hope you can give me some information about it, I really it recently. Thank you!
ne8277
Newbie
 
Posts: 1
Joined: Wed Nov 10, 2021 9:18 am

Re: CefBrowserHost::SendKeyEvent not working on text fields

Postby rjxray » Tue Nov 16, 2021 9:09 am

This is how we do it with openframeworks for windows which is a opengl based framework using GLFW

First you need a message handling function which can call your OnKeyEvent function for keyboard input messages:
Code: Select all
// handle windows messages
static myAppPtr* appInstance = nullptr;
static WNDPROC ofAppWndProc; // Openframeworks application window message procedure
static LRESULT CALLBACK msgHandler(HWND hwnd, UINT msg, WPARAM wParam, LPARAM lParam) {
   if (appInstance != nullptr) {
      switch (msg) {
      case WM_SYSCHAR:
      case WM_SYSKEYDOWN:
      case WM_SYSKEYUP:
      case WM_KEYDOWN:
      case WM_KEYUP:
      case WM_CHAR:
         if (appInstance->OnKeyEvent(msg, wParam, lParam)) return 0;
         else return(CallWindowProcW(ofAppWndProc, hwnd, msg, wParam, lParam)); // Give openframeworks the key

      default: return(CallWindowProcW(ofAppWndProc, hwnd, msg, wParam, lParam));
      }
   }
   // should never get here but this prevents compiler warning
   return(CallWindowProcW(ofAppWndProc, hwnd, msg, wParam, lParam));
}

in our case ofAppWndProc is the normal windows message handling function which you will want to receive all non-keyboard messages
Then you need to register the message handler at startup to replace your normal message handler
Code: Select all
   appInstance = this;
   appWindowHandle = WindowFromDC(wglGetCurrentDC());
   ofAppWndProc = (WNDPROC)GetWindowLongPtr(appWindowHandle, GWLP_WNDPROC);
   SetWindowLongPtrW(appWindowHandle, GWLP_WNDPROC, (LONG_PTR)msgHandler);
rjxray
Expert
 
Posts: 115
Joined: Wed Jun 07, 2017 4:31 am


Return to Support Forum

Who is online

Users browsing this forum: No registered users and 32 guests