FindReplaceResponseFilter causes content incomplete[updated]

Having problems with building or using CEF's C/C++ APIs? This forum is here to help. Please do not post bug reports or feature requests here.

FindReplaceResponseFilter causes content incomplete[updated]

Postby tiplip » Thu Jun 10, 2021 9:56 pm

hi,

updated!!

filtered file size > 512k is prone to load implete on 32bit version

the issue occurs often but not each time on the same site.

I first come across such issue from cef3239 (chromium 63), the same issue can be repoduced on cef 4389 (chromium 89) as well.
But on older version cef2623 (chromium 49), it works as expected.

what should be noticed on response filter?

What I have done on cefclient (cef 2623, 3239, 4389)

change the codes as below
Code: Select all
//const char kFindString[] = "REPLACE_THIS_STRING";
//const char kReplaceString[] = "This is the replaced string!";
const char kFindString[] = "_top";
const char kReplaceString[] = "_self";

  FilterStatus Filter(void* data_in,
                      size_t data_in_size,
                      size_t& data_in_read,
                      void* data_out,
                      size_t data_out_size,
                      size_t& data_out_written) OVERRIDE {
    DCHECK((data_in_size == 0U && !data_in) || (data_in_size > 0U && data_in));
    DCHECK_EQ(data_in_read, 0U);
    DCHECK(data_out);
    DCHECK_GT(data_out_size, 0U);
    DCHECK_EQ(data_out_written, 0U);

    // All data will be read.
    data_in_read = data_in_size;

    const size_t find_size = sizeof(kFindString) - 1;
    const size_t replace_size = sizeof(kReplaceString) - 1;

    const char* data_in_ptr = static_cast<char*>(data_in);
    char* data_out_ptr = static_cast<char*>(data_out);

    // Reset the overflow.
    std::string old_overflow;
    if (!overflow_.empty()) {
      old_overflow = overflow_;
      overflow_.clear();
    }

    const size_t likely_out_size =
        data_in_size + replace_overflow_size_ + old_overflow.size();
    if (data_out_size < likely_out_size) {
      // We'll likely need to use the overflow buffer. Size it appropriately.
      overflow_.reserve(likely_out_size - data_out_size);
    }

    if (!old_overflow.empty()) {
      // Write the overflow from last time.
      Write(old_overflow.c_str(), old_overflow.size(), WRITE_PARAMS);
    }

    // Evaluate each character in the input buffer. Track how many characters in
    // a row match kFindString. If kFindString is completely matched then write
    // kReplaceString. Otherwise, write the input characters as-is.
    for (size_t i = 0U; i < data_in_size; ++i) {
      if (data_in_ptr[i] == kFindString[find_match_offset_]) {
        // Matched the next character in the find string.
        if (++find_match_offset_ == find_size) {
          // Complete match of the find string. Write the replace string.
          //std::stringstream ss;
          //ss << ++replace_count_ << ". " << kReplaceString;
          //const std::string& replace_str = ss.str();
          Write(kReplaceString, replace_size, WRITE_PARAMS);

          // Start over looking for a match.
          find_match_offset_ = 0;
        }
        continue;
      }

      // Character did not match the find string.
      if (find_match_offset_ > 0) {
        // Write the portion of the find string that has matched so far.
        Write(kFindString, find_match_offset_, WRITE_PARAMS);

        // Start over looking for a match.
        find_match_offset_ = 0;
      }

      // Write the current character.
      Write(&data_in_ptr[i], 1, WRITE_PARAMS);
    }

    // If a match is currently in-progress we need more data. Otherwise, we're
    // done.
    return find_match_offset_ > 0 ? RESPONSE_FILTER_NEED_MORE_DATA
                                  : RESPONSE_FILTER_DONE;
  }

  //if (test_runner::IsTestURL(url, kTestUrlPath))
    return new FindReplaceResponseFilter();



then visit
https://s.taobao.com/search?spm=a21bo.21814703.201867-links-1.1.5af911d9b9hud3&q=%E5%A5%B3%E6%97%B6%E8%A3%85%E5%87%89%E9%9E%8B&cps=yes&ppath=20000%3A97466

result
page
loading-nothing.png
loading-nothing.png (14.24 KiB) Viewed 4022 times


resource missing
content incomplete
incomplete.png
incomplete.png (113.32 KiB) Viewed 4022 times
tiplip
Mentor
 
Posts: 76
Joined: Thu Mar 26, 2015 3:09 am

Re: FindReplaceResponseFilter causes content incomplete[upda

Postby tiplip » Mon Jun 14, 2021 4:07 am

what I have found as below, on 32bit version cef

js file size, 524,290 bytes at one go
FilterStatus Filter experienced data_out_written=32767 once, find_match_offset_ = 1 twice, and data_in_size = 0 once.
Write experienced data_out_size = data_out_written =1 three times

that is, for file > 512k (512,288 bytes), say 524,290 bytes, and kFindString is at the end of 32k chunk but need more data, issue occurs
tiplip
Mentor
 
Posts: 76
Joined: Thu Mar 26, 2015 3:09 am

Re: FindReplaceResponseFilter causes content incomplete[upda

Postby magreenblatt » Mon Jun 14, 2021 10:40 am

Please add a bug and include a reproduction case (e.g. a new unit test or patch to cefclient that reproduces the problem).
magreenblatt
Site Admin
 
Posts: 12382
Joined: Fri May 29, 2009 6:57 pm

Re: FindReplaceResponseFilter causes content incomplete[upda

Postby tiplip » Mon Jun 14, 2021 10:30 pm

I don't know how to file a bug and where

what I did,
    download the 74.1.19+gb62bacf+chromium-74.0.3729.157, this is the last response filter with 32k size and I just know how to reproduce the issue with 32k
    cmake solution dir on windows 10
    enlarge response_filter.html to 513k, a little more than 512k
    response_filter.zip
    (6.49 KiB) Downloaded 223 times

    change response_filter_test.cc
Code: Select all
const char kFindString[] = "_top";
const char kReplaceString[] = "_self";

rebuild cefclient in release version

in attached response_filter.html, you can search keyword _to
run cefclient.exe, navigate to test/response_filter

benchmark
incomplete1.png
incomplete1.png (140.2 KiB) Viewed 3977 times
tiplip
Mentor
 
Posts: 76
Joined: Thu Mar 26, 2015 3:09 am

Re: FindReplaceResponseFilter causes content incomplete[upda

Postby tiplip » Tue Jun 15, 2021 7:28 pm

tiplip
Mentor
 
Posts: 76
Joined: Thu Mar 26, 2015 3:09 am

Re: FindReplaceResponseFilter causes content incomplete[upda

Postby magreenblatt » Tue Jun 15, 2021 8:27 pm

We’ll need a way to reproduce the issue with currently supported versions (91 or newer).
magreenblatt
Site Admin
 
Posts: 12382
Joined: Fri May 29, 2009 6:57 pm

Re: FindReplaceResponseFilter causes content incomplete[upda

Postby tiplip » Tue Jun 15, 2021 9:24 pm

thanks.
I have no idea of max available capacity for response filter, so don't know how to reproduce it with 64k file on v91.
tiplip
Mentor
 
Posts: 76
Joined: Thu Mar 26, 2015 3:09 am

Re: FindReplaceResponseFilter causes content incomplete[upda

Postby magreenblatt » Tue Jun 15, 2021 9:38 pm

I don’t know what you mean by “max available capacity.” The buffer size is 64kb, code here.
magreenblatt
Site Admin
 
Posts: 12382
Joined: Fri May 29, 2009 6:57 pm

Re: FindReplaceResponseFilter causes content incomplete[upda

Postby tiplip » Wed Jun 16, 2021 12:47 am

from my test, key point is kFindString is at the end of 32k chunk, as attached, and at the same time, file size a little more than 512k
I found 512k is defined here code

for newer version, maybe another case, I have not download all source code and debug, so not sure yet.
tiplip
Mentor
 
Posts: 76
Joined: Thu Mar 26, 2015 3:09 am


Return to Support Forum

Who is online

Users browsing this forum: Google [Bot] and 27 guests