Repository navigation
wait_for_idle cause Ferrum::TimeoutError when visit same url again in ruby retry clause. #226
Description
I have code like this:
def retry_timeout(seconds, waiting_for_if:)
raise 'waiting_for_if must be a Proc object' unless waiting_for_if.is_a? Proc
tries = 0
begin
tries += 1
Timeout.timeout(seconds) do
while waiting_for_if.call
sleep 1
end
end
rescue TimeoutError
logger.info description if description.present?
logger.info "Timeout after waiting #{seconds} seconds, Retried #{tries} times."
retry
end
end
waiting_for_if_proc = proc do
browser.goto 'some_url'
browser.network.wait_for_idle
# do many crap
sleep 100 # wait timeout happen.
endThen load it use:
begin
retry_timeout 55, wait_for_if: wait_for_if_proc
ensure
browser.quit
end It works when first time scrap, but, if timeout happen, when goto some_url again, wait_for_idle will keep block current network request, util Ferrum::TimeoutError happen, and raise error like this:
gems/ferrum-0.11/lib/ferrum/network.rb:30:in `wait_for_idle': Timed out waiting for response. It's possible that this happened because something took a very long time (for example a page load was slow). If so, setting the :timeout option to a higher value might help. (Ferrum::TimeoutError)
But after remove all browser.network.wait_for_idle, all works quite well.
Thank you.
Activity
zw963 commentedon Dec 3, 2021
BTW, ask another question, i saw some methods like:
browser.reset,browser.network.clear(:traffic), if any of those method should be invoke to release memory before goto 'some_url' again in the retry caluse?[-]wait_for_idle cause Ferrum::TimeoutError when visit a url in retry.[/-][+]wait_for_idle cause Ferrum::TimeoutError when visit same url again in ruby retry clause.[/+]Mifrill commentedon Feb 3, 2022
@zw963 you can try play with options for
wait_for_idlemethod:Sort of, it could be, however, I don't think your case related to memory lack.
browser.resetneed to close browser tabs, so if you have many opened tabs (possibly by script on-page that opens tabs automatically) it could be a reason for many connections.browser.network.clear(:traffic)need to clear browser's cache or collected traffic, but in this case, we have an error right on the next visit same URL again, so it shouldn't be related to memory lack due to traffic.Anyway, need to play with the website to see closer and analyze the reasons. It could be fab if you are able to provide more details and the source as well, let's convert it to a discussion and proceed with it.