Articles listed within: Knowledge Base | Settings - Spider

Are GA hits blocked when a spider runs?

21st Sep 2018

Yes (and no). By default, when we spider or test a page we block all Google Analytics and DoubleClick hits from being sent. However, in order to check if 3rd-party cookies are used by Google Analytics - this happens if you have Demographics and Interest Reports, Remarketing with GA or DCM Integration - we have to allow a very small number of hits to pass through our filter. This is because some DoubleClick hits can originate from the Google Analytics hit (analytics.js file). The check for 3rd-party cookie use is important because of its privacy implications. Using 3rd-party cookies requires explicit consent under GDPR Read more


Spider Settings

13th Sep 2018

You can, and should, optimise your spider. Firstly because the spider results are the foundation of all further tests - audits are only as good as their input. Secondly, testing irrelevant or duplicate content is wasteful - each page must first be loaded by our headless browser with time allowed for all assets to load (excluded assets are images, css and video files). This takes time and can be as long as 10 seconds per page. That does not sound a lot, but crawls typically contain thousands and tens of thousands of pages, so it soon adds up. Note you Read more