How does Silktide detect cookies?

There are important considerations to be aware of when using Silktide to identify your cookies, especially if you rely on the absolute accuracy of this information for legal purposes.

Cookies can only be detected when a website sets them. Due to the nature of the web, there is no way to discover cookies other than to use as much of your website as possible, while recording any cookies you find.

The vast majority of cookies are set when a webpage is loaded, and Silktide easily detects these cookies as it always loads each of your webpages in turn.

However it is possible for cookies to be set only when specific actions are completed by a user, for example:

• Dismissing a cookie banner
• Navigating to another page (triggered via JavaScript)
• Logging in
• Searching
• Confirming your age

A bot like Silktide will not ordinarily complete these actions, and so may miss cookies set as a result. The solution is to teach Silktide to perform these steps like a person would.

We provide two frameworks for instructing Silktide to complete these interactions. For simple “one click” actions, like dismissing a cookie banner, you only need a single line of code which you can optionally set up yourself. For more complex interactions, you will need to ask your Customer Success Manager to perform additional configuration.

Websites which set a cookie when interacting with the page

Some websites use JavaScript to detect when a user clicks on any part of the page, and to set a cookie.

Crawlers like Silktide will not ordinarily see these cookies, because crawlers work by gathering links and downloading these links in parallel. If we had to ‘persist’ cookies in this manner, our platform would have to be tens or hundreds of times slower

The solution is to set up a custom interaction to ‘warm up’ your site – this will click a link on your homepage and remember the cookies from that for all subsequent pages. Ask your Customer Success Manager to set this up if you need it.

Technical details of cookie detection

1. Silktide uses headless Chrome to download and test all of your pages. This means JavaScript, AJAX etc will all run like they do in regular Chrome.
2. This also means we see cookies set by your resources, such as tracking pixels or scripts.
3. By default each download of each webpage starts with no cookies set. This allows us to identify where new cookies are being set. It also allows us to crawl many pages in parallel.
4. By default Silktide downloads webpages from AWS in the US, but you can configure this.
5. Any interactions with your pages can be configured via a series of manual steps, equivalent to a Selenium script. For example: open a URL, wait for it to load, click on a selector, wait for a selector to change etc.