Wget Snippets
Get past 401 unauthorized when using wget
- This snippet, gets past most defense when a 401 unauthorized is returned
- Will only scrape all of the images from a website.
- Changes the user agent to look like a real web browser
- Adds a cookie
wget -r -P storeInThisFolder -A jpeg,jpg,bmp,gif,png -e robots=off --cookie name=value -U 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36' http://some-domain.com/folder