Wget Snippets

Get past 401 unauthorized when using wget

  • This snippet, gets past most defense when a 401 unauthorized is returned
  • Will only scrape all of the images from a website.
  • Changes the user agent to look like a real web browser
  • Adds a cookie
wget -r -P storeInThisFolder -A jpeg,jpg,bmp,gif,png -e robots=off --cookie name=value -U 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36'  http://some-domain.com/folder