Thread: wget problem
View Single Post

   
  #3 (permalink)  
Old 02-19-2008, 05:20 PM
Mark Hill
 
Posts: n/a
Default Re: wget problem

On 15 Dec 2004 23:49:43 -0800,
guruteck@gmail.com <guruteck@gmail.com> wrote:

> Everybody knows we can use wget for recursive download of
> webpages..
> But some sites wont allow/prohibit recursive download..Is there any way
> to download (recursively ) these sites.


What errors are you getting when downloading a site? If you think it's
a robots.txt issue, you can tell wget to ignore it.

wget -erobots=off http://example.com

If you suspect that the site is blocking wget, you can change wget's
user-agent string with the -U option.

> Some one told me by changing
> port we can download from those sites..


That doesn't seem likely with a typical website/webserver.

--
Mark Hill
Reply With Quote