[Novalug] inconsistant webpage access

jerry w jerrywone@gmail.com
Fri Mar 13 01:11:13 EDT 2015


Robots.txt
Curl and wget are not regular user agents, you might have arguments that
self identify as something else, but as a netizen, maybe lying to get info
is bad.  How often you run, debugging, etc might upset the webmaster .

Try on novalug.com and see if results differ?
On Mar 13, 2015 12:56 AM, "Jon LaBadie via Novalug" <
novalug@firemountain.net> wrote:

> I'm trying to set up a script to periodically download
> several webpages.  Using curl, things seemrd to be
> working alright.  Then one site started acting up.
> I'd get "access denied" messages.
>
> $ curl http://www. ...
> <h1>Access Denied</h1><!-- by MKCounter v1.4.0: -->
> <!-- 27623 -->
>
> So far a work-around, if curl fails, use wget.
> But I'd like to understand the error.
>
> Jon
> --
> Jon H. LaBadie                  novalugml@jgcomp.com
>  11226 South Shore Rd           (703) 787-0688 (H)
>  Reston, VA  20190              (703) 935-6720 (C)
> **********************************************************************
> The Novalug mailing list is hosted by firemountain.net.
>
> To unsubscribe or change delivery options:
> http://www.firemountain.net/mailman/listinfo/novalug
>



More information about the Novalug mailing list