<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Hi guys and tnks for your answer, I added the robots.txt to my
/home/htdocs =) tnks. I have this rewrite rules:<br>
<br>
if ($http_user_agent = "Wget/*") {<br>
return 403;<br>
}<br>
<br>
if ($http_user_agent = "Teleport Pro") {<br>
return 403;<br>
}<br>
<br>
if ($http_user_agent = "WebCopier") {<br>
return 403;<br>
}<br>
<br>
I do a simple test with wget but I can download files with it, I tryed
with wget, Wget, Wget* and the last Wget/* but its not working.<br>
<br>
Nice day.<br>
<br>
Fabio Coatti escribió:
<blockquote cite="mid200712190900.32598.cova@ferrara.linux.it"
type="cite">
<pre wrap="">Alle martedì 18 dicembre 2007, Eden Li ha scritto:
</pre>
<blockquote type="cite">
<pre wrap="">wget (and many other user agents) respect robots.txt if you place it
at /robots.txt:
<a class="moz-txt-link-freetext" href="http://www.robotstxt.org/orig.html">http://www.robotstxt.org/orig.html</a>
<a class="moz-txt-link-freetext" href="http://en.wikipedia.org/wiki/Robots.txt">http://en.wikipedia.org/wiki/Robots.txt</a>
Of course malicious agents will ignore it and continue scraping your
site. It's pretty hard to block these kinds of bots since they can
mimic browser requests that would be difficult to disambiguate from
normal user requests.
</pre>
</blockquote>
<pre wrap=""><!---->
That's true. But if you look carefully to a usual web site logs, most part of
weird urls are coming from a small subset of specific user agents (basically,
scripts run by people who barely have a clue of what they are doing).
While I agree that several tools respects robots.txt, they are the "good"
ones, and I see no point in stopping them. Othe other side, malicious tools
that fakes the user agent are really difficult to stop and you have to rely
on a good configuration of the system. In the middle lies a highly amount of
hits coming from specific user agents, mostly trying to do pretty harmless
things (bounce attacks, etc..). That kind of visitors can be kept out by a
simple configuration line, and given the hig rate of them it can be worth to
use that countermeasure (naive as it is)
</pre>
<blockquote type="cite">
<pre wrap="">On 12/18/07, Fabio Coatti <a class="moz-txt-link-rfc2396E" href="mailto:cova@ferrara.linux.it"><cova@ferrara.linux.it></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Alle martedì 18 dicembre 2007, Alexis Torres Garnica ha scritto:
</pre>
<blockquote type="cite">
<pre wrap="">Hi guys, I am new to the list. Is there a way to stop or block the bots
access and wget to a nginx web server? tnks
att: alex
</pre>
</blockquote>
<pre wrap="">If with "block bots" you mean "block requests based on User Agent", you
can do this setting up something like this:
if ($http_user_agent ~ libwww-perl ) {
return 400;
}
(just an example, of course)
--
Fabio "Cova" Coatti <a class="moz-txt-link-freetext" href="http://members.ferrara.linux.it/cova">http://members.ferrara.linux.it/cova</a>
Ferrara Linux Users Group <a class="moz-txt-link-freetext" href="http://ferrara.linux.it">http://ferrara.linux.it</a>
GnuPG fp:9765 A5B6 6843 17BC A646 BE8C FA56 373A 5374 C703
Old SysOps never die... they simply forget their password.
</pre>
</blockquote>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
<br>
</body>
</html>