Delivery-Date: Tue, 29 Sep 2015 23:22:50 -0400
Return-Path: <tor-talk-bounces@lists.torproject.org>
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on moria.seul.org
X-Spam-Level: 
X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED,
	DKIM_SIGNED,FREEMAIL_FROM,RCVD_IN_DNSWL_MED,T_DKIM_INVALID,T_RP_MATCHES_RCVD
	autolearn=ham version=3.3.1
X-Original-To: archiver@seul.org
Delivered-To: archiver@seul.org
Received: from eugeni.torproject.org (eugeni.torproject.org [38.229.72.13])
	(using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by khazad-dum.seul.org (Postfix) with ESMTPS id 658541E0312;
	Tue, 29 Sep 2015 23:22:48 -0400 (EDT)
Received: from eugeni.torproject.org (localhost [127.0.0.1])
	by eugeni.torproject.org (Postfix) with ESMTP id 9E06D36E04;
	Wed, 30 Sep 2015 03:22:43 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
 by eugeni.torproject.org (Postfix) with ESMTP id 5642436DFF
 for <tor-talk@lists.torproject.org>; Wed, 30 Sep 2015 03:22:40 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at 
Received: from eugeni.torproject.org ([127.0.0.1])
 by localhost (eugeni.torproject.org [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id DvtFhVJM7CV7 for <tor-talk@lists.torproject.org>;
 Wed, 30 Sep 2015 03:22:40 +0000 (UTC)
Received: from mail-io0-f170.google.com (mail-io0-f170.google.com
 [209.85.223.170])
 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
 (Client CN "smtp.gmail.com",
 Issuer "Google Internet Authority G2" (not verified))
 by eugeni.torproject.org (Postfix) with ESMTPS id 2D9C236DCE
 for <tor-talk@lists.torproject.org>; Wed, 30 Sep 2015 03:22:40 +0000 (UTC)
Received: by ioii196 with SMTP id i196so33394539ioi.3
 for <tor-talk@lists.torproject.org>; Tue, 29 Sep 2015 20:22:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:in-reply-to:references:date:message-id:subject:from:to
 :content-type; bh=NodHxWyhRdbIBsBS8dJrcaIOw8JlVib+yEa+2eBB6Xw=;
 b=lrPb3JRpFc5oaPxspFHmpsIG/HQzgQ++eIKCS2iDAKBKDi6W55OEKVOg9ZGP4aKnhJ
 VE4UII8mHkMU/hVinGG/yDPhtHWzHqJpeDUy2bF716BXaIuBx14CxtblgaRG3rei3yrS
 5fPGMH2/T8kNqR67fFg4/f2x0+zBP00hSNbACHB0OQ8USgbo7+gutFg4lo/QB3eHEUUy
 PnRedpeaYv26OkOyjgixXeqlVHI6SwdDXSY86jo2VDoVMQJHHiFuDFASgYrWAsUlcc5A
 X7r08EKO9AfA+FpyKtkcvPl00VIsdd7pfGNbn8r66XN4IXx6L45b+UGjxpATVtVcbe8a
 fWtQ==
MIME-Version: 1.0
X-Received: by 10.107.7.231 with SMTP id g100mr2076007ioi.81.1443583357619;
 Tue, 29 Sep 2015 20:22:37 -0700 (PDT)
Received: by 10.107.170.216 with HTTP; Tue, 29 Sep 2015 20:22:37 -0700 (PDT)
Received: by 10.107.170.216 with HTTP; Tue, 29 Sep 2015 20:22:37 -0700 (PDT)
In-Reply-To: <CAJwFvsXHjUWz8YwB70V3mfTrh9F=GwSCxpensEjEhLmBy-tjhQ@mail.gmail.com>
References: <CAJwFvsWDxzLjhRLPa1xbW5EuXO1nw=BRR+uJTSqRxyu3BF9CYA@mail.gmail.com>
 <CAJwFvsXHjUWz8YwB70V3mfTrh9F=GwSCxpensEjEhLmBy-tjhQ@mail.gmail.com>
Date: Tue, 29 Sep 2015 23:22:37 -0400
Message-ID: <CAJwFvsWSyJB7c0=dL22KuVSgJazBFdPpBqS6U3TJO5hHetd3Ew@mail.gmail.com>
From: Tyler Hardin <th020394@gmail.com>
To: tor-talk@lists.torproject.org
X-Content-Filtered-By: Mailman/MimeDel 2.1.15
Subject: Re: [tor-talk] How to write program that uses Tor network
X-BeenThere: tor-talk@lists.torproject.org
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: tor-talk@lists.torproject.org
List-Id: "all discussion about theory, design,
 and development of Onion Routing" <tor-talk.lists.torproject.org>
List-Unsubscribe: <https://lists.torproject.org/cgi-bin/mailman/options/tor-talk>, 
 <mailto:tor-talk-request@lists.torproject.org?subject=unsubscribe>
List-Archive: <http://lists.torproject.org/pipermail/tor-talk/>
List-Post: <mailto:tor-talk@lists.torproject.org>
List-Help: <mailto:tor-talk-request@lists.torproject.org?subject=help>
List-Subscribe: <https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk>, 
 <mailto:tor-talk-request@lists.torproject.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: tor-talk-bounces@lists.torproject.org
Sender: "tor-talk" <tor-talk-bounces@lists.torproject.org>

Also, about rate limiting, what sort of rate limit do y'all think would be
mindful of the health of the network and the average site? I'm thinking a
maximum of 1 req per second per site and 10 reqs per second overall (in
other words, 10 sites per second). But based on my experience browsing with
TBB, that might be too much.
On Sep 29, 2015 11:14 PM, "Tyler Hardin" <th020394@gmail.com> wrote:

> Hi, I'm writing a spider in C++ and thinking about running it on the Tor
> hidden network. I'm using boost::asio for the network API. What would be
> the best/easiest way for me to retrieve pages from the Tor hidden network?
>
> P.S. Yes, I'm respecting robots.txt and rate limiting. I'm not going to
> DOS you guys.
>
-- 
tor-talk mailing list - tor-talk@lists.torproject.org
To unsubscribe or change other settings go to
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk

