Delivery-Date: Fri, 13 Mar 2015 08:51:34 -0400
Return-Path: <tor-talk-bounces@lists.torproject.org>
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on moria.seul.org
X-Spam-Level: 
X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	RCVD_IN_DNSWL_MED,T_DKIM_INVALID,T_RP_MATCHES_RCVD,URIBL_BLOCKED
	autolearn=ham version=3.3.1
X-Original-To: archiver@seul.org
Delivered-To: archiver@seul.org
Received: from eugeni.torproject.org (eugeni.torproject.org [38.229.72.13])
	(using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by khazad-dum.seul.org (Postfix) with ESMTPS id 0BC151E0870
	for <archiver@seul.org>; Fri, 13 Mar 2015 08:51:32 -0400 (EDT)
Received: from eugeni.torproject.org (localhost [127.0.0.1])
	by eugeni.torproject.org (Postfix) with ESMTP id E093633D58;
	Fri, 13 Mar 2015 12:51:25 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
 by eugeni.torproject.org (Postfix) with ESMTP id 5211F33CE7
 for <tor-talk@lists.torproject.org>; Fri, 13 Mar 2015 12:51:22 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at 
Received: from eugeni.torproject.org ([127.0.0.1])
 by localhost (eugeni.torproject.org [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id PEx5vi6k7IrE for <tor-talk@lists.torproject.org>;
 Fri, 13 Mar 2015 12:51:22 +0000 (UTC)
Received: from windmill.donncha.is (unknown [IPv6:2001:41d0:8:da8a::1])
 by eugeni.torproject.org (Postfix) with ESMTP id 0865533CDF
 for <tor-talk@lists.torproject.org>; Fri, 13 Mar 2015 12:51:22 +0000 (UTC)
Received: from [192.168.42.62] (unknown [80.111.240.26])
 by windmill.donncha.is (Postfix) with ESMTPSA id 8274F615;
 Fri, 13 Mar 2015 14:30:35 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=donncha.is; s=dkim;
 t=1426253435; bh=9csgDErWnGqmhYq96yZk5aKnr+8hiwCnK0AKCNt604Q=;
 h=Date:From:To:CC:Subject:References:In-Reply-To:From;
 b=ABlirg7jlLwY8zhDJ36a+7geC9YcYpYYqAw93bDZJqFP29ahgI9/FTGF+01WWkQKD
 pWycUMeZDHPYA/W4Bnhck362pChjaOi7aNjbj0ke9PEhisYbSwRTYcAo9j7xGJlb09
 OjTyXjcMtjUcM4Cd2jabCUOm1r+gBRIEft1gZaFU=
Message-ID: <5502DD21.9000309@donncha.is>
Date: Fri, 13 Mar 2015 12:50:41 +0000
From: Donncha O'Cearbhaill <donncha@donncha.is>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.5.0
MIME-Version: 1.0
To: George Kadianakis <desnacked@riseup.net>
References: <A6DD5B99-8E6D-4B21-AC82-14E92C8C6332@maclemon.at>	<87d24f8mb1.fsf@riseup.net>
 <5501932E.80703@donncha.is> <87fv99yuja.fsf@riseup.net>
In-Reply-To: <87fv99yuja.fsf@riseup.net>
OpenPGP: url=http://donncha.is/donncha.asc
Cc: tor-talk@lists.torproject.org
Subject: Re: [tor-talk] Load Balancing/High Availability Hidden Services
X-BeenThere: tor-talk@lists.torproject.org
X-Mailman-Version: 2.1.15
Precedence: list
Reply-To: tor-talk@lists.torproject.org
List-Id: "all discussion about theory, design,
 and development of Onion Routing" <tor-talk.lists.torproject.org>
List-Unsubscribe: <https://lists.torproject.org/cgi-bin/mailman/options/tor-talk>, 
 <mailto:tor-talk-request@lists.torproject.org?subject=unsubscribe>
List-Archive: <http://lists.torproject.org/pipermail/tor-talk/>
List-Post: <mailto:tor-talk@lists.torproject.org>
List-Help: <mailto:tor-talk-request@lists.torproject.org?subject=help>
List-Subscribe: <https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk>, 
 <mailto:tor-talk-request@lists.torproject.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============0403785364008158897=="
Errors-To: tor-talk-bounces@lists.torproject.org
Sender: "tor-talk" <tor-talk-bounces@lists.torproject.org>

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--===============0403785364008158897==
Content-Type: multipart/signed; micalg=pgp-sha512;
 protocol="application/pgp-signature";
 boundary="2HrQKJo39fDw2obW1Jd8b0C7Hj68MDlrK"

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--2HrQKJo39fDw2obW1Jd8b0C7Hj68MDlrK
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable



On 13/03/15 12:02, George Kadianakis wrote:
> Donncha O'Cearbhaill <donncha@donncha.is> writes:
>=20
>> On 11/03/15 17:40, George Kadianakis wrote:
>>> MacLemon <tor@maclemon.at> writes:
>>>
>>>> Hoi!
>>>>
>>>> I'm looking into ideas for creating =E2=80=9Cload balanced=E2=80=9D =
or =E2=80=9Chigh availability=E2=80=9D hidden services. Mostly pertaining=
 to web servers serving large-ish static files. (Let's say 5-100MB each.)=

>>>>
>>>> Load balanced as in not all requests end up at the same box to speed=
 up downloads.
>>>> High availability as in the service is still available if one box go=
es down or is taken offline for maintenance.
>>>>
>>>> So, not exactly your usual distributed-cluster setup.
>>>>
>>>>
>>>> From what I understand it would not make sense to run the same HS Ke=
y on multiple boxes since the descriptors would overwrite each other ever=
y few minutes.
>>>>
>>>> I don't think one can do something like Round-Robin DNS with HS.
>>>>
>>>> So the only way I can imagine this to work is a central redirection =
node that know about all the nodes and more or less intelligently/randoml=
y 302 redirects each file request to a known-to-it server.
>>>>
>>>> This still leaves a single-point-of-failure in form of the redirecti=
on server but would at least distribute the traffic load across multiple =
servers and cope for nodes coming and going.
>>>>
>>>> Has anyone done something like this?
>>>>
>>>
>>> An application-layer load balancer like HAProxy might be able to help=
 you.
>>>
>>> Unfortunately, there is not something equivalent to DNS Round Robin
>>> for hidden services yet. There are some ideas on how to do this on th=
e
>>> Introduction Point-layer, but a proposal still needs to be written. F=
or
>>> further reading:
>>> https://lists.torproject.org/pipermail/tor-dev/2014-April/006788.html=

>>> https://lists.torproject.org/pipermail/tor-dev/2014-May/006812.html
>>>
>>
>> I've been thinking a little about this problem. It seems like one simp=
le
>> solution would be to find a way of combining the
>> descriptors/introduction points from multiple Tor HS instances into on=
e
>> hidden service descriptor from which the client can pick intro points =
at
>> random.
>>
>> To implement such a solution like that, there needs to be a means for
>> the hidden service instances to communicate with other. They can then
>> either selected a common set of intro points or combine the individual=

>> sets of intro points selected by each instance.
>>
>> In one straight forward implementation a hidden service operator could=

>> set up a Tor relay and a hidden service on each of their load-balancin=
g
>> nodes. These load balancing hidden services SHOULD have different hidd=
en
>> service keys and could use stealth authentication for privacy (so that=

>> their introduction points are encrypted).
>>
>> A management server would contain the actual hidden service key but
>> would not need to run any hidden services. The role of the management
>> server would be to regularly fetch descriptors for the load-balancing
>> hidden services, combine the introduction points into a single
>> descriptor for the hidden service which is then published as normal.
>>
>> After the signature on the hidden service descriptor is verified there=

>> is the hidden service protocol doesn't user the permanent key/onion
>> address. As such there is no need for each of the relays to have a cop=
y
>> of the hidden service key.
>>
>> I think this provides a couple of advantages:
>>
>> 	- The hidden service key only needs to be in one place. The machine
>> holding the key would generate very little traffic and would not be
>> locatable by the publically known hidden service attacks.
>>
>> 	- The hidden service key could be stored encrypted on the management
>> server.
>>
>> 	- If any of the load balancing relays are compromised an operator
>> simply needs to stop including its introduction points in the
>> descriptor. This should minimise the need to 'revoke' hidden service k=
eys.
>>
>> The number of introduction points in a HS descriptor is currently
>> limited to 10 in Tor. This sets a limit on the number of load-balancin=
g
>> nodes that could be deployed at present.
>>
>> This approach doesn't require any changes to the Tor code base at all.=
 I
>> hope to implement a management server in the next few weeks to test ho=
w
>> this works in practice.
>>
>> It would be great to get any feedback about this proposal!
>>
>=20
> Interesting approach.
>=20
> I especially like the fact that it doesn't need little-t-tor
> modifications to work, which means that we can experiment with it and
> if its UX and threat model works out for people we can even change
> little-t-tor to improve it.
>=20
> The main problem I see is that all parts of the system will be racing
> each other constantly. A harmful race condition I can see, is this one:=

>=20
> a) Management server is about to make a new superdescriptor.
>    Management server polls HSDirs for descriptors and
>    waits till it receives them.
>=20
> b) At the same time as (a), one of the HS nodes abandons an
>    introduction point (because it expired or because it went down),
>    and publishes a new descriptor with a new intro point.
>=20
> c) The management server is done forming the superdescriptor and
>    publishes it. The management server was not aware of event (b) and
>    hence the superdescriptor includes the old introduction point of
>    that node, which means that clients who pick that IP will fail.
>=20
> I think that this is not catastrophic failure because a client will
> move on to the next intro point in the list.

Those kind of race conditions will definitely be an issue to some
extent. Is there any data about how often IP circuits fail in practice?

>=20
> It's also worth noting that in this system, availability is enforced
> by having the client try the next introduction point. That is if an HS
> node is down, introductions to it will fail and the client will only
> be successful when she moves to the intro point of an active HS
> node. Hence, we should ensure that clients don't spend 5 mins before
> moving to the next intro point, or don't abandon connecting after
> failing to connect to two bad intro points.

Is there a limitation to introduction point attempts or timeout's in
little-t tor at present?
>=20
> Other race conditions here, involve the management server missing
> descriptors from certain HS nodes and publishing incomplete
> descriptors, or not publishing anything till all HS node descriptors
> have been retrieved. We should make sure that the management server
> polls often enough that these problems are minimized.

For now I'll poll every 15 minutes. If the management server detects
that the IP set has changed it can then republish the descriptor.
Hopefully that will minimise how often clients receive expired IP's.

>=20
> On security now, it's worth having in mind that HSDir servers and HS
> clients can now monitor presense of HS nodes, by looking at the
> content of superdescriptors. Also, it will probably be easy to learn
> whether an HS is scaleable and approximately how many nodes it has,
> just by looking at the number of IPs.
>=20
> BTW, why do you say that "a hidden service operator could set up a Tor
> relay and a hidden service on each of their load-balancing nodes"? Why
> do we need a Tor relay in this case?

Ah I misspoke, I meant to say a Tor instance not a Tor relay.

>=20
> Thanks for the idea.
>=20
> Looking forward to see what happens with this!
>=20

For the management server to retrieve and upload descriptors via an
unpatched Tor instance #3523 and #14847 would need to be merged.

Thanks for the feedback,
Donncha



--2HrQKJo39fDw2obW1Jd8b0C7Hj68MDlrK
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQGcBAEBCgAGBQJVAt0oAAoJENYNZOc0WPKFsTMMAI39/Q0oHgsbUa+62nP7p4DP
ZEP2qjJlONTLa/IyeP10tZkqtT27wHQOzEW0qHsC16tbR6ZIgzv46ECbW6bkakZW
CNeiGeKp97FE1cgheDdDz8r9C0UUrS5afOfMR5W5cq4X4K0Qow/wwpadZphFDE33
iXEhpY9H54XKKINRUIU9uViXxg8cAwqh6oxk22t8hPJyf92g0d97Be402osOZ+Q7
A2ugUQxL6QBoC7a3C1GJkN5xuUSkcnLO+s8g4QtWeUwTIBZ2AfmyGMr1SvO3AvPF
k1RycXofOwHJW+bbsxgxAjk8c7H4k8cG370tK0kSLo4xpGGh6Pu8G0Bf+1ED2N+J
MYADqzH7yMvVB4mF9TOTC0HzlSCdnWH6/Zy2W+QZ9Z7dNa0tveVZc0SlDoCqlh25
7dt/rFrw+eciKclJZgzsTGilDibMf+//10HkkShAnTLBmXSRpBqFNmY674gpxmCp
zI6V1GMB9LpQbSmQrWd01qi0xpLDadD+RHwdOV0Ywg==
=FHU3
-----END PGP SIGNATURE-----

--2HrQKJo39fDw2obW1Jd8b0C7Hj68MDlrK--

--===============0403785364008158897==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

-- 
tor-talk mailing list - tor-talk@lists.torproject.org
To unsubscribe or change other settings go to
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk

--===============0403785364008158897==--

