the forums at degreez.net

Every now and then we developers are re-reminded that the internet is undergoing a phase change, from IPv4 ( 12.34.56.78 ) to the new IPv6 protocol. BitTornado has some IPv6 compatibility, but it isn't complete simply because, in order to be complete, some extensions will need to be made to the tracker protocol. I'm going to propose some extensions here and I'm hoping for some good comments from you folk.

ASSUMPTIONS: I'm going to make a very important assumption here: That almost all machines on IPv6 networks will NOT be subject to network address translation (NAT). They may be firewalled, but nearly none should be on unrouteable addresses.

IPv4 and IPv6: We will be seeing some machines with dual connectivity over IPv4 and IPv6, and some on IPv6 alone. (Obviously we'll also be seeing machines on IPv4 only; that's basically what we have now.)

ANNOUNCE BEHAVIOR AND EXTENSION:

If a peer does not have IPv4 connectivity, or if the tracker does not have IPv4 connectivity, then the peer should connect to the tracker via IPv6. No additional information is required in the announce for this instance.

If both peer and tracker have IPv4 connectivity, then the peer should announce to the tracker VIA IPv4. This is because, just as for usual IPv4 connections, the peer may be behind a network address translator and the tracker needs to be able to test the peer's connectivity.

If the peer has IPv6 capability, it should add the key "ipv6=" to the announce, containing the peer's IPv6 address in standard format, as shown here: http://www.olympus-zone.net/page_1058_en_Blue.html

The tracker should at some point perform a NAT check (although there would be no NAT) on the IPv6 address to confirm the peer actually exists at that address.

RESPONSE BEHAVIOR AND EXTENSION:

A new key will be added to the response: 'peers6'. If a peer connects to the tracker via IPv6, it will receive a peer list via the 'peers6' key; if a peer connects to the tracker via IPv4 along with the "ipv6=" key, it will receive peers via both the original 'peers' and the new 'peers6' key. The tracker will need to determine an optimal mix between the number of peers returned in each category.

The content of the 'peers6' key is a string value, much like the string returned in 'peers' when the "compact=1" is used, except each "record" in the string is 18 bytes long instead of six, consisting of a 16-byte IPv6 address and a two-byte port number.

NOTE: Due to the way IPv6 addresses are allocated, this may not be very space efficient. Internal gzip compression of this data was considered, but it would probably be better to perform gzip compression on the entire response as is already often done by BitTorrent clients.

From comments on irc://irc.freenode.net/bittorrent :

The one potential drawback to the system I proposed above is it requires the tracker to perform NAT checks for IPv6 connections for IPv6+IPv4 peers. While NAT checking is always a good idea for a tracker, sometimes it isn't feasible.

Suggestion:

New key in response: "ipv6connectrequest", with the value 1. When the peer receives this, it then performs a request to the tracker with "event=connectback". The tracker will use this to verify the peer's IPv6 address by sniffing it, and will respond with an empty 200 status return.

Well, actually the basic bittorrent protocol was perfectly ipv6-aware, the main problems came with the compact extension, and the fact that there are mixed swarms.

If DNS were set up properly for the whole internet, we could just work with the domainnames and every client could decide for itself which kind of connection it wants... unfortunately it's not the case.



Should be correct.


Obvious...



Actually I don't like the idea of having IPv4 as the default way of announcing. Especially since IPv6connections usually are "cheaper".

However I also dislike the whole idea of NAT-checking. It just is additional work for the tracker, and it's a work it isn't even meant to be made by it.
It's not for the tracker to decide which clients to connect to... however that's a different matter and has nothing to do with IPv6.


Well, if you do it the above way this would be necessarry... however I have another idea (see below).



I don't think it's a good idea to introduce a new response key, since it would exclude all clients with a working IPv6 implementation, which doesn't understand the new key.
And sending them in this "compressed" format doesn't seem to be very consistent with the usual behaviour of tracker responses (except the compact mode).


Here's my suggestion: Seperate the swarms!
Instead of trying to make an addition for client and tracker to support both address types at once, just treat it as if there were two seperate swarms, one only consisting of IPv4 clients and one only of IPv6 clients.

Technically I'm thinking of the following: If a client gets a torrent with a tracker URL, that has an A-record and an AAAA-record, it just makes two completely seperate announces, one to the IPv4-address, and one to the IPv6-address. And of course it will get two answers, one consisting of IPv4-adresses, and one consisting of IPv6-addresses.

Cons:

• More traffic for both, client and tracker, since every announce has to be done twice
• Possibility of connectiong to the same peer twice (IPv4 and Ipv6). Though probably here we can use the PeerID somehow and drop the slower of the two connections.

Pros:

• Should be easier to implement. The tracker can just keep two completely seperate lists internally. I don't know what kind of data type you had in mind for your idea above, but if you kept it as a single list, you have to search every entry for an IPv6-address, if they are requested.
• More consistency and future-proof. You wrote, that in case both (client/tracker) are only reachable via IPv6 there shouldn't be any change in the behaviour, so why should there if they are (additionally) reachable by IPv4?
• You could operate two physically seperate trackers. The AAAA-record and the A-record of the tracker doesn't need to point to the same physical machine.
• Downward-compatibility. If a client has an IPv6-only connection it will (obviously) only announce to the IPv6 tracker, while an IPv4-only client will only contact the IPv4-tracker... both doesn't need to worry about spheres they aren't able to reach.
• You can keep the compact-response. (Personally I don't like this format, however some people do. ) If you put both sort of peers into the same tracker response, you have to use the full format. But onot if you keep them seperate.

Probably it would be good to implement a new IPv6-compact-response. Of course it would need some sort of mechanism (header?) to avoid other clients, that have basic IPv6 connectivity but no knowlede of this behaviour, from assuming it's a response consisting of IPv4-addresses. I've seen quite a few clients, that weren't programmed very good: They connected to the tracker via IPv6, only because they used IPv6-aware HTTP libraries, but the programmer didn't really think/know of IPv6 for the core program.

However there is one problem: We can be sure, that the connections to the tracker will be always made with the correct protocol, but we can't know about the IP-parameter in the announce URL. Even though a client is able to connect via IPv6 to a tracker, there is no guarantee, that the reported IP-parameter isn't an v4-address, or vice versa. And there is still the possibility, that it's a domainname, which could resolve to both types of address. So probably it would be wise for the tracker to check this parameter and return one of the following error-messages: "Error: Reported an IPv4-address through IPv6-connection", "Error: Reported an IPv6-address through IPv4-connection", "Error: reported Domainname doesn't resolve to IPv6-address", "Error: reported Domainname doesn't resolve to IPv4-address".

One last thing: IF we assume, that we are only using one tracker (instead of two seperate ones), it could be wise to add an "addipv4peers=1" to the IPv6-tracker-request, which tells the tracker, that it also want's a few v4-peers in the result. But basically there shouldn't be much cases in which this is useful. Maybe if the user isn't able (or doesn't want) to recieve incoming IPv4-connections AND doesn't want to connect to the IPv4 tracker. Maybe we should drop this idea.:


Okay, basically this was my suggestion. Maybe it's better, maybe not. However I think it is..

I'm wondering: It's been about 4 month and there was no reply... did you drop the IPv6 ideas?

No, just been busy w/ other projects and not working on the client.



NAT checking actually reduces the overall load on the tracker a bit. The tracker only sends routeable peer addresses to requesters, and as a result the requesters don't announce as often.

Your proposal for separate swarms for IPv4 and IPv6 does indeed increase tracker overhead, and since the tracker is often the bottleneck for torrents I don't think it's a great idea, especially since at some point a LOT of peers will have both IPv4 and IPv6 connectability.

This isn't really convincing.
For NAT checking you don't only have network overhead, but it also requires CPU and memory... and for what? The result isn't even reliable: The tracker geting a connection doesn't mean, that the peer is also reachable for other peers, or vice versa: Maybe it's perfectly reachable, but just not for the tracker.
And how many unreachable peers are there in the first place? Experienced users should know how to open the firewall, so in the end there are probably 5-10% unreachable... and connection to everyone, just to sort out those few?
IMHO it's more work, than it is worth in the end.

A well, who cares... that's not the topic here. This thread is about IPv6 (where NAT checking shouldn't be necessary at all).



For me it's been quite some years, that I experienced trackers as bottlenecks. Nowadays they are pretty reliable.

Though the overhead is only really noticable, as long as you only have one tracker.
My thought was, that it's more likely, that people are going to have seperate trackers for the two protocols anyways. Like keeping their existent IPv4 tracker on the already existent server (which doesn't support IPv6) and just adding a new DNS-Record for an IPv6-Tracker, on a completely different machine, maybe not even operated by the same person.

At least I think it is a better way, than extending the protocol for a situation, which is only limited in its occurence and will vanish at some time.



Unfortunately I fear this won't happen. IPv6 is still very unpopular and it looks like the only way for it to get accepted will be some sort of forced migration... but that's a different issue.
Well, even if there are "a lot of peers" with both addresses, it doesn't necessarily mean, that they are willing to use both connection-types at the same time.

BTW, I think at some time it would by a nice to have "De/activate IPv4/IPv6" buttons in the client, which would work without a restart.

You may not find the idea that NAT checking reduces overall load convincing, but I've seen the statistics when a tracker turns it on.

As for the rest... You have a serious point about "keeping the swarms separate" - it would significantly simplify tracker implementation. But it would still increase the load on trackers by quite a bit, and that's certainly not desirable. I'll have to think about it.

Is it really that much more of a load? If the client really does two announces, than it should use numwant=0 on the second one, because it already has a valid list of peers so most of the remaining traffic is the announce request itself.


But since your main focus seems to be the traffic, I'm currently thinking about an alternative method to do the announce in one step with as few changes to the protocol as possible.
For this I have a question: Since you seem to have access to a few statistics I like to ask: How effective is the compact response format really? Of course the raw data is significantly fewer with only 6 bytes per peer, but usuall the complete HTTP response gets gzipped and I think, that a full bencoded peerlist should get compressed better, possibly nullifying the gains of the compact format.

I rethought my suggestion a bit and came to the following result:
Correct me if I'm wrong, but usually there should be two types of announces, shouldn't it? One to get a list of new peers if the client is short on them, and one to inform the tracker, that the client is still active. While the second one should occur only in defined intervals, specified by the tracker, the first one happens unexpected (for the tracker).
Back to the point: I suggested to make every announce to both, the IPv4 Address and to the IPv6 Address of the tracker, which obviously would double the traffic. While this makes the most sense for a situation in which both protocols are handled by different machines, it's kind of wasteful if there is indeed only one tracker.
So here my modified suggestion : Actually I think it's only necessary to make the first announce twice. Since the tracker can identify a peer by it's ID, it should be able to update it's "last announce timer" for both entries at once from the second announce on. The client may decide if it makes it's reannounces via IPv4 od IPv6, depending on the kind of peers it wants.
Unfortunately this won't allow it anymore to set up different trackers for each protocol, unless the client has a way to know if it's really talking to two different machines or not.





But if you still don't like my idea of "seperated swarms" I thought of a different way:
I tried to think of a way with as few changes to the currenct protocol as possible.
At first a few obvious things I just repeat for completness: If the tracker is listening on just one protocol there shouldn't be any change in the default behaviour. The client should do a normal announce.
If the client is just reachable by one protocol (most likely IPv4 ) the same applies.

Now to the problem case: Both tracker and client are reachable by both protocols.
IIRC you suggested adding an "ipv6="-parameter. I didn't like this and thought a bit. Currently there are three valid values for the existing "ip="-parameter (IPv4-address, IPv6-address and a DNS-Name) and none of these can contain commas. So my suggestion is: Instead of adding a completely new parameter to the URL, we can just change the existing "ip="-parameter, so that it can not only contain one value but instead a list of values, separated by commas! If a multi-protocol client wants to connect to a multi-protocol tracker (which it should know by the DNS-entries), it specifies a whole list of all its listening IP-addresses (including the one, the tracker connection is made with) to the tracker. It shall not do this, if the tracker is only reachable by one protocol!
The tracker must save this list, especially the order of the parameters. But before it should check them: If one (or maybe even more) of the specified IPv4-addresses is a private address (10.x.x.x, 192.168.x.x, and so on) and there is no remaining global/valid IPv4 address in the list, and the tracker connection was made via IPv4, it can make a nat check on the connection IP and replace the invalid private address by this one on success (replace=keep the position in the specified list). If the connection wasn't made via IPv4, the tracker should just discard the invalid IPs. It should also make a NAT check, if the specified v4 IP(s) are valid, but differ from the connecting IP(v4) because in this case it's most likely a proxy-situation. If there are DNS-names specified, the tracker may check if they resolve properly.

Okay, so much for the announce request, now to the tracker's answer:
If a connecting client doesn't specify an "ip="-parameter, or the "ip="-parameter doesn't contain a comma (=old format), it is treated as a one-protocol client and only gets results of the same protocol-type.
In the other case (a specified IP-list) the tracker knows, that the client is multi-protocolled and the answer can be one of the following:

1. "(old)plain"
   Defined by: The client didn't specify a "compact="-parameter.
   The tracker sends its result in the standard format, which is described by the very first bittorrent protocol specification (Peers as a list of beencoded dictionaries... and so on) The "ip" entry, the tracker sends with each peer-id is the first one from the list, which was transmitted in its corresponding announce URL.
2. "plain multi-ip-list"
   Defined by: The client did specify the parameter "compact=0".
   In this case we change the response a little bit: Basically it's the same as above; a plain format with all information, but instead of a single string for the "ip"-field, we change it to a beencoded list, containing all the addresses a peer specified, in the order they were specified. (In this case, the client gets all avaiable IPs for each peer, and may search the cheapest route by itself).
3. new "multi-compact"
   Defined by: The client did specify the parameter "compact=2".
   This is a new compact format, based on the existing one. While the old format was just a string of entries, which consisted of 6 bytes for each peer, and therefore could only hold IPv4 addresses, this one is different:
   It's also a string, but it can contain three different entries, which are defined in the following way:
   • IPv4-Addresses are encoded with a "4", followed by four bytes for the address and two bytes for the port
   • IPv6-Addresses are encoded by a "6" followed by 16 bytes for the address and two bytes for the port
   • DNS-Names are encoded by a "D", followed by one byte for the length, a variable-length domain name (those can be only 255 bytes long, so one byte for the length is enough) and two bytes for the port
   For the decision which IPs are transmitted, the same applies, which were said for the first case.

Even though I don't really like to make too much changes, I think it could be wise to add some sort of "peer-encoding"-key to the tracker's reply, to inform the client, which kind of enconding actually was used ("(old) plain", "plain multi-ip-list", "old compact" or "multi-compact"), this way the client doesn't need to spend to much time on trying to decipher which kind of format it actually got.
Remark: With this suggestion, the tracker should basically get a complete list of all IPs are peer is connectable to and it should respect the specified order. So a peer, which specified a list of IPv4,IPv6 signals that it wishes to be primarily connected by IPv4. The IPv6-Address should only be transmitted to IPv6-only clients. While the order IPv6,IPv4 would mean the other way 'round.
But we also give a client the possibility to request a full list, so it may decide a better route (which could be useful in case of multi-homed clients).


Well, so much from me. Personally I still prefer the seperated swarms method...

I am trying to run IPv6 tracker, but when I try to get newly published torrent, I get the following:

/announce?info_hash=~%08%b8H5%ab%e3%91%85e%fe%e5%ee%0ct%24%01%3f%00%2c&peer_id=-UT180B-%0d%a0%10%02%f1%e4%09W%7f%25%e9%f5&port=61354&uploaded=0&downloaded=0&left=0&corrupt=0&key=AAF9E3CD&event=stopped&numwant=0&compact=1&no_peer_id=1 HTTP/1.1" 200 56 "-" "uTorrent/180B(8205)"
2001:0:4137:9e66:0:fbf1:a537:c944 - - [11/Feb/2008:01:10:17] "GET /scrape?info_hash=~%08%b8H5%ab%e3%91%85e%fe%e5%ee%0ct%24%01%3f%00%2c HTTP/1.1" 200 81 "-" "uTorrent/180B(8205)"
Traceback (most recent call last):
File "/var/lib/python-support/python2.5/BitTornado/RawServer.py", line 144, in listen_forever
self.sockethandler.handle_events(events)
File "/var/lib/python-support/python2.5/BitTornado/SocketHandler.py", line 319, in handle_events
s.handler.data_came_in(s, data)
File "/var/lib/python-support/python2.5/BitTornado/HTTPHandler.py", line 155, in data_came_in
if not c.data_came_in(data) and not c.closed:
File "/var/lib/python-support/python2.5/BitTornado/HTTPHandler.py", line 46, in data_came_in
self.next_func = self.next_func(val)
File "/var/lib/python-support/python2.5/BitTornado/HTTPHandler.py", line 78, in read_header
r = self.handler.getfunc(self, self.path, self.headers)
File "/var/lib/python-support/python2.5/BitTornado/BT1/track.py", line 951, in get
return_type, rsize, params('supportcrypto'))
File "/var/lib/python-support/python2.5/BitTornado/BT1/track.py", line 782, in peerlist
cache = self.cached.setdefault(infohash,[None,None,None])[return_type]
IndexError: list index out of range
Traceback (most recent call last):
File "/var/lib/python-support/python2.5/BitTornado/RawServer.py", line 144, in listen_forever
self.sockethandler.handle_events(events)
File "/var/lib/python-support/python2.5/BitTornado/SocketHandler.py", line 319, in handle_events
s.handler.data_came_in(s, data)
File "/var/lib/python-support/python2.5/BitTornado/HTTPHandler.py", line 155, in data_came_in
if not c.data_came_in(data) and not c.closed:
File "/var/lib/python-support/python2.5/BitTornado/HTTPHandler.py", line 46, in data_came_in
self.next_func = self.next_func(val)
File "/var/lib/python-support/python2.5/BitTornado/HTTPHandler.py", line 78, in read_header
r = self.handler.getfunc(self, self.path, self.headers)
File "/var/lib/python-support/python2.5/BitTornado/BT1/track.py", line 951, in get
return_type, rsize, params('supportcrypto'))
File "/var/lib/python-support/python2.5/BitTornado/BT1/track.py", line 782, in peerlist
cache = self.cached.setdefault(infohash,[None,None,None])[return_type]
IndexError: list index out of range

any ideas? Thanks

same problem here
when client try connect to the tracker over ipv6, i get:

using :


any solution ?
Thanks.

Debian lenny
bittornado_0.3.18-7_all.deb
Maintainer: Cameron Dale