 |
the forums at degreez.net
|
| View previous topic :: View next topic |
| Author |
Message |
steveholt
Joined: 24 Mar 2006
Posts: 22
|
 Posted: Fri May 11, 2007 12:07 am Post subject: IO Error using pre-allocate and priority list |
 |
|
I always get an error on startup when using an allocation type of 'pre-allocate' with a priority list that has files set to download never. I've tried to debug it, but I can't understand the purpose of some of the variables. It seems like the 'download never' files are not initially created (though the directories and other files are), then the allocation function tries to fill all the files. I guess the allocation function (allocfunc) needs to be told not to fill disabled files, but I couldn't figure out how to do that.
Here is the output from btdownloadheadless, with DEBUG turned on for Storage and StorageWrapper. The torrent has 7 files, and only the 7th was set to download (the others were disabled (set to -1)). The error its getting is for the first file, which has no pieces in common with the 7th file.
| Code: |
saving: <torrent> (216.3 MB)
percent done: 100.0
time left: allocating disk space
download to: <torrent-directory>
download rate:
upload rate:
share rating:
seed status:
peer status:
Traceback (most recent call last):
File "BitTornado/BT1/Storage.py", line 186, in _open
return open(file, mode)
IOError: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'
Traceback (most recent call last):
File "BitTornado/BT1/Storage.py", line 240, in _get_file_handle
f = self._open(file, 'rb+')
File "BitTornado/BT1/Storage.py", line 186, in _open
return open(file, mode)
IOError: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'
writing <torrent-directory>/<fle1> from 0 to 122
ERROR:
IO Error: unable to open <torrent-directory>/<file1>: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'
saving: <torrent> (216.3 MB)
percent done: 100.0
time left: allocating disk space
download to: <torrent-directory>
download rate:
upload rate:
share rating:
seed status:
peer status:
ERROR:
IO Error: unable to open <torrent-directory>/<file1>: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'
saving: <torrent> (216.3 MB)
percent done: 0
time left: Download Failed!
download to: <torrent-directory>
download rate:
upload rate:
share rating:
seed status:
peer status:
|
|
|
| Back to top |
|
 |
steveholt
Joined: 24 Mar 2006
Posts: 22
|
|
| Back to top |
|
|
steveholt
Joined: 24 Mar 2006
Posts: 22
|
| Posted: Wed May 16, 2007 8:27 am Post subject: IO Error using pre-allocate and priority list (patched) |
|
|
I think I've managed to fix this using the patch below, but I'd really like some feedback on whether I've done something crazy here (or maybe there's a better way?). I've tested it using pre-allocate with priorities, and using normal allocation with priorities, and all seems to be fine. Here's a summary of what I did:
First, I added a new function in FileSelector to initialize the priority list. This function, initialize_priorities_now(), is a stripped down version of set_priorities_now(). It calls a new function, _initialize_disabled_files(), which is a stripped down version of _set_disabled_files(). The initialization then gets called from the initFiles() function of download_bt1 before the StorageWrapper initialization is called (which is where the allocation is done).
Next, I modified Storage to not write to files that are disabled. I did this by using the disabled_files return value from _get_disabled_files() as the working_range for disabled files. This only has the cached partial-begin and -end pieces for the disabled files (so the hash check will succeed), but not the main parts of the file. It's complicated a bit by the disabled_files return value not using the same format as the working_ranges, so I had to rearrange it in a few places.
Hopefully that makes some sense (or maybe I just muddied the waters more  ). I'm happy to answer any questions, and please, give me feedback! Here's the patch:
| Code: | Index: BitTornado/BT1/Storage.py
===================================================================
--- BitTornado/BT1/Storage.py (revision 98)
+++ BitTornado/BT1/Storage.py (working copy)
@@ -273,9 +273,10 @@
begin, end, offset, file = self.ranges[p]
if begin >= stop:
break
- r.append(( file,
- offset + max(pos, begin) - begin,
- offset + min(end, stop) - begin ))
+ if end > pos:
+ r.append(( file,
+ offset + max(pos, begin) - begin,
+ offset + min(end, stop) - begin ))
p += 1
return r
@@ -370,7 +371,7 @@
else:
midfile = os.path.join(self.bufferdir,str(f))
working_range = [(start, end, 0, midfile)]
- disabled_files.append((midfile, start, end))
+ disabled_files.append((start, end, 0, midfile))
length = end - start
self.sizes[midfile] = length
piece = pieces[0]
@@ -381,7 +382,7 @@
end_b = pieces[1]*self.piece_length
startfile = os.path.join(self.bufferdir,str(f)+'b')
working_range_b = [ ( start, end_b, 0, startfile ) ]
- disabled_files.append((startfile, start, end_b))
+ disabled_files.append((start, end_b, 0, startfile))
length = end_b - start
self.sizes[startfile] = length
offset = length
@@ -394,7 +395,7 @@
start_e = pieces[-1] * self.piece_length
endfile = os.path.join(self.bufferdir,str(f)+'e')
working_range_e = [ ( start_e, end, 0, endfile ) ]
- disabled_files.append((endfile, start_e, end))
+ disabled_files.append((start_e, end, 0, endfile))
length = end - start_e
self.sizes[endfile] = length
piece = pieces.pop(-1)
@@ -445,7 +446,7 @@
r = self._get_disabled_ranges(f)
if not r:
return
- for file, begin, end in r[2]:
+ for begin, end, offset, file in r[2]:
if not os.path.isdir(self.bufferdir):
os.makedirs(self.bufferdir)
if not exists(file):
@@ -456,7 +457,7 @@
self.tops[file] = getsize(file)
if not self.mtimes.has_key(file):
self.mtimes[file] = getmtime(file)
- self.working_ranges[f] = r[0]
+ self.working_ranges[f] = r[2]
reset_file_status = _reset_ranges
@@ -497,7 +498,7 @@
if not self.files[i][1]: # length == 0
continue
if self.disabled[i]:
- for file, start, end in self._get_disabled_ranges(i)[2]:
+ for start, end, offset, file in self._get_disabled_ranges(i)[2]:
pfiles.extend([basename(file),getsize(file),int(getmtime(file))])
continue
file = self.files[i][0]
@@ -550,7 +551,7 @@
for i in xrange(len(self.files)):
if self.disabled[i]:
- for file, start, end in self._get_disabled_ranges(i)[2]:
+ for start, end, offset, file in self._get_disabled_ranges(i)[2]:
f1 = basename(file)
if ( not pfiles.has_key(f1)
or not test(pfiles[f1],getsize(file),getmtime(file)) ):
Index: BitTornado/BT1/FileSelector.py
===================================================================
--- BitTornado/BT1/FileSelector.py (revision 98)
+++ BitTornado/BT1/FileSelector.py (working copy)
@@ -104,6 +104,30 @@
self.new_partials = None
+ def _initialize_files_disabled(self, old_priority, new_priority):
+ old_disabled = [p == -1 for p in old_priority]
+ new_disabled = [p == -1 for p in new_priority]
+ files_updated = False
+ try:
+ for f in xrange(self.numfiles):
+ if new_disabled[f] and not old_disabled[f]:
+ self.storage.disable_file(f)
+ files_updated = True
+ if old_disabled[f] and not new_disabled[f]:
+ self.storage.enable_file(f)
+ files_updated = True
+ except (IOError, OSError), e:
+ if new_disabled[f]:
+ msg = "can't open partial file for "
+ else:
+ msg = 'unable to open '
+ self.failfunc(msg + self.files[f][0] + ': ' + str(e))
+ return False
+ if files_updated:
+ self.storage.reset_file_status()
+ return True
+
+
def _set_files_disabled(self, old_priority, new_priority):
old_disabled = [p == -1 for p in old_priority]
new_disabled = [p == -1 for p in new_priority]
@@ -189,6 +213,15 @@
return new_piece_priority
+ def initialize_priorities_now(self, new_priority = None):
+ if not new_priority:
+ return
+ old_priority = self.priority
+ self.priority = new_priority
+ if not self._initialize_files_disabled(old_priority, new_priority):
+ return
+# self.piece_priority = self._set_piece_priority(new_priority)
+
def set_priorities_now(self, new_priority = None):
if not new_priority:
new_priority = self.new_priority
Index: BitTornado/download_bt1.py
===================================================================
--- BitTornado/download_bt1.py (revision 98)
+++ BitTornado/download_bt1.py (working copy)
@@ -544,6 +544,9 @@
data = data.get('resume data')
if data:
self.fileselector.unpickle(data)
+
+ if self.priority:
+ self.fileselector.initialize_priorities_now(self.priority)
self.checking = True
if old_style:
|
|
|
| Back to top |
|
|
steveholt
Joined: 24 Mar 2006
Posts: 22
|
| Posted: Thu May 17, 2007 8:07 am Post subject: |
|
|
| I edited the patch in the previous post to add one additional fix (the first block in the patch). It seems this was causing some problems, though I didn't see them until I ran a torrent with a file boundary on a piece boundary. I've updated it so the intervals it writes are guaranteed to only be in desired range. |
|
| Back to top |
|
|
Amberwolf
Joined: 20 Jul 2007
Posts: 5
|
| Posted: Fri Jul 20, 2007 8:36 pm Post subject: |
|
|
I don't suppose there is a very simple way for a non-programmer user to integrate that patch into the BitTornado client? One that doesn't involve learning how to use a compiler, or write code? 
I ask because I've had a whole bunch of repeated issues with the "download never" function of BitTornado used with multi-file torrents, and I would like to see if this would resolve the problems.
Currently using 0.3.17. |
|
| Back to top |
|
|
steveholt
Joined: 24 Mar 2006
Posts: 22
|
| Posted: Fri Jul 27, 2007 8:51 pm Post subject: |
|
|
| Amberwolf wrote: | | I don't suppose there is a very simple way for a non-programmer user to integrate that patch into the BitTornado client? One that doesn't involve learning how to use a compiler, or write code? |
You shouldn't need to compile anything or write any code. All you need to do is know how to use the patch program (available in Linux, check out gnuwin32 for a Windows version).
If you have trouble with that, I might be able to patch the 0.3.17 files for you, though I can't post them here so you'd have to PM me your email address. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
