the forums at degreez.net
http://forums.degreez.net/

IO Error using pre-allocate and priority list
http://forums.degreez.net/viewtopic.php?f=4&t=7155
Page 1 of 1

Author:  steveholt [ Thu May 10, 2007 6:07 pm ]
Post subject:  IO Error using pre-allocate and priority list

I always get an error on startup when using an allocation type of 'pre-allocate' with a priority list that has files set to download never. I've tried to debug it, but I can't understand the purpose of some of the variables. It seems like the 'download never' files are not initially created (though the directories and other files are), then the allocation function tries to fill all the files. I guess the allocation function (allocfunc) needs to be told not to fill disabled files, but I couldn't figure out how to do that.

Here is the output from btdownloadheadless, with DEBUG turned on for Storage and StorageWrapper. The torrent has 7 files, and only the 7th was set to download (the others were disabled (set to -1)). The error its getting is for the first file, which has no pieces in common with the 7th file.

Code:
saving:         <torrent> (216.3 MB)
percent done:   100.0
time left:      allocating disk space
download to:    <torrent-directory>
download rate:
upload rate:
share rating:
seed status:
peer status:
Traceback (most recent call last):
  File "BitTornado/BT1/Storage.py", line 186, in _open
    return open(file, mode)
IOError: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'
Traceback (most recent call last):
  File "BitTornado/BT1/Storage.py", line 240, in _get_file_handle
    f = self._open(file, 'rb+')
  File "BitTornado/BT1/Storage.py", line 186, in _open
    return open(file, mode)
IOError: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'
writing <torrent-directory>/<fle1> from 0 to 122





ERROR:
IO Error: unable to open <torrent-directory>/<file1>: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'

saving:         <torrent> (216.3 MB)
percent done:   100.0
time left:      allocating disk space
download to:    <torrent-directory>
download rate:
upload rate:
share rating:
seed status:
peer status:





ERROR:
IO Error: unable to open <torrent-directory>/<file1>: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'

saving:         <torrent> (216.3 MB)
percent done:   0
time left:      Download Failed!
download to:    <torrent-directory>
download rate:
upload rate:
share rating:
seed status:
peer status:

Author:  steveholt [ Fri May 11, 2007 5:32 pm ]
Post subject: 

By the way, I searched around, and these posts seem to be related to this problem:

http://forums.degreez.net/viewtopic.php?t=5181

http://forums.degreez.net/viewtopic.php?t=4320

http://forums.degreez.net/viewtopic.php?t=4866

http://forums.degreez.net/viewtopic.php?t=4684

http://forums.degreez.net/viewtopic.php?t=3083

I don't see any reports of this having been fixed, even though some of them are quite old, so I assume it hasn't and this is the same problem that has been around for a while. The difference here is that I have found a way to make it fail immediately on startup, so I hope it will be easier to diagnose than these previous ones.

Author:  steveholt [ Wed May 16, 2007 2:27 am ]
Post subject:  IO Error using pre-allocate and priority list (patched)

I think I've managed to fix this using the patch below, but I'd really like some feedback on whether I've done something crazy here (or maybe there's a better way?). I've tested it using pre-allocate with priorities, and using normal allocation with priorities, and all seems to be fine. Here's a summary of what I did:

First, I added a new function in FileSelector to initialize the priority list. This function, initialize_priorities_now(), is a stripped down version of set_priorities_now(). It calls a new function, _initialize_disabled_files(), which is a stripped down version of _set_disabled_files(). The initialization then gets called from the initFiles() function of download_bt1 before the StorageWrapper initialization is called (which is where the allocation is done).

Next, I modified Storage to not write to files that are disabled. I did this by using the disabled_files return value from _get_disabled_files() as the working_range for disabled files. This only has the cached partial-begin and -end pieces for the disabled files (so the hash check will succeed), but not the main parts of the file. It's complicated a bit by the disabled_files return value not using the same format as the working_ranges, so I had to rearrange it in a few places.

Hopefully that makes some sense (or maybe I just muddied the waters more :( ). I'm happy to answer any questions, and please, give me feedback! Here's the patch:

Code:
Index: BitTornado/BT1/Storage.py
===================================================================
--- BitTornado/BT1/Storage.py   (revision 98)
+++ BitTornado/BT1/Storage.py   (working copy)
@@ -273,9 +273,10 @@
             begin, end, offset, file = self.ranges[p]
             if begin >= stop:
                 break
-            r.append(( file,
-                       offset + max(pos, begin) - begin,
-                       offset + min(end, stop) - begin   ))
+            if end > pos:
+                r.append(( file,
+                           offset + max(pos, begin) - begin,
+                           offset + min(end, stop) - begin   ))
             p += 1
         return r
 
@@ -370,7 +371,7 @@
             else:
                 midfile = os.path.join(self.bufferdir,str(f))
                 working_range = [(start, end, 0, midfile)]
-                disabled_files.append((midfile, start, end))
+                disabled_files.append((start, end, 0, midfile))
                 length = end - start
                 self.sizes[midfile] = length
                 piece = pieces[0]
@@ -381,7 +382,7 @@
                 end_b = pieces[1]*self.piece_length
                 startfile = os.path.join(self.bufferdir,str(f)+'b')
                 working_range_b = [ ( start, end_b, 0, startfile ) ]
-                disabled_files.append((startfile, start, end_b))
+                disabled_files.append((start, end_b, 0, startfile))
                 length = end_b - start
                 self.sizes[startfile] = length
                 offset = length
@@ -394,7 +395,7 @@
                 start_e = pieces[-1] * self.piece_length
                 endfile = os.path.join(self.bufferdir,str(f)+'e')
                 working_range_e = [ ( start_e, end, 0, endfile ) ]
-                disabled_files.append((endfile, start_e, end))
+                disabled_files.append((start_e, end, 0, endfile))
                 length = end - start_e
                 self.sizes[endfile] = length
                 piece = pieces.pop(-1)
@@ -445,7 +446,7 @@
         r = self._get_disabled_ranges(f)
         if not r:
             return
-        for file, begin, end in r[2]:
+        for begin, end, offset, file in r[2]:
             if not os.path.isdir(self.bufferdir):
                 os.makedirs(self.bufferdir)
             if not exists(file):
@@ -456,7 +457,7 @@
                 self.tops[file] = getsize(file)
             if not self.mtimes.has_key(file):
                 self.mtimes[file] = getmtime(file)
-        self.working_ranges[f] = r[0]
+        self.working_ranges[f] = r[2]
 
     reset_file_status = _reset_ranges
 
@@ -497,7 +498,7 @@
             if not self.files[i][1]:    # length == 0
                 continue
             if self.disabled[i]:
-                for file, start, end in self._get_disabled_ranges(i)[2]:
+                for start, end, offset, file in self._get_disabled_ranges(i)[2]:
                     pfiles.extend([basename(file),getsize(file),int(getmtime(file))])
                 continue
             file = self.files[i][0]
@@ -550,7 +551,7 @@
 
             for i in xrange(len(self.files)):
                 if self.disabled[i]:
-                    for file, start, end in self._get_disabled_ranges(i)[2]:
+                    for start, end, offset, file in self._get_disabled_ranges(i)[2]:
                         f1 = basename(file)
                         if ( not pfiles.has_key(f1)
                              or not test(pfiles[f1],getsize(file),getmtime(file)) ):
Index: BitTornado/BT1/FileSelector.py
===================================================================
--- BitTornado/BT1/FileSelector.py   (revision 98)
+++ BitTornado/BT1/FileSelector.py   (working copy)
@@ -104,6 +104,30 @@
         self.new_partials = None
         
 
+    def _initialize_files_disabled(self, old_priority, new_priority):
+        old_disabled = [p == -1 for p in old_priority]
+        new_disabled = [p == -1 for p in new_priority]
+        files_updated = False       
+        try:
+            for f in xrange(self.numfiles):
+                if new_disabled[f] and not old_disabled[f]:
+                    self.storage.disable_file(f)
+                    files_updated = True
+                if old_disabled[f] and not new_disabled[f]:
+                    self.storage.enable_file(f)
+                    files_updated = True
+        except (IOError, OSError), e:
+            if new_disabled[f]:
+                msg = "can't open partial file for "
+            else:
+                msg = 'unable to open '
+            self.failfunc(msg + self.files[f][0] + ': ' + str(e))
+            return False
+        if files_updated:
+            self.storage.reset_file_status()
+        return True       
+
+
     def _set_files_disabled(self, old_priority, new_priority):
         old_disabled = [p == -1 for p in old_priority]
         new_disabled = [p == -1 for p in new_priority]
@@ -189,6 +213,15 @@
         return new_piece_priority       
 
 
+    def initialize_priorities_now(self, new_priority = None):
+        if not new_priority:
+            return
+        old_priority = self.priority
+        self.priority = new_priority
+        if not self._initialize_files_disabled(old_priority, new_priority):
+            return
+#        self.piece_priority = self._set_piece_priority(new_priority)
+
     def set_priorities_now(self, new_priority = None):
         if not new_priority:
             new_priority = self.new_priority
Index: BitTornado/download_bt1.py
===================================================================
--- BitTornado/download_bt1.py   (revision 98)
+++ BitTornado/download_bt1.py   (working copy)
@@ -544,6 +544,9 @@
                 data = data.get('resume data')
                 if data:
                     self.fileselector.unpickle(data)
+
+            if self.priority:
+                self.fileselector.initialize_priorities_now(self.priority)
                 
         self.checking = True
         if old_style:

Author:  steveholt [ Thu May 17, 2007 2:07 am ]
Post subject: 

I edited the patch in the previous post to add one additional fix (the first block in the patch). It seems this was causing some problems, though I didn't see them until I ran a torrent with a file boundary on a piece boundary. I've updated it so the intervals it writes are guaranteed to only be in desired range.

Author:  Amberwolf [ Fri Jul 20, 2007 2:36 pm ]
Post subject: 

I don't suppose there is a very simple way for a non-programmer user to integrate that patch into the BitTornado client? One that doesn't involve learning how to use a compiler, or write code? ;)

I ask because I've had a whole bunch of repeated issues with the "download never" function of BitTornado used with multi-file torrents, and I would like to see if this would resolve the problems.

Currently using 0.3.17.

Author:  steveholt [ Fri Jul 27, 2007 2:51 pm ]
Post subject: 

Amberwolf wrote:
I don't suppose there is a very simple way for a non-programmer user to integrate that patch into the BitTornado client? One that doesn't involve learning how to use a compiler, or write code? ;)


You shouldn't need to compile anything or write any code. All you need to do is know how to use the patch program (available in Linux, check out gnuwin32 for a Windows version).

If you have trouble with that, I might be able to patch the 0.3.17 files for you, though I can't post them here so you'd have to PM me your email address.

Author:  seraulu1 [ Wed Mar 10, 2010 4:01 am ]
Post subject:  ''liar hater''

:) :) :) ,




Thanks for the information!
how to hypnotize people

Page 1 of 1 All times are UTC - 7 hours [ DST ]
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/