the forums at degreez.net

It is currently Sat Dec 16, 2017 5:03 am

All times are UTC - 7 hours [ DST ]




Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 7 posts ] 
Author Message
PostPosted: Thu May 10, 2007 6:07 pm 
Offline

Joined: Thu Mar 23, 2006 7:52 pm
Posts: 22
I always get an error on startup when using an allocation type of 'pre-allocate' with a priority list that has files set to download never. I've tried to debug it, but I can't understand the purpose of some of the variables. It seems like the 'download never' files are not initially created (though the directories and other files are), then the allocation function tries to fill all the files. I guess the allocation function (allocfunc) needs to be told not to fill disabled files, but I couldn't figure out how to do that.

Here is the output from btdownloadheadless, with DEBUG turned on for Storage and StorageWrapper. The torrent has 7 files, and only the 7th was set to download (the others were disabled (set to -1)). The error its getting is for the first file, which has no pieces in common with the 7th file.

Code:
saving:         <torrent> (216.3 MB)
percent done:   100.0
time left:      allocating disk space
download to:    <torrent-directory>
download rate:
upload rate:
share rating:
seed status:
peer status:
Traceback (most recent call last):
  File "BitTornado/BT1/Storage.py", line 186, in _open
    return open(file, mode)
IOError: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'
Traceback (most recent call last):
  File "BitTornado/BT1/Storage.py", line 240, in _get_file_handle
    f = self._open(file, 'rb+')
  File "BitTornado/BT1/Storage.py", line 186, in _open
    return open(file, mode)
IOError: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'
writing <torrent-directory>/<fle1> from 0 to 122





ERROR:
IO Error: unable to open <torrent-directory>/<file1>: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'

saving:         <torrent> (216.3 MB)
percent done:   100.0
time left:      allocating disk space
download to:    <torrent-directory>
download rate:
upload rate:
share rating:
seed status:
peer status:





ERROR:
IO Error: unable to open <torrent-directory>/<file1>: [Errno 2] No such file or directory: '<torrent-directory>/<file1>'

saving:         <torrent> (216.3 MB)
percent done:   0
time left:      Download Failed!
download to:    <torrent-directory>
download rate:
upload rate:
share rating:
seed status:
peer status:


Top
 Profile  
 
 Post subject:
PostPosted: Fri May 11, 2007 5:32 pm 
Offline

Joined: Thu Mar 23, 2006 7:52 pm
Posts: 22
By the way, I searched around, and these posts seem to be related to this problem:

http://forums.degreez.net/viewtopic.php?t=5181

http://forums.degreez.net/viewtopic.php?t=4320

http://forums.degreez.net/viewtopic.php?t=4866

http://forums.degreez.net/viewtopic.php?t=4684

http://forums.degreez.net/viewtopic.php?t=3083

I don't see any reports of this having been fixed, even though some of them are quite old, so I assume it hasn't and this is the same problem that has been around for a while. The difference here is that I have found a way to make it fail immediately on startup, so I hope it will be easier to diagnose than these previous ones.


Top
 Profile  
 
PostPosted: Wed May 16, 2007 2:27 am 
Offline

Joined: Thu Mar 23, 2006 7:52 pm
Posts: 22
I think I've managed to fix this using the patch below, but I'd really like some feedback on whether I've done something crazy here (or maybe there's a better way?). I've tested it using pre-allocate with priorities, and using normal allocation with priorities, and all seems to be fine. Here's a summary of what I did:

First, I added a new function in FileSelector to initialize the priority list. This function, initialize_priorities_now(), is a stripped down version of set_priorities_now(). It calls a new function, _initialize_disabled_files(), which is a stripped down version of _set_disabled_files(). The initialization then gets called from the initFiles() function of download_bt1 before the StorageWrapper initialization is called (which is where the allocation is done).

Next, I modified Storage to not write to files that are disabled. I did this by using the disabled_files return value from _get_disabled_files() as the working_range for disabled files. This only has the cached partial-begin and -end pieces for the disabled files (so the hash check will succeed), but not the main parts of the file. It's complicated a bit by the disabled_files return value not using the same format as the working_ranges, so I had to rearrange it in a few places.

Hopefully that makes some sense (or maybe I just muddied the waters more :( ). I'm happy to answer any questions, and please, give me feedback! Here's the patch:

Code:
Index: BitTornado/BT1/Storage.py
===================================================================
--- BitTornado/BT1/Storage.py   (revision 98)
+++ BitTornado/BT1/Storage.py   (working copy)
@@ -273,9 +273,10 @@
             begin, end, offset, file = self.ranges[p]
             if begin >= stop:
                 break
-            r.append(( file,
-                       offset + max(pos, begin) - begin,
-                       offset + min(end, stop) - begin   ))
+            if end > pos:
+                r.append(( file,
+                           offset + max(pos, begin) - begin,
+                           offset + min(end, stop) - begin   ))
             p += 1
         return r
 
@@ -370,7 +371,7 @@
             else:
                 midfile = os.path.join(self.bufferdir,str(f))
                 working_range = [(start, end, 0, midfile)]
-                disabled_files.append((midfile, start, end))
+                disabled_files.append((start, end, 0, midfile))
                 length = end - start
                 self.sizes[midfile] = length
                 piece = pieces[0]
@@ -381,7 +382,7 @@
                 end_b = pieces[1]*self.piece_length
                 startfile = os.path.join(self.bufferdir,str(f)+'b')
                 working_range_b = [ ( start, end_b, 0, startfile ) ]
-                disabled_files.append((startfile, start, end_b))
+                disabled_files.append((start, end_b, 0, startfile))
                 length = end_b - start
                 self.sizes[startfile] = length
                 offset = length
@@ -394,7 +395,7 @@
                 start_e = pieces[-1] * self.piece_length
                 endfile = os.path.join(self.bufferdir,str(f)+'e')
                 working_range_e = [ ( start_e, end, 0, endfile ) ]
-                disabled_files.append((endfile, start_e, end))
+                disabled_files.append((start_e, end, 0, endfile))
                 length = end - start_e
                 self.sizes[endfile] = length
                 piece = pieces.pop(-1)
@@ -445,7 +446,7 @@
         r = self._get_disabled_ranges(f)
         if not r:
             return
-        for file, begin, end in r[2]:
+        for begin, end, offset, file in r[2]:
             if not os.path.isdir(self.bufferdir):
                 os.makedirs(self.bufferdir)
             if not exists(file):
@@ -456,7 +457,7 @@
                 self.tops[file] = getsize(file)
             if not self.mtimes.has_key(file):
                 self.mtimes[file] = getmtime(file)
-        self.working_ranges[f] = r[0]
+        self.working_ranges[f] = r[2]
 
     reset_file_status = _reset_ranges
 
@@ -497,7 +498,7 @@
             if not self.files[i][1]:    # length == 0
                 continue
             if self.disabled[i]:
-                for file, start, end in self._get_disabled_ranges(i)[2]:
+                for start, end, offset, file in self._get_disabled_ranges(i)[2]:
                     pfiles.extend([basename(file),getsize(file),int(getmtime(file))])
                 continue
             file = self.files[i][0]
@@ -550,7 +551,7 @@
 
             for i in xrange(len(self.files)):
                 if self.disabled[i]:
-                    for file, start, end in self._get_disabled_ranges(i)[2]:
+                    for start, end, offset, file in self._get_disabled_ranges(i)[2]:
                         f1 = basename(file)
                         if ( not pfiles.has_key(f1)
                              or not test(pfiles[f1],getsize(file),getmtime(file)) ):
Index: BitTornado/BT1/FileSelector.py
===================================================================
--- BitTornado/BT1/FileSelector.py   (revision 98)
+++ BitTornado/BT1/FileSelector.py   (working copy)
@@ -104,6 +104,30 @@
         self.new_partials = None
         
 
+    def _initialize_files_disabled(self, old_priority, new_priority):
+        old_disabled = [p == -1 for p in old_priority]
+        new_disabled = [p == -1 for p in new_priority]
+        files_updated = False       
+        try:
+            for f in xrange(self.numfiles):
+                if new_disabled[f] and not old_disabled[f]:
+                    self.storage.disable_file(f)
+                    files_updated = True
+                if old_disabled[f] and not new_disabled[f]:
+                    self.storage.enable_file(f)
+                    files_updated = True
+        except (IOError, OSError), e:
+            if new_disabled[f]:
+                msg = "can't open partial file for "
+            else:
+                msg = 'unable to open '
+            self.failfunc(msg + self.files[f][0] + ': ' + str(e))
+            return False
+        if files_updated:
+            self.storage.reset_file_status()
+        return True       
+
+
     def _set_files_disabled(self, old_priority, new_priority):
         old_disabled = [p == -1 for p in old_priority]
         new_disabled = [p == -1 for p in new_priority]
@@ -189,6 +213,15 @@
         return new_piece_priority       
 
 
+    def initialize_priorities_now(self, new_priority = None):
+        if not new_priority:
+            return
+        old_priority = self.priority
+        self.priority = new_priority
+        if not self._initialize_files_disabled(old_priority, new_priority):
+            return
+#        self.piece_priority = self._set_piece_priority(new_priority)
+
     def set_priorities_now(self, new_priority = None):
         if not new_priority:
             new_priority = self.new_priority
Index: BitTornado/download_bt1.py
===================================================================
--- BitTornado/download_bt1.py   (revision 98)
+++ BitTornado/download_bt1.py   (working copy)
@@ -544,6 +544,9 @@
                 data = data.get('resume data')
                 if data:
                     self.fileselector.unpickle(data)
+
+            if self.priority:
+                self.fileselector.initialize_priorities_now(self.priority)
                 
         self.checking = True
         if old_style:


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 17, 2007 2:07 am 
Offline

Joined: Thu Mar 23, 2006 7:52 pm
Posts: 22
I edited the patch in the previous post to add one additional fix (the first block in the patch). It seems this was causing some problems, though I didn't see them until I ran a torrent with a file boundary on a piece boundary. I've updated it so the intervals it writes are guaranteed to only be in desired range.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Jul 20, 2007 2:36 pm 
Offline

Joined: Fri Jul 20, 2007 2:21 pm
Posts: 5
I don't suppose there is a very simple way for a non-programmer user to integrate that patch into the BitTornado client? One that doesn't involve learning how to use a compiler, or write code? ;)

I ask because I've had a whole bunch of repeated issues with the "download never" function of BitTornado used with multi-file torrents, and I would like to see if this would resolve the problems.

Currently using 0.3.17.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Jul 27, 2007 2:51 pm 
Offline

Joined: Thu Mar 23, 2006 7:52 pm
Posts: 22
Amberwolf wrote:
I don't suppose there is a very simple way for a non-programmer user to integrate that patch into the BitTornado client? One that doesn't involve learning how to use a compiler, or write code? ;)


You shouldn't need to compile anything or write any code. All you need to do is know how to use the patch program (available in Linux, check out gnuwin32 for a Windows version).

If you have trouble with that, I might be able to patch the 0.3.17 files for you, though I can't post them here so you'd have to PM me your email address.


Top
 Profile  
 
 Post subject: ''liar hater''
PostPosted: Wed Mar 10, 2010 4:01 am 
Offline

Joined: Wed Mar 10, 2010 3:53 am
Posts: 1
:) :) :) ,




Thanks for the information!
how to hypnotize people


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 7 posts ] 

All times are UTC - 7 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 8 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group