You download from all the other users who are sharing the file. If there are only a few people sharing, you won't be able to download that fast. However, if it's a pretty popular file, you should be able to download as fast as your connection will allow. I've seen 400KB/s on some BT downloads.
http://bt.degreez.net/firewalled.html explains with diagrams how being firewalled (not opening/forwarding ports) limits who you can connect to, and thus limits how fast you can download.
http://userpages.umbc.edu/~hamilton/btclientconfig.html is another good page, with lots of info from the basic networking stuff involved all the way to actually configuring your stuff for BT.
There are two things that really help to configure. First, you should forward the proper ports in your router (UPnP can do this automatically) or open those ports in your firewall. That will allow you to accept incoming connections, which will give you a lot more sources to download from. Second, you should make sure you set your upload rate properly. BT is designed to work better for those who upload more, to discourage leeches. However, if you set it too high, it'll actually choke off the downloads. If you know the upload rate of your internet connection, start a little below that. If not, just keep raising it a little at a time until it starts interfering with your download speed (the BT download itself, web pages, FTP downloads - anything can be used for testing).