Talk:Save Upload Queue Waiting Time

Is the description of SUQWT misleading?

The description of SUQWT gives the impression that it expects client x when exiting to notify other clients y1, y2, ..., yN, asking them to remember the position of x's requests in their queues. When x is launched again, it notifies y1, y2, ..., yN that it has returned online and asks them to restore x's position in their queues. Is this a correct description of SUQWT? I don't see how this would accomplish the stated goal of boosting distribution of rare items that are hard to get because their source frequently goes offline.

Wouldn't it be much more effective (and simpler) to expect a client to save its own upload queue when exiting, and restore it when relaunched? To disambiguate, let's call this Save and Restore Upload Queue (SRUQ). Suppose client x implements SRUQ, and clients y1, y2, ..., yN had requests in x's upload queue when x exited. Let Y denote clients y1, y2, ..., yN. When x comes back online, x restores Y's requests in x's queue. Suppose y2's request is for a rare item; in this case y2 probably has not managed to obtain it elsewhere and still wants it from x, and thanks to SRUQ x will eventually service y2's request, even if x frequently goes offline. Suppose y3's request is for a non-rare item; in this case, y3 may obtain it elsewhere while x is offline or while the request gradually advances in x's queue. If y3 obtains it elsewhere, x will eventually discard y3's request. (Note that the Y clients benefit even if they do not themselves implement SRUQ.)

Two days ago I read a post in the eMule forums written by an eMule developer, who said eMule's developers oppose SUQWT. He said they don't want to help clients who go offline. They prefer that users stay online as much as possible. If my understanding is correct, a downloader that is offline when his request reaches the front of another client's queue will lose his position, and eMule's developers want to keep it that way. SRUQ wouldn't change that; SRUQ wouldn't help clients who go offline, it would help clients that are requesting items from clients who go offline. (SRUQ might help the requesting clients enough that eMule's developers would stop caring so much whether clients stay online.) I hope the developers of eMule will be contacted anew to ask them to consider SRUQ.

Here's a nitpick: I doubt the queue really contains requests' waiting times. Surely it contains each request's arrival time instead. If it contained waiting times, they would need to be continuously updated to stay valid, which would be very inefficient, and unnecessary since waiting time can be calculated by subtracting arrival time from current time. Thus the name "Save Upload Queue Waiting Time" seems a bit inaccurate. eMule_user_since_2010 22:49, 8 July 2010 (CEST)

I performed a test on eMule 0.50a today, and learned that exiting eMule does NOT lose one's position in other clients' upload queues. I needed to restart my computer, so my eMule was offline for about 6 minutes. When I checked the queue positions of a couple of rare files I've been slowly downloading for weeks or months, I saw that my positions had not changed. So I see no need for SUQWT, unless the description of SUQWT is misleading (as I suspected) and SRUQ is what was really intended. eMule_user_since_2010 01:24, 10 July 2010 (CEST)

A variation of SRUQ that focuses on rare items

The goal is to boost the transmission of rare items so that intermittent availability doesn't make them effectively unavailable. SRUQ goes beyond what is necessary to achieve that goal. SRUQ will tend to boost mainly rare items, since downloaders of widely sourced items will obtain them even while some sources are offline, but a variation of SRUQ limited to rare items would accomplish the goal a bit more effectively. Here is one way it could work:

1) When the client is exiting, it saves a copy of its upload queue.
2) When the client is launched, if a saved queue exists, the client restores it and deletes the copy.
3) When a request arrives, it is considered redundant if the queue already contains a request from the same requester for the same file that has an arrival time later than the session starttime (which means the request already in the queue was not one of those restored). The client appends it to the queue only if it isn't redundant. 4) When a request reaches the front of the client's queue, the client does the following:
___If the request's arrival time is earlier than the session starttime,
______Calculate the rarity of the requested item. // See note below.
______If rare,
_________Initiate the upload.
_________If another request from the same requester for the same file is queued,
____________Delete that other request (since it is redundant).
______Else
_________Delete the request from the queue.
___Else
______ // The request isn't one of those restored; it arrived during session.
______Initiate the upload.

A note about calculating rarity: A file that has few complete sources may have many incomplete sources, thanks to clients currently downloading it. This means some parts of it may actually be widely sourced (especially when the credit system causes the downloaders to quickly share the parts they've obtained, while they all wait for hard-to-get rare parts). Ideally, boosting rare items should be about boosting rare parts of files, and various mules do use techniques to try to boost the distribution of the rare parts. (See "intelligent Part Sharing," "Hide Overshare," "Share Only The Need," etc.) There is no need to boost widely sourced parts, even if there are few (or zero) complete sources. The problem is, how can an uploading client efficiently calculate which parts are rare? (The descriptions of Intelligent Part Sharing, etc., imply it's difficult to determine which parts are rare.) Clients keep track of the number of complete sources for the files they source, but that provides only a rough approximation of each part's rarity. If there is a reasonably efficient way for a client to determine the rarity of a requested part, then it should be done.

Here's yet another variation of SRUQ: During the exiting process, when saving the queue, the client only saves requests for rare items. (This would speed up the handling of restored requests by greatly reducing the number of restored requests. On the other hand, it would be less accurate about rarity, since rarity may change while the client is offline or while the request is advancing through the queue.) eMule_user_since_2010 22:49, 8 July 2010 (CEST)

Another way to boost rare items

In my opinion, p2p software should do much more to boost the transmission of rare items, because increasing their availability helps the entire community, not just the current downloader. Here's one way to do this, which is unrelated to SUQWT or SRUQ: The client maintains a second upload queue; let's call it the "rare queue." When a request arrives, the client checks whether it's the only complete source for the file containing the requested part; if so, the client appends the request to the rare queue; else the client appends it to the normal queue. The upload bandwidth is shared by both queues in a manner such that a significant fraction of the bandwidth serves the rare queue (when it isn't empty).

The two queues could be stored in a single block of RAM; if they are, the maximum size of the normal queue would shrink when the rare queue grows and grow when the rare queue shrinks. It might be most efficient (performance-wise) to implement each queue as a linked list, if both are stored in a single block of RAM.

I think this feature would work well in concert with other features such as Intelligent Part Sharing, Hide Overshare, etc., but does not depend on them. eMule_user_since_2010 22:49, 8 July 2010 (CEST)