What do we do when the origin file has been changed mid request?

[ironsmile]

The situation is this: few parts of a file have been cached on disk already. A request is reading from them and it gets to a part which is not on the disk. Request is made to the upstream server and it turns out the file has been changed. Or maybe removed.

What do we do with the request? What do we return to the client? My proposal is - close the connection.

How do we remove the parts on the disk?

[mstoykov]

So the scenario is :

1. cache (module) is asked for a file.
2. we have the first 2 parts of the file and we return them from the storage module.
3. and Then the storage module through the upstream finds out that the file is newer than the version we have?

If this is the situation:

Currently I don't think the storage modules checks anything. Even the size is not checked.

Shouldn't we do HEAD request to the upstream to check if nothing has changed before starting to read from the Cache ?
And even then it could happen in the middle ...

[ironsmile]

Additional HEAD for every part seems like too much of a trouble. I have an idea: when getting a new part of a file we also receive its headers, right? So in it there might be the E-Tag header. We just compare the received E-Tag with the E-Tag from the already stored parts. This might work. Assuming the server sends E-Tag for the whole file when Bytes-Range request is made.

For requests with no E-Tag: tough, they will stay broken I guess?

[mstoykov]

Probably a connected case:
for a file with 100 pieces the following piece ranges have been cached: 10-20, 25-35 and 40-45.
A request for pieces 10-45 comes in but pieces 25-35 are actually too old.
Currently there is no check whether a whole file or a piece is too old to be serviced, but it should be implemented on per piece basis given this case.