Monitor both cur and new folders?

[Ekleog]

Hello,

I've started using the mail exporter yesterday, and it's very convenient! I just have noticed a few issues with it, so I'll open related issues to get your opinion on them.

The one for this issue is the fact that mailexporter appears to only support monitoring one folder. This is problematic, because if I point it to the new folder sometime it misses the file, that goes straight to the cur folder; but if I point it to the cur folder then it sometimes gets the email only after the timeout expired.

So I'm thinking, would it make sense for the mailexporter to monitor both [configured-path]/cur and [configured-path]/new?

For now my workaround is all my probes end in the same folder and I have one probe configured with /new and the others with /cur, which given how the inotify code is written makes things work in practice; but that's relying on undocumented implementation details so it's not great.

Anyway, thank you for this piece of software!

[cherti]

The question is why sometimes that file goes straight to the cur-folder? I don't think this should happen, and before we start fixing a problem I would prefer to first understand how it comes into existence to then decide what the appropriate fix is.

So the question is if you can narrow down why that happens, somehow? It sounds like something is accessing that same folder or account, moving the mail files in the process.

[Ekleog]

There definitely is another client accessing the same folder concurrently: I'm monitoring a mail loop that involves an institutional email address of which I can have only one, so I'm using my personal mail account, which is often monitored by my personal mail client.

[cherti]

Can you create a subfolder that is not accessed by the mail client and sort things there and point mailexporter to that one?

If we monitor both of them concurrently, we could run into yet another race condition when the file is moved underneath mailexporter, so I'm wondering if this is actually the most resilient solution.

[Ekleog]

To be honest I probably can, but given it'd require specific configuration from all mail clients I ever use to access this account I'd rather avoid this solution, as I'm also using phone-based mail clients.

For the race condition, I guess you mean what happens if a file comes in new/, gets detected via inotify, but then is moved to cur/ before actually being read and/or deleted? First, if the move happens between the inotify and the open-for-read, there's no problem: the open-for-read will fail, but the file will be picked up again by the inotify for cur. The only issue may be if the file is moved beween the open-for-read and the delete, as the in-memory database will have been updated already and so the mail will not be handled.

In this case, I guess if the deletion of a file in new/ fails it should just be re-attempted in cur/. Does that make sense?

(Also, in practice I can say that just having inotify on both new/ and cur/ appears to be working ok enough, though I haven't actually checked that all mails do get deleted properly)