looking for more high level documentation

[rful011]

I have just spent 2 days hard slog (reading source, trial and lots of error ; ) to figuring out how to get Parallel and Listen to play nicely together to implement a program that picks up files from a spool directory and processes them, in parallel if needed.

I note that there is no mention of using a Proc as a source in the readme???

I have been doing this in a single thread for years but realised that the spool files were building up at busy times of the day.

To save anyone else who wants to have the ability to use a proc source I suggest adding this to the readme:

source may be a Proc that returns the next job. This example gets jobs from the Listen gem:

\`\`\`Ruby
job_queue = []
queue_access = Proc.new { job_queue.pop }

listener = Listen.to '.' do |modified, added, removed|
  if added
    added.each do |fn|
      job_queue << fn
    end
  end
end
listener.start # not blocking                                                                                                                                                                           
Parallel.map( queue_access, in_processes: 3 ) { |fn| fn ? process_file(iblogs, fn, dispose, extra) : sleep( interval )}
 \`\`\`

[grosser]

with threads it would be easy since you could use in_threads(3) and just pop from the queue

added an example here #344