S3 Object Naming May Be Excessively Nested

[ifadams]

Is your feature request related to a problem? Please describe.

When using the S3 connector, objects are created with the full path that one would use on a local file system for storage. While this provides for an easy "caching" method for objects, its messy in practice.

The intent of these large paths for local storage is to prevent individual directories from becoming overloaded and slowing down, however this may not make sense on object storage systems, and depending on the choice of backend may even be pathological.

Describe the solution you'd like

Consider a different naming/nesting scheme when using remote object storage for binary data.