Configuration + S3FileSystem + S3FileSystemProvider refactoring

[carlspring]

Task Description

The current implementation of the configuration is not flexible enough. It also requires you to pass properties around. The way that the properties are loaded for testing and production is different. We should implement a S3FileSystemConfiguration and use this instead of the properties/hash maps.

When creating a new S3Path, at the moment, you always have to specify the bucket like this:

S3Path path = s3FileSystem.getPath("/bucketA", "dir");

This is very inconvenient. The S3FileSystemProvider needs to be able to resolve this via the S3FileSystemConfiguration. Google's NIO2 FileSystem implementation does not require you to specify the bucket and we should follow this approach as well.

We need to come up with better solutions for the current implementations.

This is the master task for the work.

Tasks

The following tasks will need to be carried out:

• Implement an S3FileSystemConfiguration class that contains all the details for the bucket, connection, etc. ( Implement an S3FileSystemConfiguration #195 )
• Implement an S3URIBuilder class.
• Implement a S3URI class. ( Implement an S3 URI #213 )
• Drop the heavy use of Properties all over the place and replace it with the S3FileSystemConfiguration. ( Drop the heavy use of Properties all over the place and replace it with the S3FileSystemConfiguration #196 )
• Remove all the properties validating methods from the S3FileSystemProvider. The provider must be a provider and not be overpopulated with so much validation.
• Re-work the S3FileSystem to not require you to specify the bucket and resolve this from the S3FileSystemConfiguration instead. ( Re-work the S3FileSystem to not require you to specify the bucket and resolve this from the S3FileSystemConfiguration instead #197 )
• Drop the concept of the EnvironmentBuilder.
• Update the documentation.

Task Relationships

This task:

• Is a sub-task of: #
• Depends on: #
• Is a follow-up of: #
• Relates to: Issue-60: Environment based configuration + refactoring #192

Help

• Our chat channel
• Points of contact:
  • @carlspring
  • @ptirador
  • @steve-todorov

[carlspring]

Some similar things have been done in #192, however, this is only addressing (in a similar way) the problem from a test-related point-of-view. Since we'll be doing such a major refactoring, it doesn't make sense to do it in one way for the tests and in another way for the production code.

The good news, however, @mslowiak , is that we appear to be on the same page as to how this should be implemented. Please, bear with me until I've worked out most of the details on how we should approach this. I have pushed my incomplete (and therefore not fully working changes) to the massive-provider-refactoring branch. This branch will only serve as a reference of what I think we should do. We will cherry pick some of the changes from there and try to finish the work that I began (but lacked the time to properly finish on my own). It has not yet been rebased.

[mslowiak]

Yeah, it looks like we have a similar point of view. I would be really happy to help with that. Catch me on chat if the code will be ready for some refactoring stuff @carlspring.