Much faster with logging disabled

[PoeticDeath]

When the logger is disabled the strings to be logged were still being constructed. The speed difference mainly noticeable with reads, are due to each read byte being logged. Should not adversely affect logging or consistency. On the left is the speed before, while right is the speed after the patch.

[touilleMan]

Hi @PoeticDeath, and thanks for this PR ;-)

I can see there is two different things there:

1. remove logging overhead when it is not needed
2. improve handling of buffers used in get_reparse_point

Can you explain the performance gain comes from 1), 2) or both

Also what is the need for 2) (it seems like get_reparse_point only raises NotImplementedError right now

Finally while better performances is always good, from what I understand here your PR only improve the performances of the memfs demo

[touilleMan]

Hi @PoeticDeath, and thanks for this PR ;-)

I can see there is two different things there:

1. remove logging overhead when it is not needed
2. improve handling of buffers used in get_reparse_point

Can you explain the performance gain comes from 1), 2) or both

Also what is the need for 2) (it seems like get_reparse_point only raises NotImplementedError right now

Finally while better performances is always good, from what I understand here your PR only improve the performances of the memfs demo

[touilleMan]

Hi @PoeticDeath, and thanks for this PR ;-)

I can see there is two different things there:

1. remove logging overhead when it is not needed
2. improve handling of buffers used in get_reparse_point

Can you explain the performance gain comes from 1), 2) or both

Also what is the need for 2) (it seems like get_reparse_point only raises NotImplementedError right now

Finally while better performances is always good, from what I understand here your PR only improve the performances of the memfs demo

[touilleMan]

Hi @PoeticDeath, and thanks for this PR ;-)

I can see there is two different things there:

1. remove logging overhead when it is not needed
2. improve handling of buffers used in get_reparse_point

Can you explain the performance gain comes from 1), 2) or both

Also what is the need for 2) (it seems like get_reparse_point only raises NotImplementedError right now

Finally while better performances is always good, from what I understand here your PR only improve the performances of the memfs demo

[touilleMan]

Hi @PoeticDeath, and thanks for this PR ;-)

I can see there is two different things there:

1. remove logging overhead when it is not needed
2. improve handling of buffers used in get_reparse_point

Can you explain the performance gain comes from 1), 2) or both

Also what is the need for 2) (it seems like get_reparse_point only raises NotImplementedError right now

Finally while better performances is always good, from what I understand here your PR only improve the performances of the memfs demo

[touilleMan]

Hi @PoeticDeath, and thanks for this PR ;-)

I can see there is two different things there:

1. remove logging overhead when it is not needed
2. improve handling of buffers used in get_reparse_point

Can you explain the performance gain comes from 1), 2) or both

Also what is the need for 2) (it seems like get_reparse_point only raises NotImplementedError right now

Finally while better performances is always good, from what I understand here your PR only improve the performances of the memfs demo

[touilleMan]

Hi @PoeticDeath, and thanks for this PR ;-)

I can see there is two different things there:

1. remove logging overhead when it is not needed
2. improve handling of buffers used in get_reparse_point

Can you explain the performance gain comes from 1), 2) or both

Also what is the need for 2) (it seems like get_reparse_point only raises NotImplementedError right now

Finally while better performances is always good, from what I understand here your PR only improve the performances of the memfs demo

[touilleMan]

Hi @PoeticDeath, and thanks for this PR ;-)

I can see there is two different things there:

1. remove logging overhead when it is not needed
2. improve handling of buffers used in get_reparse_point

Can you explain the performance gain comes from 1), 2) or both

Also what is the need for 2) (it seems like get_reparse_point only raises NotImplementedError right now

Finally while better performances is always good, from what I understand here your PR only improve the performances of the memfs demo

[touilleMan]

Hi @PoeticDeath, and thanks for this PR ;-)

I can see there is two different things there:

1. remove logging overhead when it is not needed
2. improve handling of buffers used in get_reparse_point

Can you explain the performance gain comes from 1), 2) or both

Also what is the need for 2) (it seems like get_reparse_point only raises NotImplementedError right now

Finally while better performances is always good, from what I understand here your PR only improve the performances of the memfs demo

[touilleMan]

Hi @PoeticDeath, and thanks for this PR ;-)

I can see there is two different things there:

1. remove logging overhead when it is not needed
2. improve handling of buffers used in get_reparse_point

Can you explain the performance gain comes from 1), 2) or both

Also what is the need for 2) (it seems like get_reparse_point only raises NotImplementedError right now

Finally while better performances is always good, from what I understand here your PR only improve the performances of the memfs demo

[PoeticDeath]

I've been developing my own filesystem currently open source, named SpaceFS. Refuse, another module for use of WinFsp in Python, is great for compatibility between major OSs. But I wanted Windows specific functions to be supported where possible. Thus I looked for another solution and found your winfspy.

However almost immediately had found some performance issues. Using kernprof, a line by line profiler. I narrowed down a good chunk to the creation of debug strings even when it is disabled. To avoid this I added a simple if check of logging.root.level seen with in the first commit. This improvement will help any code or filesystem that uses your logging function.

Since then I have focused on properly implementing features, I managed to get pretty much everything to work correctly testable through WinFsp's winfsp-test. However when I got to reparse handing it was not implemented nearly at all, neither was GetStreamInfo. So I have worked on properly implementing them as well, GetStreamInfo needs some work still, many filenames will be reported correctly while a few others will not, but reparse seems to be complete. This should have little or no affect to performance, just completeness.

Sorry it took so long to respond for whatever reason GitHub did not notify me to your comment. Looking forward to working with you.