Speed up Get_RemoteCheckSum #58
Comments
The current Get_RemoteChecksum implementation uses Invoke-WebRequest which can be slow on larger files. This results in a long checksumming process during automated packaging. By switching to the .NET webclient, while losing the "interactive" download update, we significantly increase the speed of the download. For example, on a 150 mb file, it went from minutes to download to seconds.
The current Get_RemoteChecksum implementation uses Invoke-WebRequest which can be slow on larger files. This results in a long checksumming process during automated packaging. By switching to the .NET webclient, while losing the "interactive" download update, we significantly increase the speed of the download. For example, on a 150 mb file, it went from minutes to download to seconds.
The current Get_RemoteChecksum implementation uses Invoke-WebRequest which can be slow on larger files. This results in a long checksumming process during automated packaging. By switching to the .NET webclient, while losing the "interactive" download update, we significantly increase the speed of the download. For example, on a 150 mb file, it went from minutes to download to seconds. Added modified date in this commit
I don't think this would be a problem, but I am unsure if there would be any behavior changes between these two options that would cause downloads to fail using the webclient (thus making this a breaking change). If I understand correctly, Another option may be to disable the progress bar via |
|
In looking at results from google, I can see older issues with very large files (gb+) on spotty internet connections. I have not seen recent reports of issues. I will try without the progress bar to see if there is a speed improvement. Another option would be to add a flag which uses .NET client if true else uses Invoke-WebRequest? Would allow for testing across some of the automated updates with just a flag change to see at scale if issues within the appveyor builds? The packages I have tested against (not many) have not caused issues with local runs. |
The current Get_RemoteChecksum implementation uses Invoke-WebRequest with progress bar which can be slow on larger files. This results in a long checksumming process during automated packaging. By providing the option to hide the progress bar, while losing the "interactive" download update, we significantly increase the speed of the download.
|
Looks like your hunch was on the money: This was for $Url=https://c.1password.com/dist/1P/win8/1PasswordSetup-8.10.32.msi It is an order of magnitude faster without the progress bar. So similar speed with the WebClient and the Invoke-WebRequest... I would suggest we update to hide the progress bar. New update pushed |
Checklist
Is Your Feature Request Related To A Problem? Please describe.
When running Get-RemoteChecksum in the au_beforeupdate() function on a large file, the process takes a long time. In benchmarks undertaken by external pers, it is suggested that using NET's webclient vs Invoke-WebRequest is significantly faster for downloading. For example, for a 150mb file it takes minutes with Invoke_WebRequest vs seconds with the webclient.
Describe The Solution. Why is it needed?
I would suggest we replace the Invoke-WebRequest call with the .NET webclient call to significantly speed up downloads. This should speed up the checksumming process as part of au_beforeupdate(). This could also be added to other locations which download files if required.
Additional Context
No response
Related Issues
No response
The text was updated successfully, but these errors were encountered: