Fix incosistent hashing #674
Conversation
| def _dos2unix(file_content: bytes) -> bytes: | ||
| """ | ||
| Replace occurrences of Windows line terminator CR/LF with only LF. | ||
|
|
||
| :param file_content: the content of the file. | ||
| :return: the same content but with the line terminator | ||
| """ | ||
| return re.sub(b"\r\n", b"\n", file_content, flags=re.M) |
There was a problem hiding this comment.
This is the logic which caused the inconsistent hash issues when locking and publishing packages, not sure why this was here in the first place but removing this logic fixes the issue and does not cause any new issue AFAIC so can be removed safely
| return file_b | ||
| data = file.read() | ||
| if len(data) == 0: | ||
| raise ValueError(f"File cannot be empty: {file_path}") |
There was a problem hiding this comment.
@angrybayblade why can not be empty? looks like empty file is a common case.
There was a problem hiding this comment.
The IPFS daemon treats empty files differently, if you push an empty file to the IPFS node it'll produce different hash compared to using the IPFSHashTool, now the reason behind this unclear I tried looking at the IPFS docs but could not find anything related. So for now we just raise an error when a user tries to hash an empty file so they don't run into these mismatching hashes issue when publishing the packages
There was a problem hiding this comment.
@angrybayblade ok, let stay as is.
but i see a possible source of problems, if empty file will be needed for some reason
aea/helpers/ipfs/base.py
Outdated
| except UnicodeDecodeError: | ||
| return False | ||
|
|
||
|
|
||
| def _read(file_path: str) -> bytes: | ||
| """Read a file, replacing Windows line endings if it is a text file.""" |
There was a problem hiding this comment.
@angrybayblade i dont see any fix for windows line endings anymore, file is opened in binary mode, so python will preserve all chars.
probably doc string is needed to be updated?
Proposed changes
This PR fixes the inconsistent hashing issues by
Fixes
Fixes #647
Types of changes
What types of changes does your code introduce to agents-aea?
Put an
xin the boxes that applyChecklist
Put an
xin the boxes that apply.developbranch (left side). Also you should start your branch off ourdevelop.