-
Notifications
You must be signed in to change notification settings - Fork 145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Colander and None during deserialization? #276
Comments
I ran into the same issue. Any explanation would be great! |
There was already a solution to that #258 |
It wasn't merged though. Does that mean I have to search-and-replaces None values with colander.null currently? |
That's what we ended up doing. |
Ah, that's what I was afraid of. Not hard to do but I'm dreading having to remember to come back to this 2 years from now.... |
in #258 a contributor had to sign and did not do it ... |
I understand, not trying to apportion any blame. Stuff happens. |
I would be willing to contribute a PR. But before starting, which approach would the maintainers recommend to solve the None deserialization issue? Same as #258 or a recursive replace of None to colander.null before deserializing? In #258 @mmerickel mentioned the |
I'm having this exact same problem, I have a SequenceSchema that is optional, and I use the missing parameter. If I don't send the field in the json, it works, but if I send null (None), I get the '"None" is not iterable' message. |
@mmerickel is redoing the work in #258 the preferred way forward to resolve this issue for non-strings? Per your comment, strings are already handled via If that's all it takes, then I am happy to do the copy-paste from the code in the PR #258 to get this into Colander. |
#258 wouldn't work here and looking at its impl again I wouldn't accept it as-is. It's converting none explicitly to an empty string and in the case of a SequenceSchema you'd want it to deserialize to something else. WRT some other comments in this thread, I can't actually explain the scenarios in which @RangelReale None and "missing" are different concepts entirely in schema design, even if you would prefer them to be treated the same in your case. If you want to handle None, I would suggest using a preparer to delete the value entirely. At which point the "missing" semantics will kick in and you can specify a default value there such as |
Thanks for the quick reply @mmerickel. Also after reviewing your earlier comment about the proposed implementation treating For this issue, I would suggest that developers carefully review the documentation of The Null and Drop Values. This is densely packed with easy to misunderstand details. There are also two tables which illustrate Serialization Combinations and Deserialization Combinations. I think this issue can be closed, as it seems to be a matter of not understanding how Colander handles |
I'm also getting this issue with nested schema and handling 'null' values.
deserialising the json:
gives the error:
based on the documentation:
I would expect |
@RichardJones1337 you need to replace "null" with from pprint import pprint
import colander
class SchemaA(colander.MappingSchema):
val = colander.SchemaNode(colander.String())
class SchemaB(colander.MappingSchema):
a = SchemaA(missing=None)
otherval = colander.SchemaNode(colander.String(), missing=None)
schema = SchemaB()
deserialized = schema.deserialize({"a": colander.null, "otherval": "asdf"})
pprint(deserialized) => Like I said, those docs are densely packed with easy to overlook details. |
This presents inconsistent behavior with other SchemaNode instances.
would result in a (correct) outcome of {'a': {'val': None}, 'otherval': None}, so why are nested schema treated differently? Also, for the conveinience of everyone here, please could you provide a reference to the location in the documents where it states that passing None as the value to a nested MappingSchema needs to be handled differently? |
Using your example in my code above, I get:
I think you are confusing a JSON object with a Python dict. Although they might look the same with their curly braces, they are very different things. For documentation references, please see my earlier comment. |
I am fully aware of the differences between JSON and a python dict, ok let's skip the bit where I am using colander to deserialise json requests, and just jump straight to the point where I'm deserialising the dict I've extracted from the json request,
produces
(I have trimmed some lines for brevity) |
In this new example, you inserted a
=> {'a': {'val': None}, 'otherval': 'asdf'} Colander is doing the right thing here by enforcing a mapping type. |
Thanks for the explanation, I'm still not sure why something like |
It could, but IMO it would be harmful. For the sake of discussion, let's consider its impact. Let's say I deserialize an arbitrary non-mapping object to |
So, let's say there's a business model with tickets, and an assignee. An assignee is an id to a person + a role like 'tester' or 'developer'. How do you write a schema to validate both jsons like this {
"ticket-name": "",
"assignee": null
} {
"ticket-name": "",
"assignee": {
"person_id": 1,
"role": "tester"
}
} That json (i know realistically "assignee" would probably be a list and a sequenceSchema, but just for the sake of this example. Is this not possible at all?) import colander
class Assignee(colander.Schema):
person_id = colander.SchemaNode(colander.Integer)
role = colander.SchemaNode(colander.String)
class Ticket(colander.Schema):
name = colander.SchemaNode(colander.String)
assignee = Assignee(missing=None)
Ticket().deserialize({'name': 'x', 'assignee': None}) |
In that pseudo-code, you are telling Colander to assign For your use case, I think preprocessing the first JSON object, expanding it to include its nodes from the Colander schema and setting |
I am confused about the way Colander handles None values (or not handles None values) during the deserialization. For example:
In this case Colander accepts a
None
value, and keeps it becausemissing
is set toNone
. This sounds reasonable to me.Another example using a sequence:
In this case Colander does not accept None.
Yet another example using a nested schema:
Again None is not accepted. I have some more examples with nested sequences and tuples here: https://gist.github.com/tsauerwein/93c88ba3c37f936dea113a27447c0db5
What is the idea? Why does Colander accept None in some cases during the deserialization, and in some cases not? Do I have to replace all None values with
colander.null
before callingdeserialize
?Looking the tests it seems that this is even the expected behavior:
https://github.com/Pylons/colander/blob/cb1645d/colander/tests/test_colander.py#L935
https://github.com/Pylons/colander/blob/cb1645d/colander/tests/test_colander.py#L690
The text was updated successfully, but these errors were encountered: