Internal server error when searching using _source and object field set to an array mixing non-empty and empty objects

[yrodiere]

Elasticsearch Version

8.6.0

Installed Plugins

none

Java Version

bundled

OS Version

Linux yrodiere.redhat 6.0.15-300.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Dec 21 18:33:23 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

On Elasticsearch 8.6.0, the _source feature in the _search endpoint seems unable to parse documents where an object field is set to an array mixing non-empty and empty objects (e.g. {"myobject":[{"mykeyword": "foo"}, {}]}; it fails with an internal server error.

This used to work without error in Elasticsearch 8.5.3.

Interestingly, an array containing only empty objects (e.g. {"myobject":[{"mykeyword": "foo"}, {}]}) works fine; you really need to mix empty and non-empty objects to trigger the failure.

Here is the error returned by Elasticsearch 8.6.0:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_state_exception",
        "reason" : "Failed to close the XContentBuilder"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "mytest",
        "node" : "Xa5Ij10zRWWtuQx8txxWZA",
        "reason" : {
          "type" : "illegal_state_exception",
          "reason" : "Failed to close the XContentBuilder",
          "caused_by" : {
            "type" : "i_o_exception",
            "reason" : "Unclosed object or array found"
          }
        }
      }
    ],
    "caused_by" : {
      "type" : "illegal_state_exception",
      "reason" : "Failed to close the XContentBuilder",
      "caused_by" : {
        "type" : "illegal_state_exception",
        "reason" : "Failed to close the XContentBuilder",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unclosed object or array found"
        }
      }
    }
  },
  "status" : 500
}

Steps to Reproduce

# Just to make it easier to re-run the test
curl -XDELETE localhost:9200/mytest/\?pretty 2>/dev/null 1>&2 || true       
# Create index with mapping
curl -XPUT -H "Content-Type: application/json" localhost:9200/mytest/\?pretty -d'{
  "mappings": {
    "properties": {
      "myobject": {
        "type": "object",
        "properties": {
          "mykeyword": {
            "type": "keyword",
            "index": true,
            "norms": false,
            "doc_values": false
          }
        }
      }
    }
  }
}
'
# Index something 
curl -XPUT -H "Content-Type: application/json" localhost:9200/mytest/_doc/1\?pretty\&refresh=true -d'{
  "myobject": [
    {
      "mykeyword": "somevalue"
    },
    {}
  ],
  "_entity_type": "indexNameType"
}
'
# Run a search with _source="myobject.mykeyword"
curl -XPOST -H "Content-Type: application/json" localhost:9200/mytest/_search\?pretty -d'{"_source": "myobject.mykeyword"}'
# There will be one hit with ES 8.5.3, but a failure in with ES 8.6.0

Logs (if relevant)

Output of the script above on Elasticsearch 8.6.0:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "mytest"
}
{
  "_index" : "mytest",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "forced_refresh" : true,
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_state_exception",
        "reason" : "Failed to close the XContentBuilder"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "mytest",
        "node" : "Xa5Ij10zRWWtuQx8txxWZA",
        "reason" : {
          "type" : "illegal_state_exception",
          "reason" : "Failed to close the XContentBuilder",
          "caused_by" : {
            "type" : "i_o_exception",
            "reason" : "Unclosed object or array found"
          }
        }
      }
    ],
    "caused_by" : {
      "type" : "illegal_state_exception",
      "reason" : "Failed to close the XContentBuilder",
      "caused_by" : {
        "type" : "illegal_state_exception",
        "reason" : "Failed to close the XContentBuilder",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unclosed object or array found"
        }
      }
    }
  },
  "status" : 500
}

Output of the script above on Elasticsearch 8.5.3:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "mytest"
}
{
  "_index" : "mytest",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "forced_refresh" : true,
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "mytest",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "myobject" : [
            {
              "mykeyword" : "somevalue"
            }
          ]
        }
      }
    ]
  }
}

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[DaveCTurner]

Looks like a duplicate of #92480, closing this.

[yrodiere]

@DaveCTurner Are you sure about that? While the symptoms are similar, my bug occurs only in 8.6.0, and not in 8.5.3, while #92480 was reported against 8.5.3...

[DaveCTurner]

It looks like a duplicate in the sense that it appears to be hitting the same Jackson bug, and that upgrading Jackson (see #92763) fixes it.

[pgomulka]

this will be fixed by #92984

[yrodiere]

That's great to hear, thanks a lot!