apacheGH-36961: [MATLAB] Add arrow.tabular.Schema class and associa…

…ted `arrow.schema` construction function (apache#37013)

### Rationale for this change

To continue building out the tabular APIs for the MATLAB interface, this PR adds a new `arrow.tabular.Schema` class which wraps one or more `arrow.type.Field` objects and semantically describes the names and types of the columns of a tabular Arrow data structure.

To construct an `arrow.tabular.Schema` object, client code can call an associated `arrow.schema` construction function (similar to the `arrow.field` construction function).

This mirrors the tabular APIs in other Arrow bindings, like `pyarrow`.

### What changes are included in this PR?

1. New `arrow.tabular.Schema` class.
2. New `arrow.schema(fields)` construction function for creating instances of `arrow.tabular.Schema`.

**Example**:

```matlab
>> fieldA = arrow.field("A", arrow.uint8);
>> fieldB = arrow.field("B", arrow.string);
>> fieldC = arrow.field("C", arrow.timestamp);

>> fields = [fieldA, fieldB, fieldC];

>> schema = arrow.schema(fields)

schema = 

A: uint8
B: string
C: timestamp[us]

>> schema.NumFields

ans =

  int32

   3

>> schema.FieldNames

ans = 

  1×3 string array

    "A"    "B"    "C"

>> f = schema.field(3)

f = 

C: timestamp[us]

>> f = schema.field("B")

f = 

B: string
```

### Are these changes tested?

Yes.

1. Added a new test class `tSchema.m` which contains tests for `arrow.schema` and `arrow.tabular.Schema`. 

### Are there any user-facing changes?

Yes.

1. New public `arrow.tabular.Schema` class.
    1.1 **Properties**
                1.1.1 `NumFields`
        1.1.2 `FieldNames`
        1.1.3 `Fields`
    1.2 **Methods**
        1.2.1 `field(index)` where index is a valid numeric index or field name.
2. New public `arrow.schema(fields)` construction function.

### Future Directions

1. @ sgilmore10 introduced some new input validation functions that are generic and reusable in apache#36978. To avoid using multiple different approaches to input validation across the MATLAB code base, it would be a good idea to re-implement the input validation for `Schema` methods (e.g. `field`) to use these validation functions consistently.
4. Error handling in some edge cases is less than ideal right now for `Schema`. We should consider doing a more thorough review of error handling and error messages across the MATLAB code base now that we have more APIs and have seen several similar error states appear in different parts of the code base (e.g. indexing errors).
5. We may want to consider alternative construction syntaxes beyond just `arrow.schema(fields)`. For example, `arrow.schema(fieldName_1, fieldType_1, ..., fieldName_i, fieldType_i, ... fieldName_n, fieldType_n)` might be another convenient syntax that we could consider supporting.
6. We should add a `Schema` property to `RecordBatch`.
7. Consider adding a `toMATLAB` method for `Schema` which returns an empty MATLAB `table` with corresponding variable names and MATLAB types.
* Closes: apache#36961

Lead-authored-by: Kevin Gurney <[email protected]>
Co-authored-by: Kevin Gurney <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Kevin Gurney <[email protected]>

matlab/src/cpp/arrow/matlab/error/error.h

@@ -173,4 +173,8 @@ namespace arrow::matlab::error {
     static const char* UKNOWN_TIME_UNIT_ERROR_ID = "arrow:matlab:UnknownTimeUnit";
     static const char* FIELD_FAILED_TO_CREATE_TYPE_PROXY = "arrow:field:FailedToCreateTypeProxy";
     static const char* ARRAY_FAILED_TO_CREATE_TYPE_PROXY = "arrow:array:FailedToCreateTypeProxy";
+    static const char* ARROW_TABULAR_SCHEMA_INVALID_NUMERIC_FIELD_INDEX = "arrow:tabular:schema:InvalidNumericFieldIndex";
+    static const char* ARROW_TABULAR_SCHEMA_UNKNOWN_FIELD_NAME = "arrow:tabular:schema:UnknownFieldName";
+    static const char* ARROW_TABULAR_SCHEMA_AMBIGUOUS_FIELD_NAME = "arrow:tabular:schema:AmbiguousFieldName";
+    static const char* ARROW_TABULAR_SCHEMA_NUMERIC_FIELD_INDEX_WITH_EMPTY_SCHEMA = "arrow:tabular:schema:NumericFieldIndexWithEmptySchema";
 }

matlab/src/cpp/arrow/matlab/proxy/factory.cc

@@ -19,6 +19,7 @@
 #include "arrow/matlab/array/proxy/numeric_array.h"
 #include "arrow/matlab/array/proxy/string_array.h"
 #include "arrow/matlab/tabular/proxy/record_batch.h"
+#include "arrow/matlab/tabular/proxy/schema.h"
 #include "arrow/matlab/error/error.h"
 #include "arrow/matlab/type/proxy/primitive_ctype.h"
 #include "arrow/matlab/type/proxy/string_type.h"
@@ -44,6 +45,7 @@ libmexclass::proxy::MakeResult Factory::make_proxy(const ClassName& class_name,
     REGISTER_PROXY(arrow.array.proxy.StringArray   , arrow::matlab::array::proxy::StringArray);
     REGISTER_PROXY(arrow.array.proxy.TimestampArray, arrow::matlab::array::proxy::NumericArray<arrow::TimestampType>);
     REGISTER_PROXY(arrow.tabular.proxy.RecordBatch , arrow::matlab::tabular::proxy::RecordBatch);
+    REGISTER_PROXY(arrow.tabular.proxy.Schema      , arrow::matlab::tabular::proxy::Schema);
     REGISTER_PROXY(arrow.type.proxy.Field          , arrow::matlab::type::proxy::Field);
     REGISTER_PROXY(arrow.type.proxy.Float32Type    , arrow::matlab::type::proxy::PrimitiveCType<float>);
     REGISTER_PROXY(arrow.type.proxy.Float64Type    , arrow::matlab::type::proxy::PrimitiveCType<double>);

matlab/src/cpp/arrow/matlab/tabular/proxy/schema.cc

@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/matlab/error/error.h"
+#include "arrow/matlab/tabular/proxy/schema.h"
+#include "arrow/matlab/type/proxy/field.h"
+
+#include "libmexclass/proxy/ProxyManager.h"
+#include "libmexclass/error/Error.h"
+
+#include "arrow/util/utf8.h"
+
+#include <sstream>
+
+namespace arrow::matlab::tabular::proxy {
+
+    namespace {
+
+        libmexclass::error::Error makeUnknownFieldNameError(const std::string& name) {
+            using namespace libmexclass::error;
+            std::stringstream error_message_stream;
+            error_message_stream << "Unknown field name: '";
+            error_message_stream << name;
+            error_message_stream << "'.";
+            return Error{error::ARROW_TABULAR_SCHEMA_UNKNOWN_FIELD_NAME, error_message_stream.str()};
+        }
+
+        libmexclass::error::Error makeEmptySchemaError() {
+            using namespace libmexclass::error;
+            return Error{error::ARROW_TABULAR_SCHEMA_NUMERIC_FIELD_INDEX_WITH_EMPTY_SCHEMA,
+                         "Numeric indexing using the field method is not supported for schemas with no fields."};
+        }
+
+    }
+
+    Schema::Schema(std::shared_ptr<arrow::Schema> schema) : schema{std::move(schema)} {
+        REGISTER_METHOD(Schema, getFieldByIndex);
+        REGISTER_METHOD(Schema, getFieldByName);
+        REGISTER_METHOD(Schema, getNumFields);
+        REGISTER_METHOD(Schema, getFieldNames);
+        REGISTER_METHOD(Schema, toString);
+    }
+
+    libmexclass::proxy::MakeResult Schema::make(const libmexclass::proxy::FunctionArguments& constructor_arguments) {
+        namespace mda = ::matlab::data;
+        using SchemaProxy = arrow::matlab::tabular::proxy::Schema;
+
+        mda::StructArray args = constructor_arguments[0];
+        const mda::TypedArray<uint64_t> field_proxy_ids_mda = args[0]["FieldProxyIDs"];
+
+        std::vector<std::shared_ptr<arrow::Field>> fields;
+        for (const auto proxy_id : field_proxy_ids_mda) {
+            using namespace libmexclass::proxy;
+            auto proxy = std::static_pointer_cast<arrow::matlab::type::proxy::Field>(ProxyManager::getProxy(proxy_id));
+            auto field = proxy->unwrap();
+            fields.push_back(field);
+        }
+        auto schema = arrow::schema(fields);
+        return std::make_shared<SchemaProxy>(std::move(schema));
+    }
+
+    std::shared_ptr<arrow::Schema> Schema::unwrap() {
+        return schema;
+    }
+
+    void Schema::getFieldByIndex(libmexclass::proxy::method::Context& context) {
+        namespace mda = ::matlab::data;
+        using namespace libmexclass::proxy;
+        using FieldProxy = arrow::matlab::type::proxy::Field;
+        mda::ArrayFactory factory;
+
+        mda::StructArray args = context.inputs[0];
+        const mda::TypedArray<int32_t> index_mda = args[0]["Index"];
+        const auto matlab_index = int32_t(index_mda[0]);
+        // Note: MATLAB uses 1-based indexing, so subtract 1.
+        // arrow::Schema::field does not do any bounds checking.
+        const int32_t index = matlab_index - 1;
+        const auto num_fields = schema->num_fields();
+
+        if (num_fields == 0) {
+            const auto& error = makeEmptySchemaError();
+            context.error = error;
+            return;
+        }
+
+        if (matlab_index < 1 || matlab_index > num_fields) {
+            using namespace libmexclass::error;
+            const std::string& error_message_id = std::string{error::ARROW_TABULAR_SCHEMA_INVALID_NUMERIC_FIELD_INDEX};
+            std::stringstream error_message_stream;
+            error_message_stream << "Invalid field index: ";
+            error_message_stream << matlab_index;
+            error_message_stream << ". Field index must be between 1 and the number of fields (";
+            error_message_stream << num_fields;
+            error_message_stream << ").";
+            const std::string& error_message = error_message_stream.str();
+            context.error = Error{error_message_id, error_message}; 
+            return;
+        }
+
+        const auto& field = schema->field(index);
+        auto field_proxy = std::make_shared<FieldProxy>(field);
+        const auto field_proxy_id = ProxyManager::manageProxy(field_proxy);
+        const auto field_proxy_id_mda = factory.createScalar(field_proxy_id);
+
+        context.outputs[0] = field_proxy_id_mda;
+    }
+
+    void Schema::getFieldByName(libmexclass::proxy::method::Context& context) {
+        namespace mda = ::matlab::data;
+        using namespace libmexclass::proxy;
+        using FieldProxy = arrow::matlab::type::proxy::Field;
+        mda::ArrayFactory factory;
+
+        mda::StructArray args = context.inputs[0];
+        const mda::StringArray name_mda = args[0]["Name"];
+        const auto name_utf16 = std::u16string(name_mda[0]);
+        MATLAB_ASSIGN_OR_ERROR_WITH_CONTEXT(const auto name, arrow::util::UTF16StringToUTF8(name_utf16), context, error::UNICODE_CONVERSION_ERROR_ID);
+        const std::vector<std::string> names = {name};
+        MATLAB_ERROR_IF_NOT_OK_WITH_CONTEXT(schema->CanReferenceFieldsByNames(names), context, error::ARROW_TABULAR_SCHEMA_AMBIGUOUS_FIELD_NAME);
+
+        const auto field = schema->GetFieldByName(name);
+        auto field_proxy = std::make_shared<FieldProxy>(field);
+        const auto field_proxy_id = ProxyManager::manageProxy(field_proxy);
+        const auto field_proxy_id_mda = factory.createScalar(field_proxy_id);
+
+        context.outputs[0] = field_proxy_id_mda;
+    }
+
+    void Schema::getNumFields(libmexclass::proxy::method::Context& context) {
+        namespace mda = ::matlab::data;
+        mda::ArrayFactory factory;
+
+        const auto num_fields = schema->num_fields();
+        const auto num_fields_mda = factory.createScalar(num_fields);
+
+        context.outputs[0] = num_fields_mda;
+    }
+
+    void Schema::getFieldNames(libmexclass::proxy::method::Context& context) {
+        namespace mda = ::matlab::data;
+        mda::ArrayFactory factory;
+
+        const auto field_names_utf8 = schema->field_names();
+        const auto num_fields = static_cast<size_t>(schema->num_fields());
+
+        std::vector<std::u16string> field_names_utf16;
+        field_names_utf16.reserve(num_fields);
+
+        // Conver the field names from UTF-8 to UTF-16.
+        for (const auto& field_name_utf8 : field_names_utf8) {
+            MATLAB_ASSIGN_OR_ERROR_WITH_CONTEXT(const auto field_name_utf16, arrow::util::UTF8StringToUTF16(field_name_utf8), context, error::UNICODE_CONVERSION_ERROR_ID);
+            field_names_utf16.push_back(field_name_utf16);
+        }
+
+        const auto field_names_mda = factory.createArray({1, num_fields}, field_names_utf16.cbegin(), field_names_utf16.cend());
+
+        context.outputs[0] = field_names_mda;
+    }
+
+    void Schema::toString(libmexclass::proxy::method::Context& context) {
+        namespace mda = ::matlab::data;
+        mda::ArrayFactory factory;
+
+        const auto str_utf8 = schema->ToString();
+        MATLAB_ASSIGN_OR_ERROR_WITH_CONTEXT(const auto str_utf16, arrow::util::UTF8StringToUTF16(str_utf8), context, error::UNICODE_CONVERSION_ERROR_ID);
+        auto str_mda = factory.createScalar(str_utf16);
+        context.outputs[0] = str_mda;
+    }
+
+}

matlab/src/cpp/arrow/matlab/tabular/proxy/schema.h

@@ -0,0 +1,47 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+// arrow::Schema is defined in type.h.
+#include "arrow/type.h"
+
+#include "libmexclass/proxy/Proxy.h"
+
+namespace arrow::matlab::tabular::proxy {
+
+    class Schema : public libmexclass::proxy::Proxy {
+        public:
+            Schema(std::shared_ptr<arrow::Schema> Schema);
+
+            virtual ~Schema() {}
+
+            static libmexclass::proxy::MakeResult make(const libmexclass::proxy::FunctionArguments& constructor_arguments);
+
+            std::shared_ptr<arrow::Schema> unwrap();
+
+        protected:
+            void getFieldByIndex(libmexclass::proxy::method::Context& context);
+            void getFieldByName(libmexclass::proxy::method::Context& context);
+            void getNumFields(libmexclass::proxy::method::Context& context);
+            void getFieldNames(libmexclass::proxy::method::Context& context);
+            void toString(libmexclass::proxy::method::Context& context);
+
+            std::shared_ptr<arrow::Schema> schema;
+    };
+
+}

matlab/src/matlab/+arrow/+tabular/Schema.m

@@ -0,0 +1,96 @@
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements.  See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License.  You may obtain a copy of the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied.  See the License for the specific language governing
+% permissions and limitations under the License.
+
+classdef Schema < matlab.mixin.CustomDisplay
+%SCHEMA A tabular schema which semantically describes
+% the names and types of the columns of an associated tabular
+% Arrow data type.
+
+    properties (GetAccess=public, SetAccess=private, Hidden)
+        Proxy
+    end
+
+    properties (Dependent, SetAccess=private, GetAccess=public)
+        % Underlying array of Fields that the Schema wraps.
+        Fields
+        % Names of the columns in the associated tabular type.
+        FieldNames
+        % Number of fields in the schema
+        NumFields
+    end
+
+    methods
+
+        function obj = Schema(proxy)
+            arguments
+                proxy(1, 1) libmexclass.proxy.Proxy {validate(proxy, "arrow.tabular.proxy.Schema")}
+            end
+            import arrow.internal.proxy.validate
+
+            obj.Proxy = proxy;
+        end
+
+        function F = field(obj, idx)
+            idx = convertCharsToStrings(idx);
+            if ~isempty(idx) && isscalar(idx) && isnumeric(idx) && idx >= 1
+                args = struct(Index=int32(idx));
+                proxyID = obj.Proxy.getFieldByIndex(args);
+            elseif isscalar(idx) && isstring(idx)
+                name = idx;
+                args = struct(Name=name);
+                proxyID = obj.Proxy.getFieldByName(args);
+            else
+                error("arrow:tabular:schema:UnsupportedFieldIndexType", ...
+                      "Index must be a positive scalar integer or a valid field name.");
+            end
+
+            proxy = libmexclass.proxy.Proxy(Name="arrow.type.proxy.Field", ID=proxyID);
+            F = arrow.type.Field(proxy);
+        end
+
+        function fields = get.Fields(obj)
+            fields = arrow.type.Field.empty(0, obj.NumFields);
+            for ii = 1:obj.NumFields
+                fields(ii) = obj.field(ii);
+            end
+        end
+
+        function fieldNames = get.FieldNames(obj)
+            fieldNames = obj.Proxy.getFieldNames();
+        end
+
+        function numFields = get.NumFields(obj)
+            numFields = obj.Proxy.getNumFields();
+        end
+
+    end
+
+    methods (Access = private)
+
+        function str = toString(obj)
+            str = obj.Proxy.toString();
+        end
+
+    end
+
+    methods (Access=protected)
+
+        function displayScalarObject(obj)
+            disp(obj.toString());
+        end
+
+    end
+
+end

matlab/src/matlab/+arrow/schema.m

@@ -0,0 +1,34 @@
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements.  See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License.  You may obtain a copy of the License at
+%
+%   http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied.  See the License for the specific language governing
+% permissions and limitations under the License.
+
+function s = schema(fields)
+%SCHEMA Constructs an arrow.tabular.Schema object
+    arguments
+        fields(1, :) arrow.type.Field
+    end
+
+    % Extract the corresponding Proxy IDs from each of the
+    % supplied arrow.type.Field objects.
+    numFields = numel(fields);
+    fieldProxyIDs = zeros(1, numFields, "uint64");
+    for ii = 1:numFields
+        fieldProxyIDs(ii) = fields(ii).Proxy.ID;
+    end
+
+    % Construct an Arrow Schema Proxy in C++ from the supplied Field Proxy IDs.
+    args = struct(FieldProxyIDs=fieldProxyIDs);
+    proxy = arrow.internal.proxy.create("arrow.tabular.proxy.Schema", args);
+    s = arrow.tabular.Schema(proxy);
+end