diff --git a/doc/developer_guide/APIs.rst b/doc/developer_guide/APIs.rst index f74cf6f7f8..ea3f1ca774 100644 --- a/doc/developer_guide/APIs.rst +++ b/doc/developer_guide/APIs.rst @@ -48,29 +48,29 @@ TBD .. Generating API-specific code .. ============================ -.. +.. .. This section explains how to create a new API in PSyclone. PSyclone .. currently supports the following APIs: lfric and gocean. -.. +.. .. config.py .. --------- -.. +.. .. The names of the supported APIs and the default API are specified in .. ``configuration.py``. When adding a new API you must add the name you would like .. to use to the ``_supported_api_list``. -.. +.. .. parse.py .. -------- -.. +.. .. The parser reads the algorithm code and associated kernel metadata. -.. +.. .. The parser currently assumes that all APIs will use the ``invoke()`` .. API for the algorithm-to-psy layer but that the content and structure .. of the metadata in the kernel code may differ. If the algorithm API .. differs, then the parser will need to be refactored. This is beyond .. the scope of this document and is currently not considered in the .. PSyclone software architecture. -.. +.. .. The kernel metadata however, will be different from one API to .. another. To parse this kernel-API-specific metadata a .. ``KernelTypeFactory`` is provided which should return the appropriate @@ -79,7 +79,7 @@ TBD .. in the ``KernelTypeFactory`` class. If the kernel metadata happens to be .. the same as another existing API then the existing ``KernelType`` .. subclass can be used for the new API. -.. +.. .. The ``KernelType`` subclass needs to specialise the class constructor .. and initialise the ``KernelType`` base class with the .. supplied arguments. The role of the ``KernelType`` subclass is to create @@ -88,95 +88,95 @@ TBD .. this is appends the kernel-metadata-specific subclass instance is .. appended to the ``_arg_descriptors`` list provided by the ``KernelType`` .. base class. -.. +.. .. TBC -.. +.. .. This information -.. +.. .. KernelType base class assumes kernel metadata stored as a type. Searches for that type. .. Checks whether the metadata is public (it should be ?) .. Assumes iterates_over variable. .. Binding to a procedure - assumes one of two styles. .. Assumes a meta_args type .. *What about our func_args type???* -.. +.. .. type x .. meta_args= .. *meta_func=* .. iterates_over= .. code => or code = .. end type x -.. +.. .. The descriptor class ... -.. +.. .. psyGen.py .. --------- -.. +.. .. factory .. +++++++ -.. +.. .. A new file needs to be created and the following classes found in .. psyGen.py need to be subclassed. -.. +.. .. PSy, Invokes, Invoke, InvokeSchedule, Loop, Kern, Arguments, Argument .. You may also choose to subclass the Inf class if required. -.. +.. .. The subclass of the PSy class then needs to be added as an option to .. the create method in the PSyFactory class. -.. +.. .. Initialisation .. ++++++++++++++ -.. +.. .. The parser information passed to the PSy layer is used to create an .. invokes object which in turn creates a list of invoke objects. Each .. invoke object contains an InvokeSchedule which consists of loops and .. calls. Finally, a call contains an arguments object which itself .. contains a list of argument objects. -.. +.. .. To make sure the subclass versions of the above objects are created .. the __init__ methods of the subclasses must make sure they create .. the appropriate objects. -.. +.. .. Some of the baseclass constructors (__init__ methods) support the .. classname being provided. This allow them to instantiate the .. appropriate objects without knowing what they are. -.. +.. .. gen_code() .. ++++++++++ -.. +.. .. All of the above classes (with the exception of PSy which supports a .. gen() method) have the gen_code() method. This method passes the .. parent of the generation tree and expect the object to add the code .. associated with the object as a child of the parent. The object is .. then expected to call any children. This approach is powerful as it .. lets each object concentrate on the code that it is responsible for. -.. +.. .. Adding code in gen_code() .. +++++++++++++++++++++++++ -.. +.. .. The f2pygen classes have been developed to help create appropriate .. fortran code in the gen_code() method. -.. +.. .. When writing a gen_code() method for a particular object and API it is .. natural to add code as a child of the parent provided by the callee of .. the method. However, in some cases we do not want code to appear at .. the current position in the hierarchy. -.. +.. .. The add() method .. ++++++++++++++++ -.. +.. .. PSyclone supports this via the add() method -.. +.. .. explicitly place at the appropriate place in the hierarchy. For example, .. parent.parent.add(...) -.. +.. .. optional argument. default is auto. This attempts to place code in the .. expected place. For example, specify a declaration. auto finds a .. correct place to put this code. -.. +.. .. Specify position explicitly .. "before", "after", "first", "last" -.. +.. .. Sometimes don't know exactly where to place. On example that is .. supported is when you want to add something before or after a loop .. nest. start_parent_loop(). This method recurses up until the parent is @@ -333,7 +333,7 @@ grey) then we get: .. image:: dofs_cont_halos.png :width: 230 - + An example for a depth-1 halo implementation with the earlier mesh split into 2 partitions is given below, with the halo cells drawn in grey and halo dofs coloured red. An example local indexing scheme is @@ -403,9 +403,9 @@ Loop iterators -------------- In the current implementation of the LFRic API it is possible to -iterate (loop) either over cells or dofs. At the moment all coded -kernels are written to iterate over cells and all Built-in kernels are -written to iterate over dofs, but that does not have to be the case. +iterate (loop) either over cells or dofs. At the moment coded +kernels can be written to iterate over cells or dofs and all Built-in kernels +are written to iterate over dofs, but that does not have to be the case. The loop iteration information is specified in the kernel metadata. In the case of Built-ins there is kernel metadata but it is part of @@ -994,7 +994,7 @@ If an application is being built in parallel then it is possible that different invocations of PSyclone will happen simultaneously and therefore we must take care to avoid race conditions when querying the filesystem. For this reason we use ``os.open``:: - + fd = os.open(, os.O_CREAT | os.O_WRONLY | os.O_EXCL) The ``os.O_CREATE`` and ``os.O_EXCL`` flags in combination mean that @@ -1020,7 +1020,7 @@ of a given colour may be safely updated in parallel Example of the colouring of the horizontal cells used to ensure the thread-safe update of shared dofs (black circles). (Courtesy of S. Mullerworth, Met Office.) - + The loop over colours must then be performed sequentially but the loop over cells of a given colour may be done in parallel. A loop that requires colouring may be transformed using the ``Dynamo0p3ColourTrans`` @@ -1206,13 +1206,13 @@ TBD .. OpenMP Support .. -------------- -.. +.. .. Loop directives are treated as first class entities in the psyGen .. package. Therefore they can be added to psyGen's high level .. representation of the fortran code structure in the same way as calls .. and loops. Obviously it is only valid to add a loop directive outside .. of a loop. -.. +.. .. When adding a call inside a loop the placement of any additional calls .. or declarations must be specified correctly to ensure that they are .. placed at the correct location in the hierarchy. To avoid accidentally @@ -1222,7 +1222,7 @@ TBD .. f2pygen*. This method returns the location at the top of any loop .. hierarchy and before any comments immediately before the top level .. loop. -.. +.. .. The OpenMPLoopDirective object needs to know which variables are .. shared and which are private. In the current implementation default .. shared is used and private variables are listed. To determine the @@ -1233,13 +1233,13 @@ TBD .. the directive and adds each calls list of private variables, returned .. with the local_vars() method. Therefore the OpenMPLoopDirective object .. relies on calls specifying which variables they require being local. -.. +.. .. Next ... -.. +.. .. Update transformation for colours -.. -.. OpenMPLoop transformation in transformations.py. -.. +.. +.. OpenMPLoop transformation in transformations.py. +.. .. Create third transformation which goes over all loops in a schedule and .. applies the OpenMP loop transformation. diff --git a/doc/user_guide/dynamo0p3.rst b/doc/user_guide/dynamo0p3.rst index 29f00f6ad1..84e9f5d0df 100644 --- a/doc/user_guide/dynamo0p3.rst +++ b/doc/user_guide/dynamo0p3.rst @@ -796,9 +796,6 @@ support for i-first kernels point the looping (and associated parallelisation) will be put back into the PSy layer. -.. note:: Support for DoF kernels have not yet been implemented in PSyclone - (see PSyclone issue #1351 for progress). - .. _lfric-user-kernel-rules: Rules for all User-Supplied Kernels that Operate on Cell-Columns @@ -981,9 +978,6 @@ on a ``CELL_COLUMN`` without CMA Operators. Specifically: Rules for all User-Supplied Kernels that Operate on DoFs (DoF Kernels) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -.. note:: Support for DoF kernels have not yet been implemented in PSyclone - (see PSyclone issue #1351 for progress). - Kernels that have ``operates_on = DOF`` and :ref:`LFRic Built-ins` overlap significantly in their scope, and the conventions that DoF Kernels must follow are influenced @@ -1015,8 +1009,9 @@ The list of rules for DoF Kernels is as follows: to do this.) Any scalar arguments must therefore be declared in the metadata as `GH_READ` - see :ref:`below` -6) Kernels must be written to operate on a single DoF, such that single DoFs - can be provided to the Kernel within a loop over the DoFs of a field. +6) Kernels must be written to operate on a single DoF, such that field values + at the same dof location/index can be provided to the Kernel within a loop + over the DoFs of the function space of the field that is being updated. .. _lfric-api-kernel-metadata: @@ -1889,16 +1884,14 @@ operates_on The fourth type of metadata provided is ``OPERATES_ON``. This specifies that the Kernel has been written with the assumption that it is supplied with the specified data for each field/operator argument. -For user-supplied kernels this is currently only permitted to be -``CELL_COLUMN`` or ``DOMAIN``. The possible values for ``OPERATES_ON`` -and their interpretation are summarised in the following table: +The possible values for ``OPERATES_ON`` and their interpretation are +summarised in the following table: =========== ========================================================= operates_on Data passed for each field/operator argument =========== ========================================================= cell_column Single column of cells -dof Single DoF (currently :ref:`built-ins` only, but see PSyclone - issue #1351) +dof Single DoF domain All columns of cells =========== ========================================================= @@ -2477,9 +2470,6 @@ as the second argument to the kernel (after ``nlayers``). Rules for DoF Kernels ##################### -.. note:: Support for DoF kernels have not yet been implemented in PSyclone - (see PSyclone issue #1351 for progress). - The rules for kernels that have ``operates_on = DOF`` are similar to those for general-purpose kernels but, due to the restriction that only fields and scalars can be passed to them, are much fewer. The full set of rules, along @@ -2504,12 +2494,6 @@ with PSyclone's naming conventions, are: passed in separately. Again, the intent is determined from the metadata (see :ref:`meta_args `). - 3) Include the unique number of degrees of freedom for the function space. - This is an ``integer`` of kind ``i_def`` with intent ``in``. The name of - this argument is simply ``undf`` without a function space suffix (as for - general purpose kernels) since all fields will be on the same function - space. - .. _lfric-kernel-arg-intents: Argument Intents diff --git a/src/psyclone/domain/lfric/kern_call_acc_arg_list.py b/src/psyclone/domain/lfric/kern_call_acc_arg_list.py index 564e1b70e6..e6a5663911 100644 --- a/src/psyclone/domain/lfric/kern_call_acc_arg_list.py +++ b/src/psyclone/domain/lfric/kern_call_acc_arg_list.py @@ -43,7 +43,7 @@ ''' from psyclone import psyGen -from psyclone.domain.lfric import KernCallArgList +from psyclone.domain.lfric import KernCallArgList, LFRicConstants from psyclone.errors import InternalError @@ -93,6 +93,27 @@ def cell_position(self, var_accesses=None): _, ref = self.cell_ref_name(var_accesses) self.append(ref.symbol.name) + def field(self, arg, var_accesses=None): + '''Add the field array associated with the argument 'arg' to the + argument list. If supplied it also stores this access in var_accesses. + + :param arg: the field to be added. + :type arg: :py:class:`psyclone.dynamo0p3.DynKernelArgument` + :param var_accesses: optional VariablesAccessInfo instance to store + the information about variable accesses. + :type var_accesses: :py:class:`psyclone.core.VariablesAccessInfo` + + ''' + const = LFRicConstants() + suffix = const.ARG_TYPE_SUFFIX_MAPPING[arg.argument_type] + # Look-up the name of the variable that stores the reference to + # the data in this field. + sym = self._symtab.lookup_with_tag(f"{arg.name}:{suffix}") + + # Add the field data array as being read. + self.append(sym.name, var_accesses, var_access_name=sym.name, + mode=arg.access, metadata_posn=arg.metadata_index) + def stencil(self, arg, var_accesses=None): '''Add general stencil information associated with the argument 'arg' to the argument list. OpenACC requires the full dofmap to be diff --git a/src/psyclone/domain/lfric/kern_call_arg_list.py b/src/psyclone/domain/lfric/kern_call_arg_list.py index 583ff99ff7..ac4095628e 100644 --- a/src/psyclone/domain/lfric/kern_call_arg_list.py +++ b/src/psyclone/domain/lfric/kern_call_arg_list.py @@ -218,9 +218,9 @@ def mesh_height(self, var_accesses=None): '''Add mesh height (nlayers) to the argument list and if supplied stores this access in var_accesses. - :param var_accesses: optional VariablesAccessInfo instance to store + :param var_accesses: optional VariablesAccessInfo instance to store \ the information about variable accesses. - :type var_accesses: + :type var_accesses: \ :py:class:`psyclone.core.VariablesAccessInfo` ''' @@ -341,10 +341,9 @@ def field_vector(self, argvect, var_accesses=None): :param argvect: the field vector to add. :type argvect: :py:class:`psyclone.dynamo0p3.DynKernelArgument` - :param var_accesses: optional VariablesAccessInfo instance to store \ + :param var_accesses: optional VariablesAccessInfo instance to store the information about variable accesses. - :type var_accesses: \ - :py:class:`psyclone.core.VariablesAccessInfo` + :type var_accesses: :py:class:`psyclone.core.VariablesAccessInfo` ''' suffix = LFRicConstants().ARG_TYPE_SUFFIX_MAPPING[ @@ -380,11 +379,22 @@ def field(self, arg, var_accesses=None): # Look-up the name of the variable that stores the reference to # the data in this field. sym = self._symtab.lookup_with_tag(f"{arg.name}:{suffix}") - # Add the field data array as being read. - self.append(sym.name, var_accesses, var_access_name=sym.name, - mode=arg.access, metadata_posn=arg.metadata_index) - self.psyir_append(Reference(sym)) + if self._kern.iterates_over == "dof": + # If dof kernel, add access to the field by dof ref + dof_sym = self._symtab.find_or_create_integer_symbol( + "df", tag="dof_loop_idx") + self.append_array_reference(sym.name, [Reference(dof_sym)], + ScalarType.Intrinsic.INTEGER, + symbol=sym) + # Then append our symbol + name = f"{sym.name}({dof_sym.name})" + self.append(name, var_accesses, var_access_name=sym.name) + else: + # Add the field data array as being read. + self.append(sym.name, var_accesses, var_access_name=sym.name, + mode=arg.access, metadata_posn=arg.metadata_index) + self.psyir_append(Reference(sym)) def stencil_unknown_extent(self, arg, var_accesses=None): '''Add stencil information to the argument list associated with the @@ -611,23 +621,27 @@ def fs_compulsory_field(self, function_space, var_accesses=None): :py:class:`psyclone.core.VariablesAccessInfo` ''' - sym = self.append_integer_reference(function_space.undf_name) - self.append(sym.name, var_accesses) - - map_name = function_space.map_name - if self._kern.iterates_over == 'domain': - # This kernel takes responsibility for iterating over cells so - # pass the whole dofmap. - sym = self.append_array_reference(map_name, [":", ":"], - ScalarType.Intrinsic.INTEGER) - self.append(sym.name, var_accesses, var_access_name=sym.name) + if self._kern.iterates_over == "dof": + # Dofmaps and `undf` are not required for DoF kernels + pass else: - # Pass the dofmap for the cell column - cell_name, cell_ref = self.cell_ref_name(var_accesses) - sym = self.append_array_reference(map_name, [":", cell_ref], - ScalarType.Intrinsic.INTEGER) - self.append(f"{sym.name}(:,{cell_name})", - var_accesses, var_access_name=sym.name) + sym = self.append_integer_reference(function_space.undf_name) + self.append(sym.name, var_accesses) + + map_name = function_space.map_name + if self._kern.iterates_over == 'domain': + # This kernel takes responsibility for iterating over cells so + # pass the whole dofmap. + sym = self.append_array_reference(map_name, [":", ":"], + ScalarType.Intrinsic.INTEGER) + self.append(sym.name, var_accesses, var_access_name=sym.name) + else: + # Pass the dofmap for the cell column + cell_name, cell_ref = self.cell_ref_name(var_accesses) + sym = self.append_array_reference(map_name, [":", cell_ref], + ScalarType.Intrinsic.INTEGER) + self.append(f"{sym.name}(:,{cell_name})", + var_accesses, var_access_name=sym.name) def fs_intergrid(self, function_space, var_accesses=None): '''Add function-space related arguments for an intergrid kernel. diff --git a/src/psyclone/domain/lfric/lfric_kern.py b/src/psyclone/domain/lfric/lfric_kern.py index ddc3154b1d..d43b3a3f5f 100644 --- a/src/psyclone/domain/lfric/lfric_kern.py +++ b/src/psyclone/domain/lfric/lfric_kern.py @@ -557,6 +557,19 @@ def base_name(self): ''' return self._base_name + @property + def undf_name(self): + ''' + Dynamically looks up the name of the 'undf' variable for the + space that this kernel updates. + + :returns: the name of the undf variable. + :rtype: str + + ''' + field = self._arguments.iteration_space_arg() + return field.function_space.undf_name + @property def argument_kinds(self): ''' @@ -582,8 +595,8 @@ def gen_stub(self): const = LFRicConstants() supported_operates_on = const.USER_KERNEL_ITERATION_SPACES[:] # TODO #925 Add support for 'domain' kernels - # TODO #1351 Add suport for 'dof' kernels supported_operates_on.remove("domain") + # TODO #1351 Add support for 'dof' kernels supported_operates_on.remove("dof") # Check operates-on (iteration space) before generating code diff --git a/src/psyclone/domain/lfric/lfric_kern_call_factory.py b/src/psyclone/domain/lfric/lfric_kern_call_factory.py index 1dfc481593..eec261d546 100644 --- a/src/psyclone/domain/lfric/lfric_kern_call_factory.py +++ b/src/psyclone/domain/lfric/lfric_kern_call_factory.py @@ -70,6 +70,9 @@ def create(call, parent=None): # We still need a loop object though as that is where the logic # for handling halo exchanges is currently implemented. loop_type = "null" + elif call.ktype.iterates_over == "dof": + # Loop over dofs within a field. + loop_type = "dof" else: # Loop over cells, indicated by an empty string. loop_type = "" diff --git a/src/psyclone/domain/lfric/lfric_loop.py b/src/psyclone/domain/lfric/lfric_loop.py index e3bef9ea2d..16f334a2ed 100644 --- a/src/psyclone/domain/lfric/lfric_loop.py +++ b/src/psyclone/domain/lfric/lfric_loop.py @@ -44,7 +44,6 @@ from psyclone.core import AccessType from psyclone.domain.common.psylayer import PSyLoop from psyclone.domain.lfric import LFRicConstants, LFRicKern -from psyclone.domain.lfric.lfric_builtins import LFRicBuiltIn from psyclone.domain.lfric.lfric_types import LFRicTypes from psyclone.errors import GenerationError, InternalError from psyclone.f2pygen import CallGen, CommentGen @@ -225,9 +224,8 @@ def load(self, kern): # Loop bounds self.set_lower_bound("start") const = LFRicConstants() - if isinstance(kern, LFRicBuiltIn): - # If the kernel is a built-in/pointwise operation - # then this loop must be over DoFs + if kern.iterates_over == "dof": + # This loop must be over DoFs if Config.get().api_conf("lfric").compute_annexed_dofs \ and Config.get().distributed_memory \ and not kern.is_reduction: @@ -475,8 +473,9 @@ def _upper_bound_fortran(self): if self._upper_bound_name in ["ndofs", "nannexed"]: if Config.get().distributed_memory: if self._upper_bound_name == "ndofs": - result = (f"{self.field.proxy_name_indexed}%" - f"{self.field.ref_name()}%get_last_dof_owned()") + result = ( + f"{self.field.proxy_name_indexed}%" + f"{self.field.ref_name()}%get_last_dof_owned()") else: # nannexed result = ( f"{self.field.proxy_name_indexed}%" diff --git a/src/psyclone/tests/domain/lfric/dofkern_test.py b/src/psyclone/tests/domain/lfric/dofkern_test.py index 1f0dd9fda5..37c99b984b 100644 --- a/src/psyclone/tests/domain/lfric/dofkern_test.py +++ b/src/psyclone/tests/domain/lfric/dofkern_test.py @@ -31,20 +31,25 @@ # ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE # POSSIBILITY OF SUCH DAMAGE. # ----------------------------------------------------------------------------- -# Authors O. Brunt, Met Office +# Author O. Brunt, Met Office ''' -This module tests the metadata validation in LFRicKernMetadata of +This module tests metadata validation and code generation of user-supplied kernels operating on degrees of freedom (dofs) ''' - +import os import pytest from fparser import api as fpapi from psyclone.configuration import Config -from psyclone.domain.lfric import LFRicKernMetadata +from psyclone.domain.lfric import LFRicKernMetadata, LFRicLoop +from psyclone.dynamo0p3 import LFRicHaloExchange +from psyclone.parse.algorithm import parse from psyclone.parse.utils import ParseError +from psyclone.psyGen import PSyFactory +from psyclone.tests.lfric_build import LFRicBuild +from psyclone.transformations import Dynamo0p3RedundantComputationTrans @pytest.fixture(scope="module", autouse=True) @@ -55,6 +60,10 @@ def setup(): Config._instance = None +BASE_PATH = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname( + os.path.abspath(__file__)))), "test_files", "dynamo0p3") +TEST_API = "lfric" + CODE = ''' module testkern_dofs_mod type, extends(kernel_type) :: testkern_dofs_type @@ -77,7 +86,8 @@ def setup(): def test_dof_kernel_mixed_function_spaces(): - ''' Check that we raise an exception if we encounter a dof kernel + ''' + Check that we raise an exception if we encounter a dof kernel call with arguments of different function spaces. ''' @@ -93,7 +103,8 @@ def test_dof_kernel_mixed_function_spaces(): def test_dof_kernel_invalid_arg(): - ''' Check that we raise an exception if we find metadata for a dof kernel + ''' + Check that we raise an exception if we find metadata for a dof kernel which specifies arguments that are not fields or scalars. ''' @@ -120,7 +131,8 @@ def test_dof_kernel_invalid_arg(): def test_dof_kernel_invalid_field_vector(): - ''' Check that we raise an exception if we encounter metadata + ''' + Check that we raise an exception if we encounter metadata for a dof kernel with a field vector. ''' @@ -142,3 +154,211 @@ def test_dof_kernel_invalid_field_vector(): assert ("Kernel 'testkern_dofs_type' operates on 'dof' but has a vector " "argument 'gh_field*3'. This is not permitted in the LFRic API." in str(excinfo.value)) + + +def test_upper_bounds(monkeypatch, annexed, dist_mem, tmpdir): + ''' + Checks that the correct upper bound is generated for a dof-kernel for all + permutations of the `DISTRIBUTED_MEMORY` and `COMPUTE_ANNEXED_DOFS` + configuration settings. + + ''' + # Set up annexed dofs + config = Config.get() + lfric_config = config.api_conf("lfric") + monkeypatch.setattr(lfric_config, "_compute_annexed_dofs", annexed) + + _, invoke_info = parse(os.path.join(BASE_PATH, + "1.14_single_invoke_dofs.f90"), + api=TEST_API) + psy = PSyFactory(TEST_API, distributed_memory=dist_mem).create(invoke_info) + code = str(psy.gen) + + # Distributed memory + if annexed and dist_mem: + expected = (" loop0_start = 1\n" + " loop0_stop = f1_proxy%vspace%get_last_dof_annexed()" + ) + elif not annexed and dist_mem: + expected = (" loop0_start = 1\n" + " loop0_stop = f1_proxy%vspace%get_last_dof_owned()" + ) + + # Shared memory + elif not dist_mem: + expected = (" loop0_start = 1\n" + " loop0_stop = undf_w1" + ) + + assert expected in code + # Check compilation + assert LFRicBuild(tmpdir).code_compiles(psy) + + +def test_indexed_field_args(tmpdir): + ''' + Checks that the correct array references are generated for all field + arguments in a dof kernel. The index should be the same as the loop + index - 'df'. + + ''' + _, invoke_info = parse(os.path.join(BASE_PATH, + "1.14_single_invoke_dofs.f90"), + api=TEST_API) + psy = PSyFactory(TEST_API, distributed_memory=False).create(invoke_info) + code = str(psy.gen) + + expected = ("CALL testkern_dofs_code(f1_data(df), f2_data(df), " + "f3_data(df), f4_data(df), scalar_arg)") + + assert expected in code + # Check compilation + assert LFRicBuild(tmpdir).code_compiles(psy) + + +def test_redundant_comp_trans(tmpdir, monkeypatch): + ''' + Check that the correct halo exchanges are added if redundant + computation is enabled for a dof kernel called before a + user-supplied kernel. + + ''' + api_config = Config.get().api_conf(TEST_API) + monkeypatch.setattr(api_config, "_compute_annexed_dofs", True) + _, invoke_info = parse(os.path.join(BASE_PATH, + "1.14_single_invoke_dofs.f90"), + api=TEST_API) + psy = PSyFactory(TEST_API, distributed_memory=True).create(invoke_info) + + first_invoke = psy.invokes.invoke_list[0] + + # Since (redundant) computation over annexed dofs is enabled, there + # should be no halo exchange before the first kernel call + assert isinstance(first_invoke.schedule[0], LFRicLoop) + + # Now transform the first loop to perform redundant computation out to + # the level-1 halo + rtrans = Dynamo0p3RedundantComputationTrans() + rtrans.apply(first_invoke.schedule[0], options={"depth": 3}) + + # There should now be a halo exchange for f2 + assert isinstance(first_invoke.schedule[0], LFRicHaloExchange) + assert first_invoke.schedule[0].field.name == "f2" + # Check the correct depth is set + assert "halo_exchange(depth=3)" in str(psy.gen) + assert "halo_exchange(depth=1)" not in str(psy.gen) + + # There should be one halo exchange for each field that is not f1 + # (f2, f3, f4) + assert len([node for node in first_invoke.schedule.walk(LFRicHaloExchange) + if node.field.name == "f2"]) == 1 + assert len([node for node in first_invoke.schedule.walk(LFRicHaloExchange) + if node.field.name == "f3"]) == 1 + assert len([node for node in first_invoke.schedule.walk(LFRicHaloExchange) + if node.field.name == "f4"]) == 1 + # Check compiles + assert LFRicBuild(tmpdir).code_compiles(psy) + + +def test_multi_invoke_cell_dof_builtin(tmpdir, monkeypatch, annexed, dist_mem): + ''' + Check that the expected code is generated from a multi-kernel invoke with + kernels operating on different domains. + + ''' + # Set up annexed dofs + config = Config.get() + lfric_config = config.api_conf("lfric") + monkeypatch.setattr(lfric_config, "_compute_annexed_dofs", annexed) + + _, invoke_info = parse(os.path.join( + BASE_PATH, "4.17_multikernel_invokes_cell_dof_builtin.f90"), + api=TEST_API) + psy = PSyFactory(TEST_API, distributed_memory=dist_mem).create(invoke_info) + code = str(psy.gen) + + # Work through the generated code and compare what is expected to what is + # generated + + # Use statements + output = ( + " USE testkern_mod, ONLY: testkern_code\n" + " USE testkern_dofs_mod, ONLY: testkern_dofs_code\n" + ) + if dist_mem: + # Check mesh_mod is added to use statements + output += (" USE mesh_mod, ONLY: mesh_type\n") + assert output in code + + # Consistent declarations + output = ( + " REAL(KIND=r_def), intent(in) :: scalar_arg, a\n" + " TYPE(field_type), intent(in) :: f1, f2, f3, f4, m1, m2\n" + " INTEGER(KIND=i_def) cell\n" + " INTEGER(KIND=i_def) df\n" + " INTEGER(KIND=i_def) loop2_start, loop2_stop\n" + " INTEGER(KIND=i_def) loop1_start, loop1_stop\n" + " INTEGER(KIND=i_def) loop0_start, loop0_stop\n" + " INTEGER(KIND=i_def) nlayers_f1\n" + " REAL(KIND=r_def), pointer, dimension(:) :: m2_data => null()\n" + " REAL(KIND=r_def), pointer, dimension(:) :: m1_data => null()\n" + " REAL(KIND=r_def), pointer, dimension(:) :: f4_data => null()\n" + " REAL(KIND=r_def), pointer, dimension(:) :: f3_data => null()\n" + " REAL(KIND=r_def), pointer, dimension(:) :: f2_data => null()\n" + " REAL(KIND=r_def), pointer, dimension(:) :: f1_data => null()\n" + ) + assert output in code + + # Check that dof kernel is called correctly + output = ( + " DO df = loop0_start, loop0_stop, 1\n" + " CALL testkern_dofs_code(f1_data(df), f2_data(df), " + "f3_data(df), f4_data(df), scalar_arg)\n" + " END DO\n" + ) + assert output in code + + if dist_mem: + if annexed: + # Check halos are set dirty/clean for modified fields in dof kernel + output = ( + " IF (f2_proxy%is_dirty(depth=1)) THEN\n" + " CALL f2_proxy%halo_exchange(depth=1)\n" + " END IF\n" + " IF (m1_proxy%is_dirty(depth=1)) THEN\n" + " CALL m1_proxy%halo_exchange(depth=1)\n" + " END IF\n" + " IF (m2_proxy%is_dirty(depth=1)) THEN\n" + " CALL m2_proxy%halo_exchange(depth=1)\n" + " END IF\n" + ) + assert output in code + else: + # Check f1 field has halo exchange performed when annexed = true + output = ( + " CALL f1_proxy%set_dirty()\n" + " !\n" + " CALL f1_proxy%halo_exchange(depth=1)" + ) + assert output in code + + # Check cell-column kern is called correctly + output = ( + " DO cell = loop1_start, loop1_stop, 1\n" + " CALL testkern_code(nlayers_f1, a, f1_data, f2_data, m1_data, " + "m2_data, ndf_w1, undf_w1, map_w1(:,cell), ndf_w2, undf_w2, " + "map_w2(:,cell), ndf_w3, undf_w3, map_w3(:,cell))\n" + " END DO\n" + ) + assert output in code + + # Check built-in is called correctly + output = ( + " DO df = loop2_start, loop2_stop, 1\n" + " ! Built-in: inc_aX_plus_Y (real-valued fields)\n" + " f1_data(df) = 0.5_r_def * f1_data(df) + f2_data(df)\n" + " END DO\n" + ) + assert output in code + + assert LFRicBuild(tmpdir).code_compiles(psy) diff --git a/src/psyclone/tests/domain/lfric/kern_call_acc_arg_list_test.py b/src/psyclone/tests/domain/lfric/kern_call_acc_arg_list_test.py index 26f1f339bf..6e6d4a9f06 100644 --- a/src/psyclone/tests/domain/lfric/kern_call_acc_arg_list_test.py +++ b/src/psyclone/tests/domain/lfric/kern_call_acc_arg_list_test.py @@ -212,3 +212,31 @@ def test_lfric_stencil(): assert "f1: READ+WRITE" in var_info assert "f2: READ" in var_info assert "f2_stencil_dofmap: READ" in var_info + + +def test_lfric_field(): + '''Check that the method to generate a field argument returns the + field data varaible name and the correct variable access info. + + ''' + # Use the OpenACC transforms to create the required kernels + acc_par_trans = ACCParallelTrans() + acc_enter_trans = ACCEnterDataTrans() + _, invoke = get_invoke("15.1.1_builtin_and_normal_kernel_invoke_2.f90", + "lfric", + idx=0, dist_mem=False) + sched = invoke.schedule + acc_par_trans.apply(sched.children) + acc_enter_trans.apply(sched) + + # Find the first kernel: + kern = invoke.schedule.walk(psyGen.CodedKern)[0] + create_acc_arg_list = KernCallAccArgList(kern) + var_accesses = VariablesAccessInfo() + create_acc_arg_list.generate(var_accesses=var_accesses) + var_info = str(var_accesses) + # Check fields + assert "f1_data: READ+WRITE" in var_info # Written to in Built-in + assert "f2_data: READ" in var_info + assert "m1_data: READ" in var_info + assert "m2_data: READ" in var_info diff --git a/src/psyclone/tests/domain/lfric/lfric_kern_test.py b/src/psyclone/tests/domain/lfric/lfric_kern_test.py index c8ef438fe1..989a3e3e6e 100644 --- a/src/psyclone/tests/domain/lfric/lfric_kern_test.py +++ b/src/psyclone/tests/domain/lfric/lfric_kern_test.py @@ -456,3 +456,17 @@ def test_kern_not_coloured_inc(monkeypatch): assert ("Kernel 'testkern_code' has an argument with INC access and " "therefore must be coloured in order to be parallelised with " "OpenMP." in str(err.value)) + + +def test_undf_name(): + '''Tests that the LFRicKern.undf_name property returns the correct + result when called. + + ''' + _, invoke_info = parse(os.path.join(BASE_PATH, "1_single_invoke.f90"), + api=TEST_API) + psy = PSyFactory(TEST_API, distributed_memory=True).create(invoke_info) + sched = psy.invokes.invoke_list[0].schedule + kern = sched.walk(LFRicKern)[0] + + assert kern.undf_name == "undf_w1" diff --git a/src/psyclone/tests/test_files/dynamo0p3/1.14_single_invoke_dofs.f90 b/src/psyclone/tests/test_files/dynamo0p3/1.14_single_invoke_dofs.f90 index 4a8174e6b8..a165458824 100755 --- a/src/psyclone/tests/test_files/dynamo0p3/1.14_single_invoke_dofs.f90 +++ b/src/psyclone/tests/test_files/dynamo0p3/1.14_single_invoke_dofs.f90 @@ -36,16 +36,18 @@ program single_invoke_dofs ! Description: single user-defined kernel specified in an invoke call that - ! iterates over DoFs (currently not supported) + ! iterates over DoFs + use constants_mod, only: r_def use field_mod, only: field_type use testkern_dofs_mod, only: testkern_dofs_type implicit none type(field_type) :: f1, f2, f3, f4 + real(kind=r_def) :: scalar_arg call invoke( & - testkern_dofs_type(f1, f2, f3, f4) & + testkern_dofs_type(f1, f2, f3, f4, scalar_arg) & ) end program single_invoke_dofs diff --git a/src/psyclone/tests/test_files/dynamo0p3/4.17_multikernel_invokes_cell_dof_builtin.f90 b/src/psyclone/tests/test_files/dynamo0p3/4.17_multikernel_invokes_cell_dof_builtin.f90 new file mode 100644 index 0000000000..488693fe37 --- /dev/null +++ b/src/psyclone/tests/test_files/dynamo0p3/4.17_multikernel_invokes_cell_dof_builtin.f90 @@ -0,0 +1,57 @@ +! ----------------------------------------------------------------------------- +! BSD 3-Clause License +! +! Copyright (c) 2020-2024, Science and Technology Facilities Council +! All rights reserved. +! +! Redistribution and use in source and binary forms, with or without +! modification, are permitted provided that the following conditions are met: +! +! * Redistributions of source code must retain the above copyright notice, this +! list of conditions and the following disclaimer. +! +! * Redistributions in binary form must reproduce the above copyright notice, +! this list of conditions and the following disclaimer in the documentation +! and/or other materials provided with the distribution. +! +! * Neither the name of the copyright holder nor the names of its +! contributors may be used to endorse or promote products derived from +! this software without specific prior written permission. +! +! THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +! "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +! LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS +! FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE +! COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +! INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, +! BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +! LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +! CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +! LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN +! ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +! POSSIBILITY OF SUCH DAMAGE. +! ----------------------------------------------------------------------------- +! Author O. Brunt, Met Office + +program multikernel_invokes_cell_dof_builtin + + ! Description: Two user-defined kernels operating over different domains + ! ("cell-column" and "dof"), followed by an LFRic built-in kernel in the + ! same invoke. + use constants_mod, only: r_def + use field_mod, only: field_type + use testkern_dofs_mod, only: testkern_dofs_type + use testkern_mod, only: testkern_type + + implicit none + + type(field_type) :: f1, f2, f3, f4, m1, m2 + real(kind=r_def) :: a, scalar_arg + + call invoke( & + testkern_dofs_type(f1, f2, f3, f4, scalar_arg), & + testkern_type(a, f1, f2, m1, m2), & + inc_aX_plus_Y(0.5_r_def, f1, f2) & + ) + +end program multikernel_invokes_cell_dof_builtin \ No newline at end of file diff --git a/src/psyclone/tests/test_files/dynamo0p3/testkern_dofs_mod.f90 b/src/psyclone/tests/test_files/dynamo0p3/testkern_dofs_mod.f90 index 0a4e34ecc8..9ebaa90ea5 100644 --- a/src/psyclone/tests/test_files/dynamo0p3/testkern_dofs_mod.f90 +++ b/src/psyclone/tests/test_files/dynamo0p3/testkern_dofs_mod.f90 @@ -41,13 +41,13 @@ module testkern_dofs_mod implicit none - ! User-defined single kernel that operates on DoFs (currently not supported) type, extends(kernel_type) :: testkern_dofs_type - type(arg_type), dimension(4) :: meta_args = & + type(arg_type), dimension(5) :: meta_args = & (/ arg_type(gh_field, gh_real, gh_write, w1), & arg_type(gh_field, gh_real, gh_read, w1), & arg_type(gh_field, gh_real, gh_read, w1), & - arg_type(gh_field, gh_real, gh_read, w1) & + arg_type(gh_field, gh_real, gh_read, w1), & + arg_type(gh_scalar, gh_real, gh_read) & /) integer :: operates_on = DOF contains @@ -56,7 +56,13 @@ module testkern_dofs_mod contains - subroutine testkern_dofs_code(a, b, c, d) + subroutine testkern_dofs_code(a, b, c, d, scalar_arg) + implicit none + + real(kind=r_def), intent(inout) :: a + real(kind=r_def), intent(in) :: b, c, d + real(kind=r_def), intent(in) :: scalar_arg + end subroutine testkern_dofs_code end module testkern_dofs_mod