22_deep_learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# PyTorch\n",
        "\n",
        "Открытый фреймворк для построения и использования динамических графов вычислений и глубокого обучения (вспоминаем, что нейросеть -- это вычислительный граф)\n",
        "\n",
        "Альтернативы: TensorFlow, JAX, Caffe\n",
        "\n",
        "Изначально разрабатывался FaceBook's AI Research Lab (FAIR)\n",
        "\n",
        "Вместе с функционалом Python удобен для экспериментов и разработки (минимум кода при максимуме возможностей)\n",
        "\n",
        "Наиболее важные для DL возможности: автоматическое дифференцирование, вычисления на базе многомерных матриц (тензоров) - очень похож на numpy, поддержка динамических вычислительных графов (создаются при работе), поддержка вычислений на GPU, есть полезные модули (например, torchvision).\n",
        "\n",
        "Установка с официального сайта: [https://pytorch.org/get-started/locally/](https://pytorch.org/get-started/locally/)."
      ],
      "metadata": {
        "id": "wKxb2ruTo9wC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# проверка версии Питона, Пайторча и проверка доступности видеокарты\n",
        "\n",
        "import torch\n",
        "from platform import python_version\n",
        "print(python_version())\n",
        "print(torch.__version__)\n",
        "print(torch.cuda.is_available())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IiS2yVkEo94p",
        "outputId": "5f851d30-5f1d-4342-f7da-aeafb2b3a8df"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3.10.12\n",
            "2.0.1+cu118\n",
            "True\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# использование Google-диска при работе в COLAB\n",
        "\n",
        "from google.colab import drive\n",
        "import pandas as pd\n",
        "\n",
        "drive.mount(\"/content/gdrive\", force_remount=True)\n",
        "\n",
        "data_path = \"/content/gdrive/My Drive/Colab Notebooks/deep_learning_data/\"\n",
        "train_ann_path = data_path + 'train.csv'\n",
        "\n",
        "train_df = pd.read_csv(train_ann_path)\n",
        "print(train_df.head())\n",
        "\n",
        "# команды для bash пишутся с !\n",
        "!ls /content/gdrive/My\\ Drive/Colab\\ Notebooks/deep_learning_data/"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1D3xPFPUrq6A",
        "outputId": "ab28046d-bc2a-481d-9481-1c3809fd29cf"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/gdrive\n",
            "   row_id     A0T0G0C10  A0T0G1C9  A0T0G2C8  A0T0G3C7  A0T0G4C6  A0T0G5C5  \\\n",
            "0       0 -9.536743e-07 -0.000010 -0.000043 -0.000114 -0.000200 -0.000240   \n",
            "1       1 -9.536743e-07 -0.000010 -0.000043  0.000886 -0.000200  0.000760   \n",
            "2       2 -9.536743e-07 -0.000002  0.000007  0.000129  0.000268  0.000270   \n",
            "3       3  4.632568e-08 -0.000006  0.000012  0.000245  0.000492  0.000522   \n",
            "4       4 -9.536743e-07 -0.000010 -0.000043 -0.000114 -0.000200 -0.000240   \n",
            "\n",
            "   A0T0G6C4  A0T0G7C3  A0T0G8C2  ...  A8T0G1C1  A8T0G2C0  A8T1G0C1  A8T1G1C0  \\\n",
            "0 -0.000200 -0.000114 -0.000043  ... -0.000086 -0.000043 -0.000086 -0.000086   \n",
            "1 -0.000200 -0.000114 -0.000043  ... -0.000086 -0.000043  0.000914  0.000914   \n",
            "2  0.000243  0.000125  0.000001  ...  0.000084  0.000048  0.000081  0.000106   \n",
            "3  0.000396  0.000197 -0.000003  ...  0.000151  0.000100  0.000180  0.000202   \n",
            "4 -0.000200 -0.000114 -0.000043  ... -0.000086 -0.000043 -0.000086 -0.000086   \n",
            "\n",
            "   A8T2G0C0  A9T0G0C1  A9T0G1C0  A9T1G0C0     A10T0G0C0  \\\n",
            "0 -0.000043 -0.000010 -0.000010 -0.000010 -9.536743e-07   \n",
            "1 -0.000043 -0.000010 -0.000010 -0.000010 -9.536743e-07   \n",
            "2  0.000072  0.000010  0.000008  0.000019  1.046326e-06   \n",
            "3  0.000153  0.000021  0.000015  0.000046 -9.536743e-07   \n",
            "4 -0.000043 -0.000010 -0.000010 -0.000010 -9.536743e-07   \n",
            "\n",
            "                   target  \n",
            "0  Streptococcus_pyogenes  \n",
            "1     Salmonella_enterica  \n",
            "2     Salmonella_enterica  \n",
            "3     Salmonella_enterica  \n",
            "4      Enterococcus_hirae  \n",
            "\n",
            "[5 rows x 288 columns]\n",
            "train.csv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Тензоры\n",
        "\n",
        "Тензоры (torch.Tensor) - аналоги многомерных массивов пакета numpy, только могут располагаться на GPU (или поддерживать вычисления на нескольких CPU), могут быть элементами вычислительного графа и поддерживать автоматическое дифференцирование (об этом позже). Нейросеть -- это в каком-то смысле последовательность тензоров.\n",
        "\n",
        "Это фундаментальная структура данных в Pytorch (с помощью неё будут храниться и обрабатываться объекты: тексты, сигналы изображения и батчи - наборы объектов).\n",
        "\n",
        "Могут в многомерном матричном виде хранить данные определённого типа."
      ],
      "metadata": {
        "id": "cLfOZN8qpK0G"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([[1, 2, 3], [4, 5, 6]])  # тензор на CPU\n",
        "\n",
        "print(x,\n",
        "      x.shape,  # размер тензора\n",
        "      x.dtype,  # тип\n",
        "      x.device,  # где лежит\n",
        "      x.type(),  # тип\n",
        "      x.dim(),  # размерность\n",
        "      x.size(),  # размер; .shape и .size() одно и то же\n",
        "      x.numel(),  # число элементов тензора\n",
        "      sep='\\n'\n",
        "      )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cF_dn7QW38zC",
        "outputId": "69b8b8dc-9fa1-4e17-d340-3224fab05f9c"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1, 2, 3],\n",
            "        [4, 5, 6]])\n",
            "torch.Size([2, 3])\n",
            "torch.int64\n",
            "cpu\n",
            "torch.LongTensor\n",
            "2\n",
            "torch.Size([2, 3])\n",
            "6\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.cuda.FloatTensor(2, 3)  # тензор на GPU\n",
        "print(x)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "85UgwgPQ3dko",
        "outputId": "fa253197-5784-4c7e-e0e0-b95988404ae1"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[0., 0., 0.],\n",
            "        [0., 0., 0.]], device='cuda:0')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# приведение типов\n",
        "\n",
        "x = torch.IntTensor([1, 2]).float()\n",
        "print(x, '\\n')\n",
        "\n",
        "x = torch.IntTensor([1, 2]).to(torch.float64)\n",
        "print(x, '\\n')\n",
        "\n",
        "x = torch.IntTensor([1, 2]) + 0.0\n",
        "print(x, '\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O2u0rj-E3nG4",
        "outputId": "a7cff233-0be4-4829-adca-fe81f09b91cf"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([1., 2.]) \n",
            "\n",
            "tensor([1., 2.], dtype=torch.float64) \n",
            "\n",
            "tensor([1., 2.]) \n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# пустой тензор\n",
        "x = torch.empty(3, 5)\n",
        "print(x, '\\n')\n",
        "\n",
        "# тензор из единиц\n",
        "x = torch.ones(3, 5)\n",
        "print(x, '\\n')\n",
        "\n",
        "# тензор из 3.14\n",
        "x = torch.full((3, 5), 3.14, dtype=torch.float)\n",
        "print(x, '\\n')\n",
        "\n",
        "# единичный тензор (с единицами на главной диагонали)\n",
        "x = torch.eye(3, 5)\n",
        "print(x, '\\n')\n",
        "\n",
        "# случайный тензор с элементами, равномерно распределёнными на [0, 1]\n",
        "torch.manual_seed(123)\n",
        "x = torch.rand(3, 5)\n",
        "print(x, '\\n')\n",
        "\n",
        "# случайный тензор с нормально распределёнными элементами\n",
        "torch.manual_seed(123)\n",
        "x = torch.randn(3, 5)\n",
        "print(x, '\\n')\n",
        "\n",
        "# случайный тензор с числами от 2 до 4 (не включая 4)\n",
        "torch.manual_seed(123)\n",
        "x = torch.randint(2, 4, (3, 5))\n",
        "print(x, '\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ki3QTwPZ5SD-",
        "outputId": "4f4b6eb6-434b-468b-88ea-e6e7a398b554"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[2.1826e-38, 3.1616e-41, 1.0525e-38, 3.1616e-41, 0.0000e+00],\n",
            "        [0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00],\n",
            "        [1.4013e-45, 0.0000e+00, 0.0000e+00, 0.0000e+00, 9.1084e-44]]) \n",
            "\n",
            "tensor([[1., 1., 1., 1., 1.],\n",
            "        [1., 1., 1., 1., 1.],\n",
            "        [1., 1., 1., 1., 1.]]) \n",
            "\n",
            "tensor([[3.1400, 3.1400, 3.1400, 3.1400, 3.1400],\n",
            "        [3.1400, 3.1400, 3.1400, 3.1400, 3.1400],\n",
            "        [3.1400, 3.1400, 3.1400, 3.1400, 3.1400]]) \n",
            "\n",
            "tensor([[1., 0., 0., 0., 0.],\n",
            "        [0., 1., 0., 0., 0.],\n",
            "        [0., 0., 1., 0., 0.]]) \n",
            "\n",
            "tensor([[0.2961, 0.5166, 0.2517, 0.6886, 0.0740],\n",
            "        [0.8665, 0.1366, 0.1025, 0.1841, 0.7264],\n",
            "        [0.3153, 0.6871, 0.0756, 0.1966, 0.3164]]) \n",
            "\n",
            "tensor([[-0.1115,  0.1204, -0.3696, -0.2404, -1.1969],\n",
            "        [ 0.2093, -0.9724, -0.7550,  0.3239, -0.1085],\n",
            "        [ 0.2103, -0.3908,  0.2350,  0.6653,  0.3528]]) \n",
            "\n",
            "tensor([[2, 3, 2, 2, 2],\n",
            "        [2, 2, 3, 3, 2],\n",
            "        [3, 3, 2, 3, 2]]) \n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# аналог np.arange\n",
        "x = torch.arange(0, 10, 2)\n",
        "print(x, '\\n')\n",
        "\n",
        "# аналог np.linspace\n",
        "x = torch.linspace(0, 10, 3)\n",
        "print(x, '\\n')\n",
        "\n",
        "# аналог np.logspace\n",
        "x = torch.logspace(0, 1, 3)\n",
        "print(x, '\\n')\n",
        "\n",
        "# сделать тензоры по образцу (использовать такой же тип и размеры)\n",
        "print(torch.empty_like(x), '\\n')\n",
        "print(torch.zeros_like(x), '\\n')\n",
        "print(torch.ones_like(x), '\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w5V-npFm5iF7",
        "outputId": "4ad60cec-134b-479b-ef16-d3d513643712"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0, 2, 4, 6, 8]) \n",
            "\n",
            "tensor([ 0.,  5., 10.]) \n",
            "\n",
            "tensor([ 1.0000,  3.1623, 10.0000]) \n",
            "\n",
            "tensor([9.7306e-39, 3.1616e-41, 1.9549e-38]) \n",
            "\n",
            "tensor([0., 0., 0.]) \n",
            "\n",
            "tensor([1., 1., 1.]) \n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# индексация аналогичная принятой в Питоне (в частности в numpy):\n",
        "# [start:end:step]\n",
        "\n",
        "x = torch.randint(0, 10, (2, 5))\n",
        "\n",
        "# есть тонкий момент: вектор с размерностью n или тензор с размерностью 1 на n\n",
        "# например целевые значения задаются матрицей размера 1 на n, а не n-мерным\n",
        "# вектором\n",
        "print(x, '\\n')\n",
        "print((x[0], x[0, :], x[[0], :], x[:1, :]), '\\n')\n",
        "print((x[:, [1]], x[:, 1], x[:, -4]), '\\n')\n",
        "\n",
        "print('Это тензор 1x1: ', x[0, 0], '\\n')\n",
        "print('А это уже отдельный элемент: ', x[0, 0].item(), '\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8mY3J0E8PKvo",
        "outputId": "0db4ea61-e805-49c6-f3fd-28f1d233062e"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[9, 4, 1, 3, 0],\n",
            "        [0, 6, 5, 7, 9]]) \n",
            "\n",
            "(tensor([9, 4, 1, 3, 0]), tensor([9, 4, 1, 3, 0]), tensor([[9, 4, 1, 3, 0]]), tensor([[9, 4, 1, 3, 0]])) \n",
            "\n",
            "(tensor([[4],\n",
            "        [6]]), tensor([4, 6]), tensor([4, 6])) \n",
            "\n",
            "Это тензор 1x1:  tensor(9) \n",
            "\n",
            "А это уже отдельный элемент:  9 \n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# копирование\n",
        "# при использовании clone() копия остаётся в графе вычислений\n",
        "# при использовании copy_() такого не происходит\n",
        "# detach - убирает информацию, связанную с вычислительным графом, из объекта\n",
        "\n",
        "a = torch.tensor([[1, 2], [3, 4]])\n",
        "\n",
        "b = a.new_tensor(a)\n",
        "b = a.clone().detach()\n",
        "b = torch.empty_like(a).copy_(a)\n",
        "b = torch.tensor(a)\n",
        "b = a.detach().clone()  # лучше копировать так"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qb08VBWEQ8na",
        "outputId": "f9c7a37a-229b-498b-cb0f-4d3fa316cef8"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-9-cdb08a1bda04>:8: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than tensor.new_tensor(sourceTensor).\n",
            "  b = a.new_tensor(a)\n",
            "<ipython-input-9-cdb08a1bda04>:11: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
            "  b = torch.tensor(a)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# при транспонировании не происходит копирования (используется та же память)\n",
        "\n",
        "xt = x.t()\n",
        "x[0, 0] = 30\n",
        "print('x: ', x, '\\n')\n",
        "print('xt: ', xt, '\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yZArJk9sUNFa",
        "outputId": "fbd8c4fb-e426-4f8b-c384-bcbcc17bcfb1"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "x:  tensor([[30,  4,  1,  3,  0],\n",
            "        [ 0,  6,  5,  7,  9]]) \n",
            "\n",
            "xt:  tensor([[30,  0],\n",
            "        [ 4,  6],\n",
            "        [ 1,  5],\n",
            "        [ 3,  7],\n",
            "        [ 0,  9]]) \n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([[1, 2], [3, 4]])\n",
        "y = torch.tensor([[2, 2], [2, 2]])\n",
        "\n",
        "z = torch.stack((x, y), dim=0)  # состыковка тензоров (по умолчанию dim=0)\n",
        "print(z, z.shape, sep='\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "slbmO83LVHvg",
        "outputId": "da544c19-cad4-4c5e-8893-1e319bceeed0"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[[1, 2],\n",
            "         [3, 4]],\n",
            "\n",
            "        [[2, 2],\n",
            "         [2, 2]]])\n",
            "torch.Size([2, 2, 2])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x, y = z.unbind(dim=0)  # разстыковка тензоров\n",
        "print(x, y, sep='\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "07qk1KdNZoam",
        "outputId": "cb72d013-25aa-438a-962e-4405c3684de3"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1, 2],\n",
            "        [3, 4]])\n",
            "tensor([[2, 2],\n",
            "        [2, 2]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# конкатенация по 0 и 1 размерностям (отличие от состыковки в том, что\n",
        "# используются уже существующие размерности)\n",
        "\n",
        "x = torch.tensor([[1, 2], [3, 4]])\n",
        "y = torch.tensor([[2, 2], [2, 2]])\n",
        "print(torch.cat([x, y], axis=1), '\\n')\n",
        "print(torch.cat([x, y], axis=0), '\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zuSLeyrnWl1I",
        "outputId": "eb8dab8b-df7f-475e-d217-fd9e340c38e4"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1, 2, 2, 2],\n",
            "        [3, 4, 2, 2]]) \n",
            "\n",
            "tensor([[1, 2],\n",
            "        [3, 4],\n",
            "        [2, 2],\n",
            "        [2, 2]]) \n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# создание фиктивной размерности - указывается, в какую позицию вставлять\n",
        "# фиктивную\n",
        "\n",
        "x = torch.tensor([[1, 2], [3, 4]])\n",
        "print(x.unsqueeze(dim=0).shape,\n",
        "      x.unsqueeze(dim=1).shape,\n",
        "      x.unsqueeze(dim=2).shape)\n",
        "print(x[None, :, :].shape,\n",
        "      x[:, None, :].shape,\n",
        "      x[:, :, None].shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pl8ASmNZZTkk",
        "outputId": "ad03c2ca-9e0f-43af-e3bc-0abf1941cc07"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([1, 2, 2]) torch.Size([2, 1, 2]) torch.Size([2, 2, 1])\n",
            "torch.Size([1, 2, 2]) torch.Size([2, 1, 2]) torch.Size([2, 2, 1])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# удаление единичных размеров\n",
        "torch.empty(3, 1, 2, 1).squeeze().shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V1T-6Q7pcqAW",
        "outputId": "a1f9ba46-7dec-4b92-8563-1be441e45e63"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([3, 2])"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# как хранятся в памяти двухмерные тензоры? Что такое смежность тензора?\n",
        "h = torch.rand((10000, 10000))"
      ],
      "metadata": {
        "id": "9COchGokcxL_"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "s = torch.sum(h, axis=1)  # суммы строк быстрее, чем суммы столбцов"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ISzn2NzldfZr",
        "outputId": "95d7f892-c0dc-4f04-97a6-3dc80319c2c1"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 36.8 ms, sys: 0 ns, total: 36.8 ms\n",
            "Wall time: 35.5 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "s = torch.sum(h, axis=0)  # суммы столбцов медленнее, чем суммы строк"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iPntupKOdily",
        "outputId": "cdf093bb-ffbf-4be7-ee0f-ac85d0b8202b"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 81.9 ms, sys: 0 ns, total: 81.9 ms\n",
            "Wall time: 83.3 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.randn(3, 2)\n",
        "print(x.is_contiguous())  # смежный\n",
        "y = torch.transpose(x, 0, 1)  # в общем случае транспонирование -\n",
        "# это перестановка размерностей\n",
        "print(y.is_contiguous())  # несмежный\n",
        "x[0, 0] = 101  # но они делят память\n",
        "print(y[0, 0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0mFWbYnadx9U",
        "outputId": "005a59ef-11b3-483d-8a8c-200ea7a77ec2"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "True\n",
            "False\n",
            "tensor(101.)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Можно менять представление тензора с помощью view\n",
        "# Это изменение размеров, но реально данные не перемещаются\n",
        "# Pytorch просто запоминает, что тензор, заданный элементами,\n",
        "# лежащими в определённой области, имеет другой размер\n",
        "# view работает только с contiguous тензорами\n",
        "\n",
        "x = torch.arange(4*10*2).view(4, 10, 2)\n",
        "y = x.permute(2, 0, 1)\n",
        "\n",
        "print('Смежность:', x.is_contiguous())\n",
        "print('Вытягиваем:', x.view(-1))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jn8w7eFcfP7I",
        "outputId": "c87c52c7-f0f2-4d3e-b6f0-8e34632b7454"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Смежность: True\n",
            "Вытягиваем: tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,\n",
            "        18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,\n",
            "        36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,\n",
            "        54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,\n",
            "        72, 73, 74, 75, 76, 77, 78, 79])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('Смежность:', y.is_contiguous())\n",
        "print('Вытягиваем:', y.view(-1))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 228
        },
        "id": "Vc2B_mqP8Fle",
        "outputId": "2eda8f5e-6b20-4494-f79a-2314b22e326a"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Смежность: False\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "RuntimeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-21-e3ec071b72ae>\u001b[0m in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Смежность:'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_contiguous\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Вытягиваем:'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mview\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead."
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Reshape работает всегда; старается выдать view и\n",
        "# если не получается, то делает копию данных\n",
        "# При Reshape тензор может копироваться! Это долго!\n",
        "\n",
        "print('Решейпим:', y.reshape(-1))\n",
        "print('Делаем смежным и решейпим:', y.contiguous().view(-1))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JvDQLqq1876C",
        "outputId": "57637550-2596-41c1-889f-a153a3f32738"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Решейпим: tensor([ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34,\n",
            "        36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70,\n",
            "        72, 74, 76, 78,  1,  3,  5,  7,  9, 11, 13, 15, 17, 19, 21, 23, 25, 27,\n",
            "        29, 31, 33, 35, 37, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63,\n",
            "        65, 67, 69, 71, 73, 75, 77, 79])\n",
            "Делаем смежным и решейпим: tensor([ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34,\n",
            "        36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70,\n",
            "        72, 74, 76, 78,  1,  3,  5,  7,  9, 11, 13, 15, 17, 19, 21, 23, 25, 27,\n",
            "        29, 31, 33, 35, 37, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59, 61, 63,\n",
            "        65, 67, 69, 71, 73, 75, 77, 79])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.arange(8)\n",
        "(x.view(4, 2), x.view(2, -1))\n",
        "# здесь сам тензор не поменялся, так как не было присваивания x = x.view()\n",
        "# в Pytorch тензоры хранятся в формате [channel, height, width],\n",
        "# в других системах чаще [height, width, channel]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L2pSdT2N9Qvh",
        "outputId": "6c8f76f4-0134-4fb6-b5d8-d8a8287aea0a"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([[0, 1],\n",
              "         [2, 3],\n",
              "         [4, 5],\n",
              "         [6, 7]]),\n",
              " tensor([[0, 1, 2, 3],\n",
              "         [4, 5, 6, 7]]))"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(x, x.storage(), x.stride(), x.t().stride(), sep='\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XaMH-vfT-F_6",
        "outputId": "71ebbe19-32c5-4cc3-eb79-00bf0d495930"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([0, 1, 2, 3, 4, 5, 6, 7])\n",
            " 0\n",
            " 1\n",
            " 2\n",
            " 3\n",
            " 4\n",
            " 5\n",
            " 6\n",
            " 7\n",
            "[torch.storage.TypedStorage(dtype=torch.int64, device=cpu) of size 8]\n",
            "(1,)\n",
            "(1,)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-24-a894e8149624>:1: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()\n",
            "  print(x, x.storage(), x.stride(), x.t().stride(), sep='\\n')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "z = torch.rand(1, 2, 3, 4)\n",
        "z = z.permute(0, 3, 1, 2)  # NxHxWxC -> NxCxHxW\n",
        "z.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6kd_l6IB-gyP",
        "outputId": "ae7220a0-91f5-4fcd-c4f4-762fec76751e"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 4, 2, 3])"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([[0, 1], [2, 3], [4, 5], [6, 7]])\n",
        "(x.transpose(0, 1), x.t(), x.t_())  # черта _ означает in-place операцию\n",
        "# в Pandas для in-place операций есть отдельный аргумент, а в Pytorch\n",
        "# просто черта\n",
        "# За in-place операциями нужно внимательно следить, иначе можно всё сломать"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NEhF1uPNAOiK",
        "outputId": "3f1db079-fe38-4414-9cdb-8bcd41b58aab"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([[0, 2, 4, 6],\n",
              "         [1, 3, 5, 7]]),\n",
              " tensor([[0, 2, 4, 6],\n",
              "         [1, 3, 5, 7]]),\n",
              " tensor([[0, 2, 4, 6],\n",
              "         [1, 3, 5, 7]]))"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([[0, 1], [2, 3], [4, 5], [6, 7]])\n",
        "(x.flatten(), x.view(-1))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aJ78_83EAVl5",
        "outputId": "b78a822f-d14d-4159-f89e-4ccf945fa7a1"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([0, 1, 2, 3, 4, 5, 6, 7]), tensor([0, 1, 2, 3, 4, 5, 6, 7]))"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Большинство операций с тензорами аналогичны операциям в numpy\n",
        "# Большинство операций выполняются поэлементно\n",
        "# Поддерживаются операции линейной алгебры, многие из которых взяты\n",
        "# из библиотек Basic Linear Algebra Subprograms (BLAS) и Linear Algebra\n",
        "# Package (LAPACK)\n",
        "\n",
        "x = torch.tensor([[1, 2], [3, 4]])\n",
        "\n",
        "# заполнение\n",
        "x.fill_(3)  # черта _ означает выполнение на данном тензоре\n",
        "print(x, '\\n')\n",
        "# обнуление\n",
        "x.zero_()\n",
        "print(x, '\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L7vZ8YWZAxtg",
        "outputId": "e338170a-5752-42c1-b743-8c6f69d5b26f"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[3, 3],\n",
            "        [3, 3]]) \n",
            "\n",
            "tensor([[0, 0],\n",
            "        [0, 0]]) \n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([[1, 2], [3, 4]])\n",
        "y = torch.tensor([[2, 2], [2, 2]])\n",
        "v = torch.tensor([1, 2])\n",
        "\n",
        "# сложение\n",
        "print(x + y, x.add(y), sep='\\n')\n",
        "\n",
        "# черта означает inplace-операцию (меняется первый тензор):\n",
        "x.add_(y)\n",
        "print(x, '\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yeC1bUiLCrMC",
        "outputId": "19d1585a-78e8-4d7f-8124-86dc7a5c147d"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[3, 4],\n",
            "        [5, 6]])\n",
            "tensor([[3, 4],\n",
            "        [5, 6]])\n",
            "tensor([[3, 4],\n",
            "        [5, 6]]) \n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([[1, 2], [3, 4]])\n",
        "y = torch.tensor([[2, 2], [2, 2]])\n",
        "v = torch.tensor([1, 2])\n",
        "\n",
        "# поэлементное умножение\n",
        "print(x * y, x.mul(y), torch.mul(x, y), sep='\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p32kt6F77muk",
        "outputId": "e5a93c04-cb15-488c-ac53-7ab95973f0f8"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[2, 4],\n",
            "        [6, 8]])\n",
            "tensor([[2, 4],\n",
            "        [6, 8]])\n",
            "tensor([[2, 4],\n",
            "        [6, 8]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# матричное умножение\n",
        "\n",
        "# torch.matmul - операция определена над тензорами (можно указывать\n",
        "# размерность для умножения)\n",
        "# torch.mm - обычное матричное умножение, но без приведения размеров\n",
        "# (broadcasting)\n",
        "# torch.bmm - матричное умножение с поддержкой батчей\n",
        "# (b x n x m) * (b x m x p) = b x n x p\n",
        "\n",
        "print(x @ y, x.mm(y), x.matmul(y), torch.matmul(x, y), sep='\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OOWwTBUE7yte",
        "outputId": "05f525ae-b746-4337-e52a-3f7bcab6a3cf"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[ 6,  6],\n",
            "        [14, 14]])\n",
            "tensor([[ 6,  6],\n",
            "        [14, 14]])\n",
            "tensor([[ 6,  6],\n",
            "        [14, 14]])\n",
            "tensor([[ 6,  6],\n",
            "        [14, 14]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(torch.dot(v, v), v.dot(v), sep='\\n')  # скалярное умножение\n",
        "print(torch.dot(x.view(-1), y.view(-1)), '\\n')\n",
        "print(torch.mv(x, v), x.mv(v), sep='\\n')  # умножение на вектор"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dOrVZefC8e93",
        "outputId": "0aeebdf3-d46c-4c99-cca8-e18294fa66c8"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(5)\n",
            "tensor(5)\n",
            "tensor(20) \n",
            "\n",
            "tensor([ 5, 11])\n",
            "tensor([ 5, 11])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x.type(torch.DoubleTensor).log()\n",
        "# приводится к типу Double - иначе не сработал бы log\n",
        "# в классе целых тензоров нет метода log"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PkNIbecJ9D_n",
        "outputId": "bf06d118-6cea-4bf7-d728-360373804c9e"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.0000, 0.6931],\n",
              "        [1.0986, 1.3863]], dtype=torch.float64)"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# возведение в степень\n",
        "print(x.pow(2), x**2, sep='\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B7gzvmjZ940c",
        "outputId": "f4ef75f6-39da-4600-8385-a60c818c968e"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[ 1,  4],\n",
            "        [ 9, 16]])\n",
            "tensor([[ 1,  4],\n",
            "        [ 9, 16]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# суммирование\n",
        "print(torch.sum(x),\n",
        "      x.sum(),\n",
        "      x.sum().item(),\n",
        "      x.sum(axis=0),\n",
        "      x.sum(axis=1),\n",
        "      sep='\\n')\n",
        "\n",
        "print(x.sum(axis=1, keepdim=True))  # keepdim - сохранение размерности"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Co7hTMF19_fS",
        "outputId": "4875fcda-b6f9-48d0-c15a-d3ca2e40f408"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(10)\n",
            "tensor(10)\n",
            "10\n",
            "tensor([4, 6])\n",
            "tensor([3, 7])\n",
            "tensor([[3],\n",
            "        [7]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# максимум\n",
        "print(torch.max(x),\n",
        "      x.max(),\n",
        "      x.max().item(),\n",
        "      x.max(axis=0),\n",
        "      x.max(axis=1),\n",
        "      sep='\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Hd83Em6q-7jd",
        "outputId": "4f33ca2e-33e0-49b1-fef6-d7dfa55a89d6"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(4)\n",
            "tensor(4)\n",
            "4\n",
            "torch.return_types.max(\n",
            "values=tensor([3, 4]),\n",
            "indices=tensor([1, 1]))\n",
            "torch.return_types.max(\n",
            "values=tensor([2, 4]),\n",
            "indices=tensor([1, 1]))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# top k элементов и их индексы\n",
        "# k-ый элемент и его индекс\n",
        "\n",
        "x = torch.tensor([2, 1, 2, 3, 0, 4, 3])\n",
        "print(x.topk(k=2), '\\n',\n",
        "      x.kthvalue(k=2))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W9rBf-DC_hIH",
        "outputId": "5bff80cf-1e15-4372-9a47-d2875728455b"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.return_types.topk(\n",
            "values=tensor([4, 3]),\n",
            "indices=tensor([5, 3])) \n",
            " torch.return_types.kthvalue(\n",
            "values=tensor(1),\n",
            "indices=tensor(1))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([1., 2., 3.])\n",
        "\n",
        "# стандартное отклонение и среднее\n",
        "# дисперсия и среднее\n",
        "print(torch.std_mean(x), '\\n',\n",
        "      torch.var_mean(x))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MlEGk66ADk6p",
        "outputId": "90a9eabc-1a62-4563-9775-bbd15074e274"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(tensor(1.), tensor(2.)) \n",
            " (tensor(1.), tensor(2.))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# не надо вручную приводить размеры, размножая тензор, когда:\n",
        "# (1) две размерности совпадают\n",
        "# (2) одна размерность равна 1\n",
        "\n",
        "x = torch.tensor([[1, 2], [3, 4]])\n",
        "y = torch.tensor([10, 20])\n",
        "\n",
        "x + y"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hVu8stIYD08S",
        "outputId": "354101f8-80d9-4f05-c932-d3aa2f27413d"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[11, 22],\n",
              "        [13, 24]])"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([[1, 2]])\n",
        "y = torch.tensor([[0], [3]])\n",
        "\n",
        "print(x, x.shape)\n",
        "print(y, y.shape)\n",
        "z = x + y\n",
        "print(z, z.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UVI5qLWqEWmA",
        "outputId": "ea8a7526-8248-4b4b-f2eb-90e3f31035b1"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1, 2]]) torch.Size([1, 2])\n",
            "tensor([[0],\n",
            "        [3]]) torch.Size([2, 1])\n",
            "tensor([[1, 2],\n",
            "        [4, 5]]) torch.Size([2, 2])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# при переводе в numpy необходимо, чтобы тензор находился\n",
        "# в CPU, а не на GPU\n",
        "\n",
        "x = torch.tensor([[1, 2], [3, 4]])\n",
        "print(x.numpy())\n",
        "y = x.cpu().detach().numpy()  # правильнее так\n",
        "print(y)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i0kYG4A-Eu3o",
        "outputId": "ca8b8f93-333d-4fef-a8bc-766f1bdd538d"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1 2]\n",
            " [3 4]]\n",
            "[[1 2]\n",
            " [3 4]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x[0, 0] = 10\n",
        "print(y)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wxaVFpW5FdhA",
        "outputId": "1c17a20b-d910-4731-b8fc-2bce51b832cf"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[10  2]\n",
            " [ 3  4]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# из numpy.array в pytorch.tensor\n",
        "\n",
        "import numpy as np\n",
        "torch.from_numpy(np.ones((2, 2)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l4H2WL3vFjni",
        "outputId": "69c2545c-12a7-47f4-e8ec-cce35352493d"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 1.],\n",
              "        [1., 1.]], dtype=torch.float64)"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# сохранение и загрузка тензоров\n",
        "torch.save(x, 'x-file')\n",
        "x2 = torch.load(\"x-file\")\n",
        "x2"
      ],
      "metadata": {
        "id": "p3HCeGHY6ZXS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4972bada-dd75-4f20-ef98-db2fe7bea509"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[10,  2],\n",
              "        [ 3,  4]])"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# А это универсальный способ, которым все пользуются при сохранении\n",
        "# Все параметры нейронки заносят в словарь и сохраняют\n",
        "\n",
        "mydict = {'x': x, 'y': y}\n",
        "torch.save(mydict, 'mydict')\n",
        "mydict2 = torch.load('mydict')\n",
        "mydict2"
      ],
      "metadata": {
        "id": "aK2NK5s36j8a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "094f2180-2167-4a0d-b584-d6187e35a333"
      },
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'x': tensor([[10,  2],\n",
              "         [ 3,  4]]),\n",
              " 'y': array([[10,  2],\n",
              "        [ 3,  4]])}"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# GPU или CPU\n",
        "# переменные и модели на разных устройствах не видят друг друга\n",
        "# их надо перенести на один вычислитель\n",
        "# на GPU вычисления производятся существенно быстрее из-за параллелизма\n",
        "\n",
        "x = x.cuda()\n",
        "x.is_cuda"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TpAoC3TaF68z",
        "outputId": "6e633e58-b5ba-4143-92f9-18ca5b77a57c"
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = x.cpu()\n",
        "x.is_cuda"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X5MMNl5vIChu",
        "outputId": "f132faaa-bfb9-4d15-cfa9-8ac8b7b196c8"
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "metadata": {},
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "z = z.to('cpu', torch.double)\n",
        "z.is_cuda"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MGyxMR3cIH3E",
        "outputId": "98081101-9972-4159-8333-0334b50d8b4f"
      },
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# определяем доступное для хранения тензоров/нейросетей устройство\n",
        "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
        "z.to(device)  # перенос на доступное устройство"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5fy201d1IfWI",
        "outputId": "b648bd1e-9ca2-4266-9ae2-fa906e58f63c"
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 2.],\n",
              "        [4., 5.]], device='cuda:0', dtype=torch.float64)"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# информация о GPU\n",
        "\n",
        "print('Using device:', device)\n",
        "print()\n",
        "\n",
        "if device.type == 'cuda':\n",
        "  print(torch.cuda.get_device_name(0))\n",
        "  print('Memory Usage:')\n",
        "  print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3, 1), 'GB')\n",
        "  print('Cached:', round(torch.cuda.memory_reserved(0)/1024**3, 1), 'GB')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fMniZaVCI9yf",
        "outputId": "68e6dbd8-20c8-41d9-d857-401f5d2f03bd"
      },
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using device: cuda:0\n",
            "\n",
            "Tesla T4\n",
            "Memory Usage:\n",
            "Allocated: 0.0 GB\n",
            "Cached: 0.0 GB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# чистка памяти\n",
        "\n",
        "import gc\n",
        "\n",
        "gc.collect()  # Питоновский\n",
        "\n",
        "# в таком контекстном менеджере\n",
        "with torch.no_grad():  # чистка памяти GPU работает с этой инструкцией\n",
        "  torch.cuda.empty_cache()"
      ],
      "metadata": {
        "id": "G8GiKYWvJuKZ"
      },
      "execution_count": 51,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.randn(5000, 5000)"
      ],
      "metadata": {
        "id": "6cMZWUtrJ62j"
      },
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# CPU\n",
        "import datetime\n",
        "t1 = datetime.datetime.now()\n",
        "_ = torch.matmul(x, x)\n",
        "t2 = datetime.datetime.now()\n",
        "print(f'CPU time: {t2-t1} s')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dTdH4hDdKt-K",
        "outputId": "a0cee125-fbac-477d-fcd5-05ec29e90836"
      },
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU time: 0:00:03.665134 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# CPU2GPU\n",
        "t1 = datetime.datetime.now()\n",
        "x = x.to('cuda:0')\n",
        "t2 = datetime.datetime.now()\n",
        "print(f'CPU2GPU time: {t2-t1} s')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p_hvAaOWKzS9",
        "outputId": "19c1e61c-41cd-491d-8d26-956c960bf56d"
      },
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU2GPU time: 0:00:00.024019 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# GPU\n",
        "t1 = datetime.datetime.now()\n",
        "x = x + 0.0  # просто первая операция (без неё иногда неадекватно работает)\n",
        "_ = torch.matmul(x, x)\n",
        "t2 = datetime.datetime.now()\n",
        "print(f'GPU time: {t2-t1} s')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1NwrPxcQK9LQ",
        "outputId": "082b1e3f-fe37-42ba-a720-288ef87dde25"
      },
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GPU time: 0:00:00.597797 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def set_seed(seed):\n",
        "  np.random.seed(seed)\n",
        "  torch.manual_seed(seed)\n",
        "  if torch.cuda.is_available():  # для GPU отдельный seed\n",
        "    torch.cuda.manual_seed(seed)\n",
        "    torch.cuda.manual_seed_all(seed)\n",
        "\n",
        "set_seed(42)\n",
        "\n",
        "# есть стохастические операции на GPU\n",
        "# сделаем их детерминированными для воспроизводимости\n",
        "torch.backends.cudnn.deterministic = True\n",
        "torch.backends.cudnn.benchmark = False"
      ],
      "metadata": {
        "id": "qC-ZpEu2LBcB"
      },
      "execution_count": 56,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Автодифференцирование (AutoGrad)\n",
        "\n",
        "backward -- функция обратного прохода; именно при её вызове автоматически считаются производные по всем переменным, у которых requires_grad=True\n",
        "\n",
        "inplace-операции не работают в графе\n",
        "\n",
        "Сначала приведём пример автоматического дифференцирования, мы зададим функцию y=sin(x)*(sin^2(x)+cos^2(x)), а Pytorch автоматически вычислит производную"
      ],
      "metadata": {
        "id": "zeciqirE-khN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.autograd import Variable\n",
        "\n",
        "x = torch.linspace(-2, 2, 101, dtype=torch.float32, requires_grad=True)\n",
        "# сейчас в Pytorch-е переменной является любой тензор, у которого\n",
        "# requires_grad=True, поэтому следующая строка закомментирована\n",
        "# x = Variable(x, requires_grad=True)\n",
        "y = torch.sin(x) * (torch.sin(x) ** 2 + torch.cos(x) ** 2)  # это прямой проход\n",
        "# в дальнейшем часто будем определять класс с методом y.forward()\n",
        "\n",
        "y.sum().backward()  # превращаем в число (только от таких функций берётся\n",
        "# градиент) и делаем backward\n",
        "g = x.grad  # взятие производной в каждой точке\n",
        "g"
      ],
      "metadata": {
        "id": "dUqy3XsXNQ_v",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b588ed1a-5a1a-4317-8c2b-958faed1d702"
      },
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([-0.4161, -0.3795, -0.3421, -0.3043, -0.2660, -0.2272, -0.1881, -0.1487,\n",
              "        -0.1090, -0.0691, -0.0292,  0.0108,  0.0508,  0.0907,  0.1304,  0.1700,\n",
              "         0.2092,  0.2482,  0.2867,  0.3248,  0.3624,  0.3993,  0.4357,  0.4713,\n",
              "         0.5062,  0.5403,  0.5735,  0.6058,  0.6372,  0.6675,  0.6967,  0.7248,\n",
              "         0.7518,  0.7776,  0.8021,  0.8253,  0.8473,  0.8678,  0.8870,  0.9048,\n",
              "         0.9211,  0.9359,  0.9492,  0.9611,  0.9713,  0.9801,  0.9872,  0.9928,\n",
              "         0.9968,  0.9992,  1.0000,  0.9992,  0.9968,  0.9928,  0.9872,  0.9801,\n",
              "         0.9713,  0.9611,  0.9492,  0.9359,  0.9211,  0.9048,  0.8870,  0.8678,\n",
              "         0.8473,  0.8253,  0.8021,  0.7776,  0.7518,  0.7248,  0.6967,  0.6675,\n",
              "         0.6372,  0.6058,  0.5735,  0.5403,  0.5062,  0.4713,  0.4357,  0.3993,\n",
              "         0.3624,  0.3248,  0.2867,  0.2482,  0.2092,  0.1700,  0.1304,  0.0907,\n",
              "         0.0508,  0.0108, -0.0292, -0.0691, -0.1090, -0.1487, -0.1881, -0.2272,\n",
              "        -0.2660, -0.3043, -0.3421, -0.3795, -0.4161])"
            ]
          },
          "metadata": {},
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# над целочисленными тензорами нельзя взять производную\n",
        "x = torch.linspace(-2, 2, 3, dtype=torch.float32)\n",
        "x.requires_grad_()  # указываем, что хотим вычислять производную\n",
        "x"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j-utT6xGWdhH",
        "outputId": "4988ea30-7f2d-4a1a-ff17-2e96022d2ba0"
      },
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([-2.,  0.,  2.], requires_grad=True)"
            ]
          },
          "metadata": {},
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([1.], requires_grad=True)\n",
        "y = torch.tensor([2.], requires_grad=True)\n",
        "z = torch.tensor([3.], requires_grad=True)\n",
        "\n",
        "f = (x + y) * (y - z)  # прямой проход\n",
        "\n",
        "f.backward()  # обратный проход\n",
        "# производные по x, по y и по z\n",
        "print((x.grad, y.grad, z.grad))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C9iGMT6DW8h4",
        "outputId": "f400355f-e34d-4ba5-902f-b77cf74e3487"
      },
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(tensor([-1.]), tensor([2.]), tensor([-3.]))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip3 install torchviz"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X66qH4NxY1H6",
        "outputId": "12effb19-3ac4-4008-e11c-b649d56e2fbe"
      },
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: torchviz in /usr/local/lib/python3.10/dist-packages (0.0.2)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from torchviz) (2.0.1+cu118)\n",
            "Requirement already satisfied: graphviz in /usr/local/lib/python3.10/dist-packages (from torchviz) (0.20.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (3.12.2)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (4.7.1)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (3.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch->torchviz) (2.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch->torchviz) (3.27.2)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch->torchviz) (16.0.6)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->torchviz) (2.1.3)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->torchviz) (1.3.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# визуализация графа вычислений (сумма, разность и перемножение)\n",
        "from torchviz import make_dot\n",
        "make_dot(f)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 383
        },
        "id": "32AYvtSKYqDz",
        "outputId": "7a0730d3-f9f7-4f6e-cb2e-6ef301641b9e"
      },
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.43.0 (0)\n -->\n<!-- Title: %3 Pages: 1 -->\n<svg width=\"347pt\" height=\"271pt\"\n viewBox=\"0.00 0.00 347.00 271.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 267)\">\n<title>%3</title>\n<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-267 343,-267 343,4 -4,4\"/>\n<!-- 139353516433680 -->\n<g id=\"node1\" class=\"node\">\n<title>139353516433680</title>\n<polygon fill=\"#caff70\" stroke=\"black\" points=\"195.5,-31 141.5,-31 141.5,0 195.5,0 195.5,-31\"/>\n<text text-anchor=\"middle\" x=\"168.5\" y=\"-7\" font-family=\"monospace\" font-size=\"10.00\"> (1)</text>\n</g>\n<!-- 139357197176016 -->\n<g id=\"node2\" class=\"node\">\n<title>139357197176016</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"213,-86 124,-86 124,-67 213,-67 213,-86\"/>\n<text text-anchor=\"middle\" x=\"168.5\" y=\"-74\" font-family=\"monospace\" font-size=\"10.00\">MulBackward0</text>\n</g>\n<!-- 139357197176016&#45;&gt;139353516433680 -->\n<g id=\"edge10\" class=\"edge\">\n<title>139357197176016&#45;&gt;139353516433680</title>\n<path fill=\"none\" stroke=\"black\" d=\"M168.5,-66.79C168.5,-60.07 168.5,-50.4 168.5,-41.34\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"172,-41.19 168.5,-31.19 165,-41.19 172,-41.19\"/>\n</g>\n<!-- 139357197188736 -->\n<g id=\"node3\" class=\"node\">\n<title>139357197188736</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"160,-141 71,-141 71,-122 160,-122 160,-141\"/>\n<text text-anchor=\"middle\" x=\"115.5\" y=\"-129\" font-family=\"monospace\" font-size=\"10.00\">AddBackward0</text>\n</g>\n<!-- 139357197188736&#45;&gt;139357197176016 -->\n<g id=\"edge1\" class=\"edge\">\n<title>139357197188736&#45;&gt;139357197176016</title>\n<path fill=\"none\" stroke=\"black\" d=\"M124.25,-121.75C131.97,-114.03 143.4,-102.6 152.72,-93.28\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"155.31,-95.64 159.91,-86.09 150.36,-90.69 155.31,-95.64\"/>\n</g>\n<!-- 139357197187920 -->\n<g id=\"node4\" class=\"node\">\n<title>139357197187920</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"101,-196 0,-196 0,-177 101,-177 101,-196\"/>\n<text text-anchor=\"middle\" x=\"50.5\" y=\"-184\" font-family=\"monospace\" font-size=\"10.00\">AccumulateGrad</text>\n</g>\n<!-- 139357197187920&#45;&gt;139357197188736 -->\n<g id=\"edge2\" class=\"edge\">\n<title>139357197187920&#45;&gt;139357197188736</title>\n<path fill=\"none\" stroke=\"black\" d=\"M60.94,-176.98C70.65,-169.07 85.32,-157.11 97,-147.58\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"99.33,-150.2 104.87,-141.17 94.9,-144.78 99.33,-150.2\"/>\n</g>\n<!-- 139353516440480 -->\n<g id=\"node5\" class=\"node\">\n<title>139353516440480</title>\n<polygon fill=\"lightblue\" stroke=\"black\" points=\"77.5,-263 23.5,-263 23.5,-232 77.5,-232 77.5,-263\"/>\n<text text-anchor=\"middle\" x=\"50.5\" y=\"-239\" font-family=\"monospace\" font-size=\"10.00\"> (1)</text>\n</g>\n<!-- 139353516440480&#45;&gt;139357197187920 -->\n<g id=\"edge3\" class=\"edge\">\n<title>139353516440480&#45;&gt;139357197187920</title>\n<path fill=\"none\" stroke=\"black\" d=\"M50.5,-231.92C50.5,-224.22 50.5,-214.69 50.5,-206.43\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"54,-206.25 50.5,-196.25 47,-206.25 54,-206.25\"/>\n</g>\n<!-- 139357197186000 -->\n<g id=\"node6\" class=\"node\">\n<title>139357197186000</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"220,-196 119,-196 119,-177 220,-177 220,-196\"/>\n<text text-anchor=\"middle\" x=\"169.5\" y=\"-184\" font-family=\"monospace\" font-size=\"10.00\">AccumulateGrad</text>\n</g>\n<!-- 139357197186000&#45;&gt;139357197188736 -->\n<g id=\"edge4\" class=\"edge\">\n<title>139357197186000&#45;&gt;139357197188736</title>\n<path fill=\"none\" stroke=\"black\" d=\"M160.58,-176.75C152.72,-169.03 141.07,-157.6 131.58,-148.28\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"133.84,-145.6 124.25,-141.09 128.94,-150.59 133.84,-145.6\"/>\n</g>\n<!-- 139357197185472 -->\n<g id=\"node8\" class=\"node\">\n<title>139357197185472</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"267,-141 178,-141 178,-122 267,-122 267,-141\"/>\n<text text-anchor=\"middle\" x=\"222.5\" y=\"-129\" font-family=\"monospace\" font-size=\"10.00\">SubBackward0</text>\n</g>\n<!-- 139357197186000&#45;&gt;139357197185472 -->\n<g id=\"edge7\" class=\"edge\">\n<title>139357197186000&#45;&gt;139357197185472</title>\n<path fill=\"none\" stroke=\"black\" d=\"M178.25,-176.75C185.97,-169.03 197.4,-157.6 206.72,-148.28\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"209.31,-150.64 213.91,-141.09 204.36,-145.69 209.31,-150.64\"/>\n</g>\n<!-- 139353516436960 -->\n<g id=\"node7\" class=\"node\">\n<title>139353516436960</title>\n<polygon fill=\"lightblue\" stroke=\"black\" points=\"196.5,-263 142.5,-263 142.5,-232 196.5,-232 196.5,-263\"/>\n<text text-anchor=\"middle\" x=\"169.5\" y=\"-239\" font-family=\"monospace\" font-size=\"10.00\"> (1)</text>\n</g>\n<!-- 139353516436960&#45;&gt;139357197186000 -->\n<g id=\"edge5\" class=\"edge\">\n<title>139353516436960&#45;&gt;139357197186000</title>\n<path fill=\"none\" stroke=\"black\" d=\"M169.5,-231.92C169.5,-224.22 169.5,-214.69 169.5,-206.43\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"173,-206.25 169.5,-196.25 166,-206.25 173,-206.25\"/>\n</g>\n<!-- 139357197185472&#45;&gt;139357197176016 -->\n<g id=\"edge6\" class=\"edge\">\n<title>139357197185472&#45;&gt;139357197176016</title>\n<path fill=\"none\" stroke=\"black\" d=\"M213.58,-121.75C205.72,-114.03 194.07,-102.6 184.58,-93.28\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"186.84,-90.6 177.25,-86.09 181.94,-95.59 186.84,-90.6\"/>\n</g>\n<!-- 139357197187056 -->\n<g id=\"node9\" class=\"node\">\n<title>139357197187056</title>\n<polygon fill=\"lightgrey\" stroke=\"black\" points=\"339,-196 238,-196 238,-177 339,-177 339,-196\"/>\n<text text-anchor=\"middle\" x=\"288.5\" y=\"-184\" font-family=\"monospace\" font-size=\"10.00\">AccumulateGrad</text>\n</g>\n<!-- 139357197187056&#45;&gt;139357197185472 -->\n<g id=\"edge8\" class=\"edge\">\n<title>139357197187056&#45;&gt;139357197185472</title>\n<path fill=\"none\" stroke=\"black\" d=\"M277.9,-176.98C268.04,-169.07 253.15,-157.11 241.28,-147.58\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"243.29,-144.7 233.3,-141.17 238.9,-150.16 243.29,-144.7\"/>\n</g>\n<!-- 139353516438640 -->\n<g id=\"node10\" class=\"node\">\n<title>139353516438640</title>\n<polygon fill=\"lightblue\" stroke=\"black\" points=\"315.5,-263 261.5,-263 261.5,-232 315.5,-232 315.5,-263\"/>\n<text text-anchor=\"middle\" x=\"288.5\" y=\"-239\" font-family=\"monospace\" font-size=\"10.00\"> (1)</text>\n</g>\n<!-- 139353516438640&#45;&gt;139357197187056 -->\n<g id=\"edge9\" class=\"edge\">\n<title>139353516438640&#45;&gt;139357197187056</title>\n<path fill=\"none\" stroke=\"black\" d=\"M288.5,-231.92C288.5,-224.22 288.5,-214.69 288.5,-206.43\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"292,-206.25 288.5,-196.25 285,-206.25 292,-206.25\"/>\n</g>\n</g>\n</svg>\n",
            "text/plain": [
              "<graphviz.graphs.Digraph at 0x7ebea176a650>"
            ]
          },
          "metadata": {},
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "f = (x + y) * (y - z)\n",
        "z = f.detach()\n",
        "print(f.requires_grad, z.requires_grad)\n",
        "\n",
        "# когда просто нужен прямой проход - и не надо считать градиенты\n",
        "# делаем в таком контекстном менеджере\n",
        "with torch.no_grad():\n",
        "  f = (x + y) * (y - z)\n",
        "print(f.requires_grad)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xTQRho99zeK0",
        "outputId": "e707e595-5d5f-477a-c8c9-635a28c729d9"
      },
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "True False\n",
            "False\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([1., 2, 3])\n",
        "w = torch.tensor([1., 1, 1], requires_grad=True)\n",
        "z = w @ x\n",
        "z.backward()\n",
        "print((x.grad, w.grad))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EiaJfqwHYy7A",
        "outputId": "369f925d-0e0f-465f-f325-0f211b3933ae"
      },
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(None, tensor([1., 2., 3.]))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "z = w @ x\n",
        "z.backward()\n",
        "# видим, что идёт накопление градиентов при обратном проходе\n",
        "# для каждого тензора как бы хранится некий счётчик градиентов\n",
        "print((x.grad, w.grad))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bYJkOKFidH1w",
        "outputId": "0ed704c3-fec5-4f12-b75f-859b718cb7b8"
      },
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(None, tensor([2., 4., 6.]))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "with torch.no_grad():  # просто прямой проход (нет накопления)\n",
        "  z = w @ x\n",
        "  #z.backward()\n",
        "print((x.grad, w.grad))\n",
        "\n",
        "w.grad.data.zero_()  # обнуление счётчика градиентов\n",
        "z = w @ x\n",
        "z.backward()\n",
        "print((x.grad, w.grad))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fPqvwxbCy1DH",
        "outputId": "e077f555-2ff1-4185-cb8f-30e2b7841d7e"
      },
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(None, tensor([2., 4., 6.]))\n",
            "(None, tensor([1., 2., 3.]))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# w.numpy() - будет ошибка\n",
        "# (запрещено переводить в numpy, если requires_grad=True)\n",
        "w.detach().numpy()  # создаётся копия, которую можно в numpy\n",
        "# у неё (после detach) requires_grad=False"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mWVdjjJszP6w",
        "outputId": "a7c0db04-2f0b-4c74-ce1a-e6fea34a3c96"
      },
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1., 1., 1.], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Иллюстрация взятия градиента - с detach\n",
        "x = torch.tensor([2.], requires_grad=True)\n",
        "\n",
        "y = x * x\n",
        "y.detach_()  # отсоединяем от графа вычислений (теперь это просто\n",
        "# константная матрица)\n",
        "z = x * y\n",
        "z.backward()\n",
        "\n",
        "print(x.grad)  # (2*2*x)' = 4"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KY08Xuqtzsvk",
        "outputId": "bde99ef1-d6a8-417b-fc1a-0ff6c03c5adc"
      },
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([4.])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Ещё одна иллюстрация detach()\n",
        "x = torch.tensor([2.], requires_grad=True)\n",
        "print(x)\n",
        "print(x.detach())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v3mhHVJt14_1",
        "outputId": "b476aad4-6377-4335-b5e2-f600234491ce"
      },
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([2.], requires_grad=True)\n",
            "tensor([2.])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# динамический граф вычислений в цикле\n",
        "x = torch.tensor([[1, 2], [3, 4]], requires_grad=True, dtype=torch.float)\n",
        "x0 = x\n",
        "for _ in range(2):\n",
        "  x = x * x  # x превращается во внутреннюю вершину графа вычислений\n",
        "\n",
        "z = x.mean()  # здесь будет 1/4 !\n",
        "z.backward()\n",
        "print(x, x.grad, sep='\\n')\n",
        "print(x0, x0.grad, sep='\\n')  # градиент лежит здесь\n",
        "# а в x.grad не лежит, так как x превратился во внутреннюю вершину графа\n",
        "# вычислений"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LfA7Tm1F25tZ",
        "outputId": "80274faa-07e4-4d82-9a28-672414bdf4f8"
      },
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[  1.,  16.],\n",
            "        [ 81., 256.]], grad_fn=<MulBackward0>)\n",
            "None\n",
            "tensor([[1., 2.],\n",
            "        [3., 4.]], requires_grad=True)\n",
            "tensor([[ 1.,  8.],\n",
            "        [27., 64.]])\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-69-a21748d1d555>:9: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:486.)\n",
            "  print(x, x.grad, sep='\\n')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor([[1., 2, 3, 4]], requires_grad=True)\n",
        "z = x ** 2 / 2\n",
        "loss = z.sum(dim=1)\n",
        "\n",
        "# возможно вычислять отдельные производные по компонентам\n",
        "# backward для первого элемента z\n",
        "z.backward(torch.FloatTensor([[1, 0, 0, 0]]), retain_graph=True)\n",
        "print(x.grad.data)\n",
        "x.grad.data.zero_()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nJwggIch6Mbe",
        "outputId": "50d6ad04-f9fa-4288-99b4-eefa9a0c815e"
      },
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1., 0., 0., 0.]])\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0., 0., 0., 0.]])"
            ]
          },
          "metadata": {},
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# backward с равными весами для всех элементов z\n",
        "z.backward(torch.FloatTensor([[1, 1, 1, 1]]), retain_graph=True)\n",
        "print(x.grad.data)\n",
        "x.grad.data.zero_()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JTGGHUzn_RLW",
        "outputId": "e8aae0f9-0a64-47f5-cba4-4f5ab5c5b985"
      },
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1., 2., 3., 4.]])\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0., 0., 0., 0.]])"
            ]
          },
          "metadata": {},
          "execution_count": 71
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# обычный backward для loss\n",
        "loss.backward()  # ~ loss.backward(torch.FloatTensor([1.0]))\n",
        "print(x.grad.data)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q1VmByD8_xsm",
        "outputId": "1185dc91-4d72-41a9-b11a-a87f694dbdd6"
      },
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1., 2., 3., 4.]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "a = torch.rand(1, 4, requires_grad=True)\n",
        "b = a**2\n",
        "c = b * 2\n",
        "d = c.mean()\n",
        "e = c.sum()\n",
        "\n",
        "# если так\n",
        "d.backward(retain_graph=True)\n",
        "# тут будет ошибка (RuntimeError: Trying to backward through the graph a\n",
        "# second time...), если не добавлять retain_graph=True\n",
        "e.backward(retain_graph=True)\n",
        "# градиенты будут суммироваться"
      ],
      "metadata": {
        "id": "xDCiLNEb_3Mq"
      },
      "execution_count": 73,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Предобработка данных, нейросети\n",
        "\n",
        "TensorDataset -- для представления датасета.\n",
        "Часто приходится определять свой датасет, наследуя этот класс.\n",
        "\n",
        "DataLoader -- подаёт батчами данные, позволяет итерироваться по датасету, автоматически формируя батчи (и делая некоторые сопутствующие действия).\n",
        "\n",
        "В DataLoader может передаваться сэмплер."
      ],
      "metadata": {
        "id": "-bqfR8XFPPq5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.utils.data import TensorDataset, DataLoader\n",
        "\n",
        "x = torch.from_numpy(np.vstack([np.arange(10, dtype='float32'),\n",
        "                                np.ones(10, dtype='float32')]).T)\n",
        "y = torch.from_numpy(np.arange(10, dtype='float32')[:, np.newaxis] ** 2)\n",
        "\n",
        "train_ds = TensorDataset(x, y)\n",
        "train_dl = DataLoader(train_ds, batch_size=4, shuffle=True)\n",
        "# хоть и batch_size=4 всё равно последний может быть более короткий\n",
        "#дополнительным ключевым словом можно запретить использовать этот короткий батч\n",
        "\n",
        "for xb, yb in train_dl:\n",
        "  print(xb)\n",
        "  print(yb)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8tptHCMwR9IA",
        "outputId": "9c6bbce2-4ed8-4d6a-9c8a-62092635fb64"
      },
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[4., 1.],\n",
            "        [6., 1.],\n",
            "        [9., 1.],\n",
            "        [3., 1.]])\n",
            "tensor([[16.],\n",
            "        [36.],\n",
            "        [81.],\n",
            "        [ 9.]])\n",
            "tensor([[7., 1.],\n",
            "        [5., 1.],\n",
            "        [2., 1.],\n",
            "        [0., 1.]])\n",
            "tensor([[49.],\n",
            "        [25.],\n",
            "        [ 4.],\n",
            "        [ 0.]])\n",
            "tensor([[1., 1.],\n",
            "        [8., 1.]])\n",
            "tensor([[ 1.],\n",
            "        [64.]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# есть много вспомогательных полезных функций\n",
        "# например, можно разделить датасет на части\n",
        "\n",
        "train_size = int(0.8 * len(train_ds))\n",
        "test_size = len(train_ds) - train_size\n",
        "train_dataset, test_dataset = torch.utils.data.random_split(train_ds,[train_size, test_size])"
      ],
      "metadata": {
        "id": "JokrNRDcUvaS"
      },
      "execution_count": 75,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# как передать список в нейронку?\n",
        "\n",
        "# список массивов numpy\n",
        "my_x = [np.array([[1., 2], [3, 4]]), np.array([[5., 6], [7, 8]])]\n",
        "# ещё один список массивов numpy (целевые значения)\n",
        "my_y = [np.array([4.]), np.array([2.])]\n",
        "\n",
        "# преобразуем в torch.Tensor\n",
        "tensor_x = torch.tensor(my_x)\n",
        "tensor_y = torch.tensor(my_y)\n",
        "\n",
        "# создаём датасет\n",
        "my_dataset = TensorDataset(tensor_x, tensor_y)\n",
        "# создаём dataloader\n",
        "my_dataloader = DataLoader(my_dataset)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OOFERn6lWESB",
        "outputId": "c6d73682-269b-43aa-81dd-790b6b1d74ac"
      },
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-76-7516ba1365ff>:9: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:245.)\n",
            "  tensor_x = torch.tensor(my_x)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# загрузка готовых датасетов (например, MNIST) с трансформациями\n",
        "from torchvision import datasets, transforms\n",
        "\n",
        "train_loader = torch.utils.data.DataLoader(\n",
        "    datasets.MNIST('../data',\n",
        "                   train=True,\n",
        "                   download=True,\n",
        "                   transform=transforms.Compose([\n",
        "                      transforms.ColorJitter(brightness=(0.6, 1),\n",
        "                                             contrast=(0.8,1)),\n",
        "                      transforms.RandomRotation((-15, 15)),\n",
        "                      # перевод в тензор + нормировка на отрезок\n",
        "                      transforms.ToTensor(),\n",
        "                      transforms.Normalize((0.1307,), (0.3081,))\n",
        "                   ])),\n",
        "    batch_size=64, shuffle=True, num_workers=2, pin_memory=True)\n",
        "\n",
        "# с num_workers лучше поэкспериментировать!\n",
        "# Это число подпроцессов для загрузки данных\n",
        "# Чаще pin_memory=True, если используем GPU. Предварительно копирует данные\n",
        "# на GPU"
      ],
      "metadata": {
        "id": "Ij_Q7rkoan76"
      },
      "execution_count": 77,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torchvision\n",
        "\n",
        "# трансформации можно и нужно делать разные для обучения и теста (да или нет?)\n",
        "# и тест не перемешиваем\n",
        "\n",
        "trainset = torchvision.datasets.CIFAR10(root='./data',\n",
        "                                        train=True,\n",
        "                                        download=True,\n",
        "                                        transform=transforms.ToTensor())\n",
        "\n",
        "testset = torchvision.datasets.CIFAR10(root='./data',\n",
        "                                       train=False,\n",
        "                                       download=True,\n",
        "                                       transform=transforms.ToTensor())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NJ_BocuFcpBM",
        "outputId": "74780717-7d12-4953-f2e5-9cb188c98248"
      },
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# своя трансформация\n",
        "\n",
        "class Noise():\n",
        "  \"\"\"\n",
        "  Add Gaussian Noise to a tensor\n",
        "  \"\"\"\n",
        "  def __init__(self, mean, stddev):\n",
        "    self.mean = mean\n",
        "    self.stddev = stddev\n",
        "\n",
        "  def __call__(self, tensor):\n",
        "    # прибавление Гауссовского шума к текущему тензору\n",
        "    noise = torch.zeros_like(tensor).normal_(self.mean, self.stddev)\n",
        "    return tensor.add_(noise)\n",
        "\n",
        "  def __repr__(self):\n",
        "    repr = f'{self.__class__.__name__} (mean={self.mean}, stddev={self.stddev})'\n",
        "    return repr"
      ],
      "metadata": {
        "id": "pin8sBcterbh"
      },
      "execution_count": 79,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# создание своего датасета\n",
        "\n",
        "class CustomTextDataset(TensorDataset):\n",
        "  \"\"\"\n",
        "  Simple Dataset initializes with X and y vectors\n",
        "  \"\"\"\n",
        "  def __init__(self, X, y=None):\n",
        "    self.data = list(zip(X, y))\n",
        "    # Сортировка по длине нулевого элемента в кортеже\n",
        "    self.data = sorted(self.data, key=lambda x: len(x[0]))\n",
        "\n",
        "  def __len__(self):\n",
        "    #raise NotImplementedError\n",
        "    return len(self.data)\n",
        "\n",
        "  def __getitem__(self, idx):\n",
        "    return self.data[idx]"
      ],
      "metadata": {
        "id": "3m-k5VtyfxHK"
      },
      "execution_count": 80,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from torch import nn\n",
        "\n",
        "# простейший линейный слой\n",
        "model = nn.Linear(in_features=2, out_features=1, bias=True)\n",
        "print(model.weight)\n",
        "print(model.bias)\n",
        "list(model.parameters())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_tvXo6CSvgqc",
        "outputId": "b46499d9-cbc6-41f2-c332-f3f3315614a0"
      },
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Parameter containing:\n",
            "tensor([[ 0.1045, -0.3301]], requires_grad=True)\n",
            "Parameter containing:\n",
            "tensor([0.1802], requires_grad=True)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Parameter containing:\n",
              " tensor([[ 0.1045, -0.3301]], requires_grad=True),\n",
              " Parameter containing:\n",
              " tensor([0.1802], requires_grad=True)]"
            ]
          },
          "metadata": {},
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# простейшее задание нейронной сети\n",
        "net = nn.Sequential(nn.Linear(10, 5),\n",
        "                     nn.ReLU(),\n",
        "                     nn.Linear(5, 2))"
      ],
      "metadata": {
        "id": "fc6k6Y-owrg-"
      },
      "execution_count": 82,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from collections import OrderedDict\n",
        "\n",
        "# с удобными именами с помощью OrderedDict\n",
        "net1 = nn.Sequential(OrderedDict([('hidden_linear', nn.Linear(10, 5)),\n",
        "                                  ('hidden_activation', nn.ReLU()),\n",
        "                                  ('output', nn.Linear(5, 2))]))\n",
        "X = torch.rand(3, 10)\n",
        "\n",
        "# когда делаем прямой проход, то явно не вызываем метод forward, а используем\n",
        "# имя нейросети: net(X)\n",
        "# так как до и после вызова неявно ещё кое-что выполняется,\n",
        "# ex: _forward_pre_hooks\n",
        "net(X)\n",
        "# батч из трёх десятимерных объектов прошёл через нейросеть\n",
        "# на выходе 3 двухмерных объекта"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ovw8Yfo4w8WF",
        "outputId": "90397123-fe98-4c91-d478-00d1d081769f"
      },
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[-0.1452,  0.2723],\n",
              "        [-0.0796,  0.3274],\n",
              "        [-0.0342,  0.3743]], grad_fn=<AddmmBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import torch.nn.functional as F\n",
        "\n",
        "# простейшее задание нейронной сети в виде класса\n",
        "class MLP(nn.Module):\n",
        "  def __init__(self):\n",
        "    # инициализация параметров\n",
        "    # обращение к инициализации родителя\n",
        "    super().__init__()  #super(MLP, self).__init__()\n",
        "    self.hidden = nn.Linear(10, 5)  # скрытый слой\n",
        "    self.out = nn.Linear(5, 2)  # выходной слой\n",
        "\n",
        "  def forward(self, X):\n",
        "    # как обрабатываются данные и получается ответ\n",
        "    return self.out(F.relu(self.hidden(X)))\n",
        "\n",
        "net2 = MLP()"
      ],
      "metadata": {
        "id": "cm0JaREixcnR"
      },
      "execution_count": 84,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# простейшее задание нейронной сети в виде класса\n",
        "# с присоединением модулей с помощью .add_module()\n",
        "# используется, когда хотим совсем немного модифицировать уже\n",
        "# имеющуюся нейросеть\n",
        "class NNN(torch.nn.Module):\n",
        "  def __init__(self):\n",
        "    super(NNN, self).__init__()\n",
        "\n",
        "    self.layers = torch.nn.Sequential()\n",
        "    self.layers.add_module('lin1', torch.nn.Linear(10, 5))\n",
        "    self.layers.add_module('relu1', torch.nn.ReLU())\n",
        "    self.layers.add_module('lin2', torch.nn.Linear(5, 2))\n",
        "\n",
        "  def forward(self, input):\n",
        "    return self.layers(input)\n",
        "\n",
        "net3 = NNN()"
      ],
      "metadata": {
        "id": "nUfGRie0yltE"
      },
      "execution_count": 85,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "net"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3okV00Yoz0s0",
        "outputId": "792b7481-105f-43c4-8db1-b46fb6a01474"
      },
      "execution_count": 86,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Sequential(\n",
              "  (0): Linear(in_features=10, out_features=5, bias=True)\n",
              "  (1): ReLU()\n",
              "  (2): Linear(in_features=5, out_features=2, bias=True)\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "net2"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TtlGokyW0lya",
        "outputId": "85e19965-d5f2-4bf1-d843-d9eba3354b1a"
      },
      "execution_count": 87,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "MLP(\n",
              "  (hidden): Linear(in_features=10, out_features=5, bias=True)\n",
              "  (out): Linear(in_features=5, out_features=2, bias=True)\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "net3"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DHATHVfa0nFu",
        "outputId": "e71d8241-3f57-4b99-a612-60c05cd32f34"
      },
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NNN(\n",
              "  (layers): Sequential(\n",
              "    (lin1): Linear(in_features=10, out_features=5, bias=True)\n",
              "    (relu1): ReLU()\n",
              "    (lin2): Linear(in_features=5, out_features=2, bias=True)\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 88
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# доступ к слоям и параметрам нейросети\n",
        "net[0], net2.hidden, net3.layers.relu1"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1FN-bBPy0oOV",
        "outputId": "d2970553-114e-4693-d6c0-1f77616c7254"
      },
      "execution_count": 89,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(Linear(in_features=10, out_features=5, bias=True),\n",
              " Linear(in_features=10, out_features=5, bias=True),\n",
              " ReLU())"
            ]
          },
          "metadata": {},
          "execution_count": 89
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "net.state_dict()  # все параметры сети\n",
        "# словарь параметров сети, в котором хранится текущее состояние сети\n",
        "# его нужно сохранить, если хотим сохранить текущее состояние"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YQEz4Ej61dNP",
        "outputId": "f967d27c-4c0a-4c28-913e-76928c7e3c8b"
      },
      "execution_count": 90,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "OrderedDict([('0.weight',\n",
              "              tensor([[-0.1457, -0.0371, -0.1284,  0.2098, -0.2496, -0.1458, -0.0893, -0.1901,\n",
              "                        0.0298, -0.3123],\n",
              "                      [ 0.2856, -0.2686,  0.2441,  0.0526, -0.1027,  0.1954,  0.0493,  0.2555,\n",
              "                        0.0346, -0.0997],\n",
              "                      [ 0.0850, -0.0858,  0.1331,  0.2823,  0.1828, -0.1382,  0.1825,  0.0566,\n",
              "                        0.1606, -0.1927],\n",
              "                      [-0.3130, -0.1222, -0.2426,  0.2595,  0.0911,  0.1310,  0.1000, -0.0055,\n",
              "                        0.2475, -0.2247],\n",
              "                      [ 0.0199, -0.2158,  0.0975, -0.1089,  0.0969, -0.0659,  0.2623, -0.1874,\n",
              "                       -0.1886, -0.1886]])),\n",
              "             ('0.bias', tensor([ 0.2844,  0.1054,  0.3043, -0.2610, -0.3137])),\n",
              "             ('2.weight',\n",
              "              tensor([[-0.3499, -0.3008,  0.1811,  0.1601,  0.3716],\n",
              "                      [-0.2310, -0.3049,  0.2373, -0.1808,  0.2714]])),\n",
              "             ('2.bias', tensor([-0.1061,  0.2558]))])"
            ]
          },
          "metadata": {},
          "execution_count": 90
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "net[0].bias.data, net2.hidden.bias.data"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FZktmyD01kSQ",
        "outputId": "966259af-1c96-4fd3-ecad-7cdeebcc1ae4"
      },
      "execution_count": 91,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([ 0.2844,  0.1054,  0.3043, -0.2610, -0.3137]),\n",
              " tensor([ 0.1657,  0.0800, -0.0031, -0.2405, -0.2709]))"
            ]
          },
          "metadata": {},
          "execution_count": 91
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "net[0].bias.grad, net2.hidden.bias.grad\n",
        "# не было обучения (Back Propagation), поэтому None"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TVw1UtQp16d1",
        "outputId": "54538984-6cbc-40d8-fd13-dc6f4ee3ce69"
      },
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(None, None)"
            ]
          },
          "metadata": {},
          "execution_count": 92
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# считаем число параметров (меняющихся при обучении)\n",
        "numel_list = [p.numel() for p in net.parameters() if p.requires_grad == True]\n",
        "sum(numel_list), numel_list"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MfqxYjiU2CXG",
        "outputId": "c2c87bbd-f5a2-4b03-99be-44e9cc8d3525"
      },
      "execution_count": 93,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(67, [50, 5, 10, 2])"
            ]
          },
          "metadata": {},
          "execution_count": 93
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# разделение параметров\n",
        "# будет один и тот же блок (одинаковые веса) в нескольких позициях\n",
        "shared = nn.Linear(8, 8)\n",
        "net = nn.Sequential(nn.Linear(4, 8),\n",
        "                    nn.ReLU(),\n",
        "                    shared,\n",
        "                    nn.ReLU(),\n",
        "                    shared,\n",
        "                    nn.ReLU(),\n",
        "                    nn.Linear(8, 1))"
      ],
      "metadata": {
        "id": "1Hw06meO2Q4D"
      },
      "execution_count": 94,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# задание своих модулей (слоёв)\n",
        "\n",
        "# центрирующий слой без параметров\n",
        "class CenteredLayer(nn.Module):\n",
        "  def __init__(self):\n",
        "    super().__init__()\n",
        "\n",
        "  def forward(self, X):\n",
        "    return X - X.mean()"
      ],
      "metadata": {
        "id": "B2O3EBcH3YAU"
      },
      "execution_count": 95,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# явное прописывание весов\n",
        "# если не сделать nn.Parameter(), то функционировать нейронка сможет, но\n",
        "# она не будет обучаться (так как параметры не будут зарегистрированы как\n",
        "# параметры)\n",
        "\n",
        "class MyNetworkWithParams(nn.Module):\n",
        "  def __init__(self,input_size, hidden_size, output_size):\n",
        "    super(MyNetworkWithParams,self).__init__()\n",
        "    # параметр автоматически регистрируется как параметр модуля\n",
        "    self.layer1_weights = nn.Parameter(torch.randn(input_size, hidden_size))\n",
        "    self.layer1_bias = nn.Parameter(torch.randn(hidden_size))\n",
        "    self.layer2_weights = nn.Parameter(torch.randn(hidden_size, output_size))\n",
        "    self.layer2_bias = nn.Parameter(torch.randn(output_size))\n",
        "\n",
        "  def forward(self,x):\n",
        "    h1 = torch.matmul(x,self.layer1_weights) + self.layer1_bias\n",
        "    h1_act = torch.max(h1, torch.zeros(h1.size())) # ReLU\n",
        "    output = torch.matmul(h1_act,self.layer2_weights) + self.layer2_bias\n",
        "    return output\n",
        "\n",
        "net4 = MyNetworkWithParams(32, 128, 10)"
      ],
      "metadata": {
        "id": "2t2-l_QA3pM4"
      },
      "execution_count": 96,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# со списком слоёв; их нужно хранить в nn.ModuleList() -- в этом случае\n",
        "# значения в этом списке автоматически регистрируются как параметры нейронки\n",
        "class MyNet(nn.Module):\n",
        "\n",
        "  def __init__(self,n_hidden_layers):\n",
        "    super(MyNet,self).__init__()\n",
        "    self.n_hidden_layers = n_hidden_layers\n",
        "    self.final_layer = nn.Linear(128, 10)\n",
        "    self.act = nn.ReLU()\n",
        "    self.hidden = []\n",
        "    for i in range(n_hidden_layers):\n",
        "      self.hidden.append(nn.Linear(128, 128))\n",
        "    self.hidden = nn.ModuleList(self.hidden)  # это важно!\n",
        "\n",
        "  def forward(self,x):\n",
        "    h = x\n",
        "    for i in range(self.n_hidden_layers):\n",
        "      h = self.hidden[i](h)\n",
        "      h = self.act(h)\n",
        "    out = self.final_layer(h)\n",
        "    return out\n",
        "\n",
        "# PS: есть ещё nn.ModuleDict()"
      ],
      "metadata": {
        "id": "gkvwE2BF46DJ"
      },
      "execution_count": 97,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# инициализация (можно применять и по отдельным модулям)\n",
        "\n",
        "def init_normal(m):\n",
        "  if type(m) == nn.Linear:\n",
        "    nn.init.normal_(m.weight, mean=0, std=0.01)\n",
        "    nn.init.zeros_(m.bias)\n",
        "\n",
        "def init_normal_v2(m):\n",
        "  if isinstance(m, nn.Linear):\n",
        "    torch.nn.init.xavier_uniform_(m.weight)\n",
        "    m.bias.data.fill_(0.01)\n",
        "\n",
        "# для каждого модуля нейросети запускается функция init_normal\n",
        "net.apply(init_normal)\n",
        "print(net[0].weight.data[0], net[0].bias.data[0])\n",
        "\n",
        "net.apply(init_normal_v2)\n",
        "print(net[0].weight.data[0], net[0].bias.data[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Yar9LvD75JwI",
        "outputId": "9f60211e-9652-435c-a5ea-fa89dacea3ad"
      },
      "execution_count": 98,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([-0.0041,  0.0016,  0.0080,  0.0009]) tensor(0.)\n",
            "tensor([ 0.6209, -0.3102, -0.3821,  0.4065]) tensor(0.0100)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# способ, где копируются веса в нужное место\n",
        "# эталонный тензор, который нужно скопировать в матрицу весов\n",
        "def init_custom(m):\n",
        "  if type(m) == nn.Linear:\n",
        "    rw = torch.randn(m.weight.data.size())\n",
        "    m.weight.data.copy_(rw)"
      ],
      "metadata": {
        "id": "mm3h1JjB5Ymd"
      },
      "execution_count": 99,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# как обучать нейросеть?\n",
        "# представлена функция, которая выполняет одну эпоху обучения\n",
        "\n",
        "# в Pytorch нужно всё прописывать, а не просто fit как в sklearn\n",
        "\n",
        "def train_epoch(model, train_loader, criterion, optimizer):\n",
        "  model.train()  # входим в нужный нам режим train\n",
        "  running_loss = 0.0\n",
        "  # цикл по батчам\n",
        "  for batch_idx, (data, target) in enumerate(train_loader):\n",
        "    optimizer.zero_grad(set_to_none=True)  # ~ model.zero_grad()\n",
        "    # = 0 чтобы не накапливались\n",
        "    data = data.to(device)\n",
        "    target = target.to(device)  # перенос на device\n",
        "    outputs = model(data)  # получили выход сетки\n",
        "    loss = criterion(outputs, target)  # посчитали для этого выхода лосс\n",
        "    running_loss += loss.item()\n",
        "    loss.backward()  # вычислили градиенты loss по параметрам сети (w)\n",
        "    optimizer.step()  # cдалем шаг по антиградиенту - обновляем веса сети\n",
        "    running_loss /= len(train_loader)\n",
        "  return running_loss"
      ],
      "metadata": {
        "id": "ExencbKD6_sP"
      },
      "execution_count": 100,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Loss-функции\n",
        "\n",
        "# [S] CrossEntropyLoss = Softmax + CrossEntropy\n",
        "# тот же эффект - logSoftmax + NLLLoss\n",
        "loss = nn.CrossEntropyLoss()\n",
        "a = torch.tensor([[1.0, 2.0, 3.0]])\n",
        "y = torch.tensor([1])\n",
        "\n",
        "print(a, y, loss(a, y))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4hrl3s587RWG",
        "outputId": "1d478cc0-2c9a-4cd3-b2f2-8cd7e87b72aa"
      },
      "execution_count": 101,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[1., 2., 3.]]) tensor([1]) tensor(1.4076)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "y = torch.tensor([])\n",
        "preds = torch.tensor([[1]])"
      ],
      "metadata": {
        "id": "od52lmZe4Xzo"
      },
      "execution_count": 102,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# своя Loss-функция\n",
        "\n",
        "class CustomNLLLoss(nn.Module):\n",
        "  def __init__(self):\n",
        "    super().__init__()\n",
        "\n",
        "  def forward(self, x, y):\n",
        "    # x должен быть выходом из слоя LogSoftmax\n",
        "    log_prob = -1.0 * x\n",
        "    # Get log_prob based on y class_index as loss=-mean(ylogp)\n",
        "    loss = log_prob.gather(1, y.unsqueeze(1))\n",
        "    loss = loss.mean()\n",
        "    return loss\n",
        "\n",
        "criterion = CustomNLLLoss() # nn.NLLLoss()\n",
        "CustomNLLLossClass = criterion(preds, y)"
      ],
      "metadata": {
        "id": "HUviq2EZ7Zkf"
      },
      "execution_count": 103,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model = nn.Sequential(OrderedDict([('features', nn.Linear(10, 5)),\n",
        "                                   ('hidden_activation', nn.ReLU()),\n",
        "                                   ('classifier', nn.Linear(5, 2))]))"
      ],
      "metadata": {
        "id": "6jN8x1_AzSHr"
      },
      "execution_count": 104,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# свой темп для каждого слоя\n",
        "\n",
        "from torch import optim\n",
        "\n",
        "optim.SGD([\n",
        "    {'params': model.features.parameters()},\n",
        "    {'params': model.classifier.parameters(), 'lr': 1e-3}\n",
        "    ], lr=1e-2, momentum=0.9)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qZytPrJ47q0k",
        "outputId": "cab0ff83-1095-4ee9-f300-a185403f9b1a"
      },
      "execution_count": 105,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SGD (\n",
              "Parameter Group 0\n",
              "    dampening: 0\n",
              "    differentiable: False\n",
              "    foreach: None\n",
              "    lr: 0.01\n",
              "    maximize: False\n",
              "    momentum: 0.9\n",
              "    nesterov: False\n",
              "    weight_decay: 0\n",
              "\n",
              "Parameter Group 1\n",
              "    dampening: 0\n",
              "    differentiable: False\n",
              "    foreach: None\n",
              "    lr: 0.001\n",
              "    maximize: False\n",
              "    momentum: 0.9\n",
              "    nesterov: False\n",
              "    weight_decay: 0\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 105
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# программаторы темпов обучения (программы изменения темпов обучения)\n",
        "\n",
        "from torch.optim.lr_scheduler import StepLR, ReduceLROnPlateau, CosineAnnealingLR\n",
        "\n",
        "# сообщаем параметры оптимизатору!\n",
        "optimizer = optim.SGD(net.parameters(), lr)\n",
        "\n",
        "# сообщаем оптимизатор \"шедьюлеру\"\n",
        "scheduler_1 = ReduceLROnPlateau(optimizer,\n",
        "                                factor=0.1,\n",
        "                                patience=1,\n",
        "                                threshold=0.1)\n",
        "\n",
        "# умножаем на gamma через каждые step_size шагов\n",
        "scheduler_2 = StepLR(optimizer, step_size=1, gamma=0.1)\n",
        "\n",
        "# ....\n",
        "\n",
        "# шаг на эпоху должен быть один, сделаем его после валидации\n",
        "if not is_train:\n",
        "  scheduler_1.step(running_loss / (i + 1))\n",
        "  scheduler_2.step()"
      ],
      "metadata": {
        "id": "114CGxrc71rv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 245
        },
        "outputId": "1c245536-5c75-4ec3-b5f2-7356963ade5d"
      },
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-106-2ed01978164e>\u001b[0m in \u001b[0;36m<cell line: 6>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;31m# сообщаем параметры оптимизатору!\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0moptimizer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moptim\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSGD\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnet\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;31m# сообщаем оптимизатор \"шедьюлеру\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'lr' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# свой оптимизатор\n",
        "\n",
        "class OptimizerTemplate:\n",
        "\n",
        "  def __init__(self, params, lr):\n",
        "    self.params = list(params)\n",
        "    self.lr = lr\n",
        "\n",
        "  def zero_grad(self):\n",
        "    ## все градиенты обнуляем\n",
        "    for p in self.params:\n",
        "      if p.grad is not None:\n",
        "        p.grad.detach_() # For second-order optimizers important\n",
        "        p.grad.zero_()\n",
        "\n",
        "  @torch.no_grad()\n",
        "  def step(self):\n",
        "    ## Apply update step to all parameters\n",
        "    for p in self.params:\n",
        "      if p.grad is None: # пропускаем параметры без градиентов\n",
        "        continue\n",
        "      self.update_param(p)\n",
        "\n",
        "  def update_param(self, p):\n",
        "    # To be implemented in optimizer-specific classes\n",
        "    raise NotImplementedError"
      ],
      "metadata": {
        "id": "Qv9vIEoaC6xW"
      },
      "execution_count": 107,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class SGD(OptimizerTemplate):\n",
        "\n",
        "  def __init__(self, params, lr):\n",
        "    super().__init__(params, lr)\n",
        "\n",
        "  def update_param(self, p):\n",
        "    p_update = -self.lr * p.grad\n",
        "    p.add_(p_update) # In-place => saves memory + doesn't create c. graph"
      ],
      "metadata": {
        "id": "zG2Ie2zhDFBV"
      },
      "execution_count": 108,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dqKVy65750aE",
        "outputId": "6c338079-f400-4988-84db-0a33f6d995a3"
      },
      "execution_count": 109,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Sequential(\n",
              "  (features): Linear(in_features=10, out_features=5, bias=True)\n",
              "  (hidden_activation): ReLU()\n",
              "  (classifier): Linear(in_features=5, out_features=2, bias=True)\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# сохранение/загрузка сети\n",
        "\n",
        "torch.save(model, \"/tmp/model.pth\")  # вся нейронка\n",
        "model = torch.load(\"/tmp/model.pth\")\n",
        "\n",
        "torch.save(model.state_dict(), \"/tmp/model.pth\") # только параметры\n",
        "model = nn.Sequential(OrderedDict([('features', nn.Linear(10, 5)),\n",
        "                                   ('hidden_activation', nn.ReLU()),\n",
        "                                   ('classifier', nn.Linear(5, 2))]))\n",
        "model.load_state_dict(torch.load(\"/tmp/model.pth\")) # только параметры\n",
        "model.to(device) # вот сейчас переносим\n",
        "model.eval()"
      ],
      "metadata": {
        "id": "SLj0VhAODiRt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b4c96db5-8d9b-4478-da65-67459bda3c81"
      },
      "execution_count": 110,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Sequential(\n",
              "  (features): Linear(in_features=10, out_features=5, bias=True)\n",
              "  (hidden_activation): ReLU()\n",
              "  (classifier): Linear(in_features=5, out_features=2, bias=True)\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 110
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "state = {\n",
        "    'epoch': epoch + 1,\n",
        "    'state_dict': net.state_dict(),\n",
        "    'optimizer' : optimizer.state_dict()\n",
        "}\n",
        "\n",
        "torch.save(state, './my_checkpoint.pth')"
      ],
      "metadata": {
        "id": "Rv1zMBFVDq0d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 245
        },
        "outputId": "d796418c-a3c2-46a9-d73b-ee3d03ca91a5"
      },
      "execution_count": 111,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-111-89a3e1f09bd9>\u001b[0m in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m state = {\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0;34m'epoch'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mepoch\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m     \u001b[0;34m'state_dict'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mnet\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0;34m'optimizer'\u001b[0m \u001b[0;34m:\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m }\n",
            "\u001b[0;31mNameError\u001b[0m: name 'epoch' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# своя функция активации\n",
        "\n",
        "@torch.jit.script\n",
        "def fused_gelu(x):\n",
        "  return x * 0.5 * (1.0 + torch.erf(x / 1.41421))"
      ],
      "metadata": {
        "id": "3z76eMwxDw5D"
      },
      "execution_count": 112,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class MySigmoid (nn.Module):\n",
        "\n",
        "  def __init__(self):\n",
        "    super().__init__()\n",
        "    self.name = self.__class__.__name__\n",
        "    self.config = {\"name\": self.name}\n",
        "\n",
        "  def forward(self, x):\n",
        "    return 1 / (1 + torch.exp(-x))"
      ],
      "metadata": {
        "id": "aju4BhqYD0gR"
      },
      "execution_count": 113,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# использование нейросетей\n",
        "\n",
        "# валидация\n",
        "model.eval() # переводим сеть в нужный режим -\n",
        "# меняет поведение forward() - влияет на DO, BN\n",
        "with torch.no_grad(): # контекстный менеджер, в котором не вычисляются\n",
        "# градиенты, даже у тензоров с requires_grad=True\n",
        "  train, y_train = train_dataset.tensors\n",
        "  # train, y_train = train.to(device), y_train.to(device)\n",
        "  # здесь лучше сделать цикл по dataloader, иначе model может не поместиться\n",
        "  train_preds = model(train)\n",
        "  train_loss = loss_fn(train_preds, y_train).item()"
      ],
      "metadata": {
        "id": "tsA0xHQrD8xx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 245
        },
        "outputId": "d21e12a4-3c22-4710-bcb0-3bf845865d85"
      },
      "execution_count": 114,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AttributeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-114-432e6c6a9b97>\u001b[0m in \u001b[0;36m<cell line: 6>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# контекстный менеджер, в котором не вычисляются\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;31m# градиенты, даже у тензоров с requires_grad=True\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m   \u001b[0mtrain\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_dataset\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtensors\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m   \u001b[0;31m# train, y_train = train.to(device), y_train.to(device)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m   \u001b[0;31m# здесь лучше сделать цикл по dataloader, иначе model может не поместиться\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mAttributeError\u001b[0m: 'Subset' object has no attribute 'tensors'"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Transfer Learning\n",
        "# В torchvision.models есть готовые модели, см.\n",
        "# https://pytorch.org/vision/stable/models.html\n",
        "\n",
        "# берём готовую модель\n",
        "from torchvision import models\n",
        "transfer_model = models.resnet34(pretrained=True)\n",
        "transfer_model.eval() # переводим сеть в нужный режим,\n",
        "# чтобы нормально работали BN и DO"
      ],
      "metadata": {
        "id": "KDrIYb89EK7H",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "05ac35b6-45ce-4232-c286-8b003dbf4556"
      },
      "execution_count": 115,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet34_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet34_Weights.DEFAULT` to get the most up-to-date weights.\n",
            "  warnings.warn(msg)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ResNet(\n",
              "  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
              "  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (relu): ReLU(inplace=True)\n",
              "  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
              "  (layer1): Sequential(\n",
              "    (0): BasicBlock(\n",
              "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "    (1): BasicBlock(\n",
              "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "    (2): BasicBlock(\n",
              "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "  )\n",
              "  (layer2): Sequential(\n",
              "    (0): BasicBlock(\n",
              "      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (downsample): Sequential(\n",
              "        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      )\n",
              "    )\n",
              "    (1): BasicBlock(\n",
              "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "    (2): BasicBlock(\n",
              "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "    (3): BasicBlock(\n",
              "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "  )\n",
              "  (layer3): Sequential(\n",
              "    (0): BasicBlock(\n",
              "      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (downsample): Sequential(\n",
              "        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      )\n",
              "    )\n",
              "    (1): BasicBlock(\n",
              "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "    (2): BasicBlock(\n",
              "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "    (3): BasicBlock(\n",
              "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "    (4): BasicBlock(\n",
              "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "    (5): BasicBlock(\n",
              "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "  )\n",
              "  (layer4): Sequential(\n",
              "    (0): BasicBlock(\n",
              "      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (downsample): Sequential(\n",
              "        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      )\n",
              "    )\n",
              "    (1): BasicBlock(\n",
              "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "    (2): BasicBlock(\n",
              "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    )\n",
              "  )\n",
              "  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))\n",
              "  (fc): Linear(in_features=512, out_features=1000, bias=True)\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# заморозка слоёв (если хотим попытаться дообучить)\n",
        "for name, param in model_cnn.named_parameters():\n",
        "  if (\"bn\" not in name): # BN лучше не замораживать!\n",
        "    param.requires_grad = False"
      ],
      "metadata": {
        "id": "DiRGPyHzERPm",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 228
        },
        "outputId": "1a753390-8783-4aa4-9ceb-a8c2868f2652"
      },
      "execution_count": 116,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-116-e486b63c28ba>\u001b[0m in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# заморозка слоёв (если хотим попытаться дообучить)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodel_cnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnamed_parameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m   \u001b[0;32mif\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\"bn\"\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# BN лучше не замораживать!\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequires_grad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'model_cnn' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from torchvision import models\n",
        "\n",
        "resnet18 = models.resnet18(pretrained=True)\n",
        "\n",
        "for parameter in resnet18.parameters(): # тут на самом деле веса,\n",
        "# но не как dict\n",
        "  parameter.requires_grad = False\n",
        "\n",
        "# заменяем слой\n",
        "resnet18.fc = nn.Linear(512, 10)\n",
        "\n",
        "# если будем дообучать, то будет обучаться только resnet18.fc"
      ],
      "metadata": {
        "id": "Nbk7BqRLEWUx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d252e2b4-fce6-43cf-eeb8-bd011640fa1a"
      },
      "execution_count": 117,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet18_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet18_Weights.DEFAULT` to get the most up-to-date weights.\n",
            "  warnings.warn(msg)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from torchvision import models\n",
        "from torchsummary import summary\n",
        "\n",
        "vgg = models.vgg16()\n",
        "vgg.to(device)\n",
        "summary(vgg, (3, 224, 224))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yyCaKIqyEgKO",
        "outputId": "896bb3e6-3f75-42ac-84b3-98f2ff233365"
      },
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "----------------------------------------------------------------\n",
            "        Layer (type)               Output Shape         Param #\n",
            "================================================================\n",
            "            Conv2d-1         [-1, 64, 224, 224]           1,792\n",
            "              ReLU-2         [-1, 64, 224, 224]               0\n",
            "            Conv2d-3         [-1, 64, 224, 224]          36,928\n",
            "              ReLU-4         [-1, 64, 224, 224]               0\n",
            "         MaxPool2d-5         [-1, 64, 112, 112]               0\n",
            "            Conv2d-6        [-1, 128, 112, 112]          73,856\n",
            "              ReLU-7        [-1, 128, 112, 112]               0\n",
            "            Conv2d-8        [-1, 128, 112, 112]         147,584\n",
            "              ReLU-9        [-1, 128, 112, 112]               0\n",
            "        MaxPool2d-10          [-1, 128, 56, 56]               0\n",
            "           Conv2d-11          [-1, 256, 56, 56]         295,168\n",
            "             ReLU-12          [-1, 256, 56, 56]               0\n",
            "           Conv2d-13          [-1, 256, 56, 56]         590,080\n",
            "             ReLU-14          [-1, 256, 56, 56]               0\n",
            "           Conv2d-15          [-1, 256, 56, 56]         590,080\n",
            "             ReLU-16          [-1, 256, 56, 56]               0\n",
            "        MaxPool2d-17          [-1, 256, 28, 28]               0\n",
            "           Conv2d-18          [-1, 512, 28, 28]       1,180,160\n",
            "             ReLU-19          [-1, 512, 28, 28]               0\n",
            "           Conv2d-20          [-1, 512, 28, 28]       2,359,808\n",
            "             ReLU-21          [-1, 512, 28, 28]               0\n",
            "           Conv2d-22          [-1, 512, 28, 28]       2,359,808\n",
            "             ReLU-23          [-1, 512, 28, 28]               0\n",
            "        MaxPool2d-24          [-1, 512, 14, 14]               0\n",
            "           Conv2d-25          [-1, 512, 14, 14]       2,359,808\n",
            "             ReLU-26          [-1, 512, 14, 14]               0\n",
            "           Conv2d-27          [-1, 512, 14, 14]       2,359,808\n",
            "             ReLU-28          [-1, 512, 14, 14]               0\n",
            "           Conv2d-29          [-1, 512, 14, 14]       2,359,808\n",
            "             ReLU-30          [-1, 512, 14, 14]               0\n",
            "        MaxPool2d-31            [-1, 512, 7, 7]               0\n",
            "AdaptiveAvgPool2d-32            [-1, 512, 7, 7]               0\n",
            "           Linear-33                 [-1, 4096]     102,764,544\n",
            "             ReLU-34                 [-1, 4096]               0\n",
            "          Dropout-35                 [-1, 4096]               0\n",
            "           Linear-36                 [-1, 4096]      16,781,312\n",
            "             ReLU-37                 [-1, 4096]               0\n",
            "          Dropout-38                 [-1, 4096]               0\n",
            "           Linear-39                 [-1, 1000]       4,097,000\n",
            "================================================================\n",
            "Total params: 138,357,544\n",
            "Trainable params: 138,357,544\n",
            "Non-trainable params: 0\n",
            "----------------------------------------------------------------\n",
            "Input size (MB): 0.57\n",
            "Forward/backward pass size (MB): 218.78\n",
            "Params size (MB): 527.79\n",
            "Estimated Total Size (MB): 747.15\n",
            "----------------------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# ещё способ посмотреть на параметры модели\n",
        "from prettytable import PrettyTable\n",
        "\n",
        "def count_parameters(model):\n",
        "  table = PrettyTable([\"Modules\", \"Parameters\"])\n",
        "  total_params = 0\n",
        "  for name, parameter in model.named_parameters():\n",
        "    if not parameter.requires_grad: continue\n",
        "    params = parameter.numel()\n",
        "    table.add_row([name, params])\n",
        "    total_params+=params\n",
        "  print(table)\n",
        "  print(f\"Total Trainable Params: {total_params}\")\n",
        "  return total_params\n",
        "\n",
        "count_parameters(vgg)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oH4plMJoEjXk",
        "outputId": "446427b7-53c6-44ba-bb0a-35741bbd2dec"
      },
      "execution_count": 119,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---------------------+------------+\n",
            "|       Modules       | Parameters |\n",
            "+---------------------+------------+\n",
            "|  features.0.weight  |    1728    |\n",
            "|   features.0.bias   |     64     |\n",
            "|  features.2.weight  |   36864    |\n",
            "|   features.2.bias   |     64     |\n",
            "|  features.5.weight  |   73728    |\n",
            "|   features.5.bias   |    128     |\n",
            "|  features.7.weight  |   147456   |\n",
            "|   features.7.bias   |    128     |\n",
            "|  features.10.weight |   294912   |\n",
            "|   features.10.bias  |    256     |\n",
            "|  features.12.weight |   589824   |\n",
            "|   features.12.bias  |    256     |\n",
            "|  features.14.weight |   589824   |\n",
            "|   features.14.bias  |    256     |\n",
            "|  features.17.weight |  1179648   |\n",
            "|   features.17.bias  |    512     |\n",
            "|  features.19.weight |  2359296   |\n",
            "|   features.19.bias  |    512     |\n",
            "|  features.21.weight |  2359296   |\n",
            "|   features.21.bias  |    512     |\n",
            "|  features.24.weight |  2359296   |\n",
            "|   features.24.bias  |    512     |\n",
            "|  features.26.weight |  2359296   |\n",
            "|   features.26.bias  |    512     |\n",
            "|  features.28.weight |  2359296   |\n",
            "|   features.28.bias  |    512     |\n",
            "| classifier.0.weight | 102760448  |\n",
            "|  classifier.0.bias  |    4096    |\n",
            "| classifier.3.weight |  16777216  |\n",
            "|  classifier.3.bias  |    4096    |\n",
            "| classifier.6.weight |  4096000   |\n",
            "|  classifier.6.bias  |    1000    |\n",
            "+---------------------+------------+\n",
            "Total Trainable Params: 138357544\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "138357544"
            ]
          },
          "metadata": {},
          "execution_count": 119
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "UniE9yph7HBw"
      },
      "execution_count": 119,
      "outputs": []
    }
  ]
}