loading_data_from_txt_to_huggingface_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HuggingFace Dataset Import\n",
    "\n",
    "In this notebook, we'll learn how to push a parallel text files to a huggingface dataset format. I have taken the example of Esperanto (epo) and English (eng) dataset. The similar methods could be applied to your dataset as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: datasets in /root/miniconda3/lib/python3.9/site-packages (1.16.1)\n",
      "Requirement already satisfied: dill in /root/miniconda3/lib/python3.9/site-packages (from datasets) (0.3.4)\n",
      "Requirement already satisfied: numpy>=1.17 in /root/miniconda3/lib/python3.9/site-packages (from datasets) (1.20.3)\n",
      "Requirement already satisfied: fsspec[http]>=2021.05.0 in /root/miniconda3/lib/python3.9/site-packages (from datasets) (2021.10.0)\n",
      "Requirement already satisfied: xxhash in /root/miniconda3/lib/python3.9/site-packages (from datasets) (2.0.2)\n",
      "Requirement already satisfied: pandas in /root/miniconda3/lib/python3.9/site-packages (from datasets) (1.3.4)\n",
      "Requirement already satisfied: requests>=2.19.0 in /root/miniconda3/lib/python3.9/site-packages (from datasets) (2.26.0)\n",
      "Requirement already satisfied: aiohttp in /root/miniconda3/lib/python3.9/site-packages (from datasets) (3.8.0)\n",
      "Requirement already satisfied: packaging in /root/miniconda3/lib/python3.9/site-packages (from datasets) (20.9)\n",
      "Requirement already satisfied: tqdm>=4.62.1 in /root/miniconda3/lib/python3.9/site-packages (from datasets) (4.62.3)\n",
      "Requirement already satisfied: huggingface-hub<1.0.0,>=0.1.0 in /root/miniconda3/lib/python3.9/site-packages (from datasets) (0.1.3.dev0)\n",
      "Requirement already satisfied: pyarrow!=4.0.0,>=3.0.0 in /root/miniconda3/lib/python3.9/site-packages (from datasets) (6.0.0)\n",
      "Requirement already satisfied: multiprocess in /root/miniconda3/lib/python3.9/site-packages (from datasets) (0.70.12.2)\n",
      "Requirement already satisfied: filelock in /root/miniconda3/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.1.0->datasets) (3.3.2)\n",
      "Requirement already satisfied: pyyaml in /root/miniconda3/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.1.0->datasets) (5.4.1)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /root/miniconda3/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.1.0->datasets) (3.10.0.2)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /root/miniconda3/lib/python3.9/site-packages (from packaging->datasets) (2.4.7)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /root/miniconda3/lib/python3.9/site-packages (from requests>=2.19.0->datasets) (3.3)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /root/miniconda3/lib/python3.9/site-packages (from requests>=2.19.0->datasets) (1.26.7)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /root/miniconda3/lib/python3.9/site-packages (from requests>=2.19.0->datasets) (2.0.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /root/miniconda3/lib/python3.9/site-packages (from requests>=2.19.0->datasets) (2021.10.8)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /root/miniconda3/lib/python3.9/site-packages (from aiohttp->datasets) (1.2.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /root/miniconda3/lib/python3.9/site-packages (from aiohttp->datasets) (5.2.0)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /root/miniconda3/lib/python3.9/site-packages (from aiohttp->datasets) (1.7.2)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /root/miniconda3/lib/python3.9/site-packages (from aiohttp->datasets) (1.2.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /root/miniconda3/lib/python3.9/site-packages (from aiohttp->datasets) (20.3.0)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /root/miniconda3/lib/python3.9/site-packages (from aiohttp->datasets) (4.0.0)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /root/miniconda3/lib/python3.9/site-packages (from pandas->datasets) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in /root/miniconda3/lib/python3.9/site-packages (from pandas->datasets) (2020.5)\n",
      "Requirement already satisfied: six>=1.5 in /root/miniconda3/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas->datasets) (1.16.0)\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import list_datasets\n",
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset Preparation\n",
    "\n",
    "The dataset used here is the eng-esp data from the Tatoeba challenge. All the available languages can be found [here](https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/master/Data.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-12-04 16:10:10--  https://object.pouta.csc.fi/Tatoeba-Challenge/eng-epo.tar\n",
      "Resolving object.pouta.csc.fi (object.pouta.csc.fi)... 86.50.254.18, 86.50.254.19\n",
      "Connecting to object.pouta.csc.fi (object.pouta.csc.fi)|86.50.254.18|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 35747840 (34M) [application/x-tar]\n",
      "Saving to: ‘eng-epo.tar’\n",
      "\n",
      "eng-epo.tar         100%[===================>]  34.09M  10.3MB/s    in 3.5s    \n",
      "\n",
      "2021-12-04 16:10:15 (9.79 MB/s) - ‘eng-epo.tar’ saved [35747840/35747840]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://object.pouta.csc.fi/Tatoeba-Challenge/eng-epo.tar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/eng-epo/\n",
      "data/eng-epo/train.src.gz\n",
      "data/eng-epo/dev.trg\n",
      "data/eng-epo/train.id.gz\n",
      "data/eng-epo/test.trg\n",
      "data/eng-epo/test.id\n",
      "data/eng-epo/dev.src\n",
      "data/eng-epo/dev.id\n",
      "data/eng-epo/test.src\n",
      "data/eng-epo/train.trg.gz\n"
     ]
    }
   ],
   "source": [
    "!tar -xvf  'eng-epo.tar'\n",
    "!gunzip 'data/eng-epo/train.src.gz'\n",
    "!gunzip 'data/eng-epo/train.trg.gz'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘data/train’: File exists\n",
      "mkdir: cannot create directory ‘data/dev’: File exists\n",
      "mkdir: cannot create directory ‘data/test’: File exists\n"
     ]
    }
   ],
   "source": [
    "!mkdir data/train\n",
    "!mkdir data/dev\n",
    "!mkdir data/test\n",
    "!mv data/eng-epo/train.src data/eng-epo/train.trg data/train/\n",
    "!mv data/eng-epo/dev.src data/eng-epo/dev.trg data/dev/\n",
    "!mv data/eng-epo/test.src data/eng-epo/test.trg data/test/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the text files into the HuggingFace Dataset\n",
    "After creating the above files, these files could easily be loaded into the HuggingFace dataset module and then be used further for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset tatoeba/enepo to /root/.cache/huggingface/datasets/tatoeba/enepo/1.1.0/ceb9a251901cd131eb363dc49fb49e5e6484d55d41aa2ac44cade040f854562e...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "011a55b85d2c495282bd3c29294d816a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dataset = load_dataset(\"load_dataset_script.py\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'translation': {'en': 'Accessible Table Caption',\n",
       "  'epo': 'Atingebla tabela apudskribo'}}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset['train'][1]"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "ee6759b9d6e8105c157b4d850d6f7f326467de02dec6fbe61f5d7e4feab800cd"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}