Web_scraping_for_LLM_in_2024.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "LMx4qMhlcnOA",
        "5JzWHcoCc1uC"
      ],
      "authorship_tag": "ABX9TyMd6P8Sk+QTlmP/dGvqp3zP",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/trancethehuman/ai-workshop-code/blob/main/Web_scraping_for_LLM_in_2024.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Let's setup our test: Get my competitors' pricing from their websites\n",
        "\n",
        "This is real. I am doing this not for shits and giggles.\n",
        "\n",
        "I'm building an interactive learning platform where content is taught using AI. Seems like everyone else is focusing on augmenting the authoring process and not the learning experience, but whatever."
      ],
      "metadata": {
        "id": "9My-G2T0ZCN0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "competitor_sites = [\n",
        "    {\n",
        "        \"name\": \"Articulate 360 by Adobe\",\n",
        "        \"url\": \"https://www.articulate.com/360/pricing/freelancers\"\n",
        "    },\n",
        "    {\n",
        "        \"name\": \"7taps\",\n",
        "        \"url\": \"https://www.7taps.com/pricing\"\n",
        "    },\n",
        "    {\n",
        "        \"name\": \"Mindsmith AI\",\n",
        "        \"url\": \"https://www.mindsmith.ai/pricing\"\n",
        "    },\n",
        "    {\n",
        "        \"name\": \"Cards-microlearning\",\n",
        "        \"url\": \"https://www.cards-microlearning.com/en/tarifs\"\n",
        "    },\n",
        "]"
      ],
      "metadata": {
        "id": "5svJvWdJZPMw"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Let's setup cost calculations\n",
        "\n",
        "So we can compare them side-by-side"
      ],
      "metadata": {
        "id": "uH7ec-0ddM8X"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can calculate how much it'll cost by using OpenAI's `tiktoken` library.\n",
        "\n",
        "(side note: as of today, OpenAI hasn't updated `tiktoken` with the actual algorithm used to in `gpt-4o`, so we'll guesstimate using `gpt-4` tokenization encoding (cl100k_base)."
      ],
      "metadata": {
        "id": "QJWAErlShOZ7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install tiktoken --quiet"
      ],
      "metadata": {
        "id": "SNMRdbmWhe6D",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bc3ec804-d7f4-4d68-e5de-b42eb8c55d98"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/1.1 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/1.1 MB\u001b[0m \u001b[31m1.5 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.5/1.1 MB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m12.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import tiktoken\n",
        "\n",
        "def count_tokens(input_string: str) -> int:\n",
        "    tokenizer = tiktoken.get_encoding(\"cl100k_base\")\n",
        "\n",
        "    tokens = tokenizer.encode(input_string)\n",
        "\n",
        "    return len(tokens)\n",
        "\n",
        "def calculate_cost(input_string: str, cost_per_million_tokens: float = 5) -> float:\n",
        "    num_tokens = count_tokens(input_string)\n",
        "\n",
        "    total_cost = (num_tokens / 1_000_000) * cost_per_million_tokens\n",
        "\n",
        "    return total_cost\n",
        "\n",
        "# Example usage:\n",
        "input_string = \"What's the difference between beer nuts and deer nuts? Beer nuts are about 5 dollars. Deer nuts are just under a buck.\"\n",
        "cost = calculate_cost(input_string)\n",
        "print(f\"The total cost for using gpt-4o is: $US {cost:.6f}\")"
      ],
      "metadata": {
        "id": "hGspe9jidY2t",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2fc19946-79aa-4146-e836-0759368bac7a"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The total cost for using gpt-4o is: $US 0.000135\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Additionally, I want to see the test results in a nice table, so let's set that up."
      ],
      "metadata": {
        "id": "k8QPEEMmipKm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install prettytable tqdm --quiet"
      ],
      "metadata": {
        "id": "sIu1-oLziuPO"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from typing import List, Callable, Dict\n",
        "from prettytable import PrettyTable, ALL\n",
        "from tqdm import tqdm\n",
        "\n",
        "def view_scraped_content(scrape_url_functions: List[Dict[str, Callable[[str], str]]], sites_list: List[Dict[str, str]], characters_to_display: int = 500, table_max_width: int = 50) -> List[Dict[str, str]]:\n",
        "    content_table_headers = [\"Site Name\"] + [f\"{func['name']} content\" for func in scrape_url_functions]\n",
        "    cost_table_headers = [\"Site Name\"] + [f\"{func['name']} cost\" for func in scrape_url_functions]\n",
        "\n",
        "    content_table = PrettyTable()\n",
        "    content_table.field_names = content_table_headers\n",
        "\n",
        "    cost_table = PrettyTable()\n",
        "    cost_table.field_names = cost_table_headers\n",
        "\n",
        "    scraped_data = []\n",
        "\n",
        "    for site in sites_list:\n",
        "        content_row = [site['name']]\n",
        "        cost_row = [site['name']]\n",
        "        site_data = {\"provider\": site['name'], \"sites\": []}\n",
        "\n",
        "        for scrape_function in scrape_url_functions:\n",
        "            function_name = scrape_function['name']\n",
        "            for _ in tqdm([site], desc=f\"Processing site {site['name']} using {function_name}\"):\n",
        "                try:\n",
        "                    content = scrape_function['function'](site['url'])\n",
        "                    content_snippet = content[:characters_to_display]\n",
        "                    content_row.append(content_snippet)\n",
        "\n",
        "                    cost = calculate_cost(content)\n",
        "                    cost_row.append(f\"${cost:.6f}\")\n",
        "\n",
        "                    site_data[\"sites\"].append({\"name\": function_name, \"content\": content})\n",
        "                except Exception as e:\n",
        "                    error_message = f\"Error: {str(e)}\"\n",
        "                    content_row.append(error_message)\n",
        "                    cost_row.append(\"Error\")\n",
        "\n",
        "                    site_data[\"sites\"].append({\"name\": function_name, \"content\": error_message})\n",
        "                    continue\n",
        "\n",
        "        content_table.add_row(content_row)\n",
        "        cost_table.add_row(cost_row)\n",
        "        scraped_data.append(site_data)\n",
        "\n",
        "    content_table.max_width = table_max_width\n",
        "    content_table.hrules = ALL\n",
        "\n",
        "    cost_table.max_width = table_max_width\n",
        "    cost_table.hrules = ALL\n",
        "\n",
        "    print(\"Content Table:\")\n",
        "    print(content_table)\n",
        "\n",
        "    print(\"\\nCost Table:\\nThis is how much it would cost to use gpt-4o to parse this content for extraction.\")\n",
        "    print(cost_table)\n",
        "\n",
        "    return scraped_data\n",
        "\n"
      ],
      "metadata": {
        "id": "K593G9Xh8Lln"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Setup all the scrapers\n",
        "\n",
        "Let's setup all of our scrapers."
      ],
      "metadata": {
        "id": "cqfJC7InYcm5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Beautiful Soup"
      ],
      "metadata": {
        "id": "LMx4qMhlcnOA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install requests beautifulsoup4 --quiet"
      ],
      "metadata": {
        "id": "mtEoxl-j4wvf"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Beautiful Soup utility functions\n",
        "\n",
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "\n",
        "def beautiful_soup_scrape_url(url: str):\n",
        "    response = requests.get(url)\n",
        "    soup = BeautifulSoup(response.content, 'html.parser')\n",
        "    return str(soup)\n"
      ],
      "metadata": {
        "id": "jqwpe5AuZBYA"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Reader API by Jina AI\n",
        "\n",
        "Let's setup Jina AI's scrape method. This one is dead easy."
      ],
      "metadata": {
        "id": "5JzWHcoCc1uC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "\n",
        "def scrape_jina_ai(url: str) -> str:\n",
        "  response = requests.get(\"https://r.jina.ai/\" + url)\n",
        "  return response.text"
      ],
      "metadata": {
        "id": "5ccgu2JZc4xL"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Firecrawl from Mendable."
      ],
      "metadata": {
        "id": "xhGYMijjNrvF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install firecrawl-py --quiet"
      ],
      "metadata": {
        "id": "c59LRZZ-NtuR"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import firecrawl\n",
        "import getpass\n",
        "\n",
        "FIRECRAWL_API_KEY = getpass.getpass(\"Mendable API Key: \")\n",
        "\n",
        "def scrape_firecrawl(url: str):\n",
        "    app = firecrawl.FirecrawlApp(api_key=FIRECRAWL_API_KEY)\n",
        "    scraped_data = app.scrape_url(url)[\"markdown\"]\n",
        "    return scraped_data"
      ],
      "metadata": {
        "id": "-AuHJTiwOQID",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1835e2e3-4395-42a8-e4ef-806db5b7dc5e"
      },
      "execution_count": 11,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mendable API Key: ··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Let's run all the scrapers and display them in our comparison table"
      ],
      "metadata": {
        "id": "uEfyleLPaTY5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "list_of_scraper_functions = [\n",
        "      {\"name\": \"Beautiful Soup\", \"function\": beautiful_soup_scrape_url},\n",
        "      {\"name\": \"Firecrawl\", \"function\": scrape_firecrawl},\n",
        "      {\"name\": \"Jina AI\", \"function\": scrape_jina_ai}\n",
        "      ]\n",
        "\n",
        "all_content = view_scraped_content(list_of_scraper_functions, competitor_sites, 700, 20)"
      ],
      "metadata": {
        "collapsed": true,
        "id": "tl8T54D69ojR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2a32ced6-d2a1-46a6-d41c-32dfca099bb4"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Processing site Articulate 360 by Adobe using Beautiful Soup: 100%|██████████| 1/1 [00:00<00:00,  5.85it/s]\n",
            "Processing site Articulate 360 by Adobe using Firecrawl: 100%|██████████| 1/1 [00:09<00:00,  9.60s/it]\n",
            "Processing site Articulate 360 by Adobe using Jina AI: 100%|██████████| 1/1 [00:00<00:00,  2.16it/s]\n",
            "Processing site 7taps using Beautiful Soup: 100%|██████████| 1/1 [00:00<00:00,  4.12it/s]\n",
            "Processing site 7taps using Firecrawl: 100%|██████████| 1/1 [00:04<00:00,  4.83s/it]\n",
            "Processing site 7taps using Jina AI: 100%|██████████| 1/1 [00:00<00:00,  2.19it/s]\n",
            "Processing site Mindsmith AI using Beautiful Soup: 100%|██████████| 1/1 [00:00<00:00,  4.20it/s]\n",
            "Processing site Mindsmith AI using Firecrawl: 100%|██████████| 1/1 [00:04<00:00,  4.74s/it]\n",
            "Processing site Mindsmith AI using Jina AI: 100%|██████████| 1/1 [00:00<00:00,  2.47it/s]\n",
            "Processing site Cards-microlearning using Beautiful Soup: 100%|██████████| 1/1 [00:00<00:00,  3.71it/s]\n",
            "Processing site Cards-microlearning using Firecrawl: 100%|██████████| 1/1 [00:05<00:00,  5.18s/it]\n",
            "Processing site Cards-microlearning using Jina AI: 100%|██████████| 1/1 [00:06<00:00,  6.72s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Content Table:\n",
            "+----------------------+------------------------+----------------------+----------------------+\n",
            "|      Site Name       | Beautiful Soup content |  Firecrawl content   |   Jina AI content    |\n",
            "+----------------------+------------------------+----------------------+----------------------+\n",
            "|  Articulate 360 by   |         <html>         |    [Skip to main     |  Title: Freelancer   |\n",
            "|        Adobe         | <head><title>403 Forbi |  content](#content)  |     Pricing for      |\n",
            "|                      |  dden</title></head>   |                      |   Articulate 360 -   |\n",
            "|                      |         <body>         | [![Articulate](https | Everything You Need  |\n",
            "|                      | <center><h1>403 Forbid | ://www.articulate.co | to Create E‑Learning |\n",
            "|                      |   den</h1></center>    | m/wp-content/uploads |                      |\n",
            "|                      | <hr/><center>nginx</ce | /2023/06/articulate- | URL Source: https:// |\n",
            "|                      |         nter>          | logo.svg)](https://w | www.articulate.com/3 |\n",
            "|                      |        </body>         | ww.articulate.com/)  | 60/pricing/freelance |\n",
            "|                      |        </html>         |                      |          rs          |\n",
            "|                      |                        | *   [Products](https |                      |\n",
            "|                      |                        | ://www.articulate.co |  Markdown Content:   |\n",
            "|                      |                        |       m/360/)        |  Our personal plan   |\n",
            "|                      |                        |     *   [Articulate  | includes everything  |\n",
            "|                      |                        | 360 Platform](https: |  you need to create  |\n",
            "|                      |                        | //www.articulate.com |  e-learning on your  |\n",
            "|                      |                        |        /360/)        |         own.         |\n",
            "|                      |                        |                      |                      |\n",
            "|                      |                        |     *   [Rise 360](h |  Articulate 360 for  |\n",
            "|                      |                        | ttps://www.articulat |     Freelancers      |\n",
            "|                      |                        |   e.com/360/rise/)   | -------------------- |\n",
            "|                      |                        |                      |      ----------      |\n",
            "|                      |                        |     *   [Storyline 3 |                      |\n",
            "|                      |                        | 60](https://www.arti | ![Image 1](https://w |\n",
            "|                      |                        | culate.com/360/story | ww.articulate.com/wp |\n",
            "|                      |                        |        line/)        | -content/uploads/202 |\n",
            "|                      |                        |                      |    3/10/pricing-     |\n",
            "|                      |                        |     *   [Reach 360]( |      illo.png)       |\n",
            "|                      |                        | https://www.articula |                      |\n",
            "|                      |                        |  te.com/360/reach/)  | [Personal Plan](http |\n",
            "|                      |                        |                      | s://www.articulate.c |\n",
            "|                      |                        |     *   [Articulate  |       om/360/)       |\n",
            "|                      |                        | 360 Teams](https://w | -------------------- |\n",
            "|                      |                        | ww.articulate.com/36 | -------------------- |\n",
            "|                      |                        |      0/teams/)       |       --------       |\n",
            "|                      |                        |                      |                      |\n",
            "|                      |                        | *   [Pricing](https: |    **Perfect for     |\n",
            "|                      |                        | //www.articulate.com |     individuals.     |\n",
            "|                      |                        |    /360/pricing/)    | Everything you need  |\n",
            "|                      |                        |     *   [Pricing Ove |      to create       |\n",
            "|                      |                        | rview](https://www.a |    e-learning.**     |\n",
            "|                      |                        | rticulate.com/360/pr |                      |\n",
            "|                      |                        |       icing/)        |  1,099$USDper user,  |\n",
            "|                      |                        |                      |       annually       |\n",
            "|                      |                        |                      |   For individuals    |\n",
            "|                      |                        |                      |                      |\n",
            "|                      |                        |                      |    All prices USD    |\n",
            "|                      |                        |                      |                      |\n",
            "|                      |                        |                      |     ### Features     |\n",
            "|                      |                        |                      |                      |\n",
            "|                      |                        |                      |     #### Course      |\n",
            "|                      |                        |                      |    Authoring Apps    |\n",
            "|                      |                        |                      |                      |\n",
            "|                      |                        |                      | Rise 360, Storyline  |\n",
            "|                      |                        |                      |         360,         |\n",
            "+----------------------+------------------------+----------------------+----------------------+\n",
            "|        7taps         |    <!DOCTYPE html>     |  [MicrolearningCONF  | Title: 7taps pricing |\n",
            "|                      |  <!-- Last Published:  | ep.12  |  Book a FRE |        plans         |\n",
            "|                      |    Tue May 14 2024     | E seat →](https://ww |                      |\n",
            "|                      |   18:38:07 GMT+0000    | w.microlearningconf. | URL Source: https:// |\n",
            "|                      | (Coordinated Universal |         com)         | www.7taps.com/pricin |\n",
            "|                      |  Time) --><html data-  |                      |          g           |\n",
            "|                      |          wf-           | [![](https://assets- |                      |\n",
            "|                      | domain=\"www.7taps.com\" | global.website-files |  Markdown Content:   |\n",
            "|                      | data-wf-page=\"6210eab9 | .com/6040dae83112172 |     How is 7taps     |\n",
            "|                      |   77ab3b4dedf4d5f1\"    | e5cc9b16f/65c262c6c8 | different from other |\n",
            "|                      | data-wf-site=\"6040dae8 | b053c756732bba_Frame |    microlearning     |\n",
            "|                      |   3112172e5cc9b16f\"    | .webp)![](https://as |       vendors?       |\n",
            "|                      | lang=\"en\"><head><meta  | sets-global.website- |                      |\n",
            "|                      |     charset=\"utf-      | files.com/6040dae831 |         add          |\n",
            "|                      |    8\"/><title>7taps    | 12172e5cc9b16f/65c26 |                      |\n",
            "|                      |        pricing         | 2c6c8b053c756732bba_ | 7taps pioneered the  |\n",
            "|                      |   plans</title><meta   | Frame.webp)![](https | concept of effective |\n",
            "|                      | content=\"See detailed  |      ://assets-      |    microlearning     |\n",
            "|                      |   pricing plans for    | global.website-files |  implementation in   |\n",
            "|                      |  7taps Microlearning.  | .com/6040dae83112172 |  the enterprise L&D  |\n",
            "|                      |  Enterprise plan free  | e5cc9b16f/65c262c6c8 | landscape four years |\n",
            "|                      |  trial available. Try  | b053c756732bba_Frame | ago, and we continue |\n",
            "|                      |    #1 microlearning    | .webp)![](https://as |   to dominate this   |\n",
            "|                      | platform today!\" name= | sets-global.website- | space. Unlike other  |\n",
            "|                      |  \"description\"/><meta  | files.com/6040dae831 |  vendors who might   |\n",
            "|                      | content=\"7taps pricing | 12172e5cc9b16f/65c26 |   hand you a tool    |\n",
            "|                      | plans\" property=\"og:ti | 2c6c8b053c756732bba_ |  without a roadmap,  |\n",
            "|                      |      tle\"/><meta       | Frame.webp)![](https | we're here to guide  |\n",
            "|                      | content=\"See detailed  |      ://assets-      |   you through the    |\n",
            "|                      |   pricing plans for    | global.website-files |     intricacies,     |\n",
            "|                      |  7taps Microlearning.  | .com/6040dae83112172 | ensuring you harness |\n",
            "|                      |  Enterprise plan free  | e5cc9b16f/65c262c6c8 |  the true potential  |\n",
            "|                      |  trial available. Try  | b053c756732bba_Frame | of microlearning. We |\n",
            "|                      |    #1 microlearning    | .webp)![](https://as |    uniquely offer    |\n",
            "|                      | platform today!\" prope | sets-global.website- |   lightning-fast,    |\n",
            "|                      | rty=\"og:description\"/> | files.com/6040dae831 |     high-quality     |\n",
            "|                      |   <meta content=\"ht    | 12172e5cc9b16f/65c26 |    microlearning     |\n",
            "|                      |                        |   2c6c8b053c75673    |    creation that     |\n",
            "|                      |                        |                      |    enables you to    |\n",
            "|                      |                        |                      |     connect with     |\n",
            "|                      |                        |                      |  learners anywhere   |\n",
            "|                      |                        |                      |  through versatile   |\n",
            "|                      |                        |                      |   sharing options.   |\n",
            "|                      |                        |                      |   Moreover, 7taps'   |\n",
            "|                      |                        |                      | [science-driven meth |\n",
            "|                      |                        |                      | odology](https://www |\n",
            "|                      |                        |                      |         .7ta         |\n",
            "+----------------------+------------------------+----------------------+----------------------+\n",
            "|     Mindsmith AI     |    <!DOCTYPE html>     | [![](https://assets- |    Title: Pricing    |\n",
            "|                      |   <!-- This site was   | global.website-files |                      |\n",
            "|                      | created in Webflow. ht | .com/63210902df3949e | URL Source: https:// |\n",
            "|                      | tps://www.webflow.com  | 6e16cb030/64653794e1 | www.mindsmith.ai/pri |\n",
            "|                      |      --><!-- Last      | 15e4ac4b553aa0_logo. |         cing         |\n",
            "|                      | Published: Wed May 15  |       svg)](/)       |                      |\n",
            "|                      | 2024 23:22:32 GMT+0000 |                      |  Markdown Content:   |\n",
            "|                      | (Coordinated Universal | [Pricing](/pricing)  |       ![Image        |\n",
            "|                      |  Time) --><html data-  |   [About](/about)    |  1](https://assets-  |\n",
            "|                      | wf-domain=\"www.mindsmi | [Book a Demo](https: | global.website-files |\n",
            "|                      | th.ai\" data-wf-page=\"6 | //app.apollo.io/#/me | .com/63210902df3949e |\n",
            "|                      | 54aa6a027f185dda8fec0d | et/ethan_webb_63c/de | 6e16cb030/65675d24c2 |\n",
            "|                      | 0\" data-wf-site=\"63210 |         mo)          | 36608822d95212_Featu |\n",
            "|                      |  902df3949e6e16cb030\"  |    [Blog](/blog)     | red%20icon%20Free.pn |\n",
            "|                      | lang=\"en\"><head><meta  |                      |          g)          |\n",
            "|                      |     charset=\"utf-      | [Log in](https://app |                      |\n",
            "|                      | 8\"/><title>Pricing</ti |   .mindsmith.ai/)    |      #### Free       |\n",
            "|                      |       tle><meta        |                      |                      |\n",
            "|                      |  content=\"Pricing and  | [Get Started](https: |          $0          |\n",
            "|                      | plans for teams of all | //app.mindsmith.ai/s |          --          |\n",
            "|                      | sizes, from individual |        ignup)        |                      |\n",
            "|                      | creators to eLearning  |                      |  Try out Mindsmith   |\n",
            "|                      | teams. Start for free  | [Sign Up](https://ap |       for free       |\n",
            "|                      |   or talk to a real    | p.mindsmith.ai/signu |                      |\n",
            "|                      | person to learn more!\" |          p)          | [Create an account]( |\n",
            "|                      | name=\"description\"/><m |                      | https://app.mindsmit |\n",
            "|                      | eta content=\"Pricing\"  | Plans for all sizes  |     h.ai/signup)     |\n",
            "|                      | property=\"og:title\"/>< | ===================  |                      |\n",
            "|                      | meta content=\"Pricing  |                      |       ![Image        |\n",
            "|                      | and plans for teams of | We provide discounts |  2](https://assets-  |\n",
            "|                      |    all sizes, from     |   to educators and   | global.website-files |\n",
            "|                      | individual creators to |      non-profit      | .com/63210902df3949e |\n",
            "|                      | eLearning teams. Start |    organizations.    | 6e16cb030/6567658146 |\n",
            "|                      |  for free or talk to   | [Contact Us.](mailto | e245f22fece858_Check |\n",
            "|                      |                        | :ethan@mindsmith.ai? |     %20icon.png)     |\n",
            "|                      |                        | subject=Non-profit%2 |                      |\n",
            "|                      |                        | 0%26%20Education%20d |        Create        |\n",
            "|                      |                        |      iscounts)       |                      |\n",
            "|                      |                        |                      |       ![Image        |\n",
            "|                      |                        |       Monthly        |  3](https://assets-  |\n",
            "|                      |                        |                      | global.website-files |\n",
            "|                      |                        |        Annual        | .com/63210902df3949e |\n",
            "|                      |                        |                      | 6e16cb030/6567658146 |\n",
            "|                      |                        |       Save 20%       | e245f22fece858_Check |\n",
            "|                      |                        |                      |     %20icon.png)     |\n",
            "|                      |                        | ![](https://assets-  |                      |\n",
            "|                      |                        | global.website-files |  Two active shared   |\n",
            "|                      |                        | .com/63210902df3949e |        lesson        |\n",
            "|                      |                        | 6e16cb030/65675d24c2 |                      |\n",
            "|                      |                        | 36608822d95212_Featu |       ![Image        |\n",
            "|                      |                        | red%20icon%20Free.pn |  4](https://assets-  |\n",
            "|                      |                        |          g)          | global.website-files |\n",
            "|                      |                        |                      | .com/63210902df3949e |\n",
            "|                      |                        |                      | 6e16cb030/6567658146 |\n",
            "|                      |                        |                      | e245f22fece858_Check |\n",
            "|                      |                        |                      |      %20icon.pn      |\n",
            "+----------------------+------------------------+----------------------+----------------------+\n",
            "| Cards-microlearning  |    <!DOCTYPE html>     | [![Logo Cards micro- |    Title: Micro-     |\n",
            "|                      |  <!-- Last Published:  | learning](https://as | learning cards from  |\n",
            "|                      |    Thu Apr 25 2024     | sets-global.website- | €89 | IA | Learning  |\n",
            "|                      |   18:33:08 GMT+0000    | files.com/642b30c7da |       Routine®       |\n",
            "|                      | (Coordinated Universal | f3c9185c06eaca/642b3 |                      |\n",
            "|                      |  Time) --><html data-  | 11ae57799309b8fc2df_ |     URL Source:      |\n",
            "|                      | wf-domain=\"www.cards-  | Logo-Cards-Horizonta | https://www.cards-mi |\n",
            "|                      |   microlearning.com\"   |  l_Noir.png)](/en)   | crolearning.com/en/t |\n",
            "|                      | data-wf-page=\"6446e0b1 |                      |        arifs         |\n",
            "|                      |   2b3fc3ef452f4a94\"    |       Platform       |                      |\n",
            "|                      | data-wf-site=\"642b30c7 |                      |  Markdown Content:   |\n",
            "|                      |   daf3c9185c06eaca\"    | [Functionalities](/e | HT cost/ additional  |\n",
            "|                      | lang=\"fr\"><head><meta  |  n/fonctionnalites-  |       license        |\n",
            "|                      |     charset=\"utf-      |    microlearning)    |                      |\n",
            "|                      |   8\"/><title>Micro-    |   [MAY ✨](/en/ia)   |      €3.6/month      |\n",
            "|                      |  learning cards from   | [Learning Routine®]( |                      |\n",
            "|                      |  €89 | IA | Learning   |    /en/learning-     |      €3.6/month      |\n",
            "|                      | Routine®</title><link  |       routine)       |                      |\n",
            "|                      | href=\"https://www.card |      [Authoring      |      €3.6/month      |\n",
            "|                      | s-microlearning.com/ta |   tool](/en/outil-   |                      |\n",
            "|                      |         rifs\"          |       auteur)        |     password,        |\n",
            "|                      |  hreflang=\"x-default\"  | [Microsoft Teams](/e |   Apple, Google &    |\n",
            "|                      | rel=\"alternate\"/><link |    n/integration-    |      Microsoft       |\n",
            "|                      | href=\"https://www.card |   microsoft-teams)   |                      |\n",
            "|                      | s-microlearning.com/ta |                      |     password,        |\n",
            "|                      | rifs\" hreflang=\"fr-FR\" |       Use case       |   Apple, Google &    |\n",
            "|                      | rel=\"alternate\"/><link |                      |      Microsoft       |\n",
            "|                      | href=\"https://www.card | [Onboarding](/en/lp/ |                      |\n",
            "|                      | s-microlearning.com/en |  formulaire-acces)   |     password,        |\n",
            "|                      | /tarifs\" hreflang=\"en\" | [Mobile Learning](/e |   Apple, Google &    |\n",
            "|                      | rel=\"alternate\"/><meta |   n/lp/formulaire-   |      Microsoft       |\n",
            "|                      |  content=\"Discover 3   |        acces)        |                      |\n",
            "|                      |     offers to take     | [Change Management]( |        \\+ AD         |\n",
            "|                      |   advantage of Cards   |  /en/lp/formulaire-  |   synchronization    |\n",
            "|                      |  micro-learning in a   |        acces)        |  (Active Directory)  |\n",
            "|                      |          100%          | [Standards/Complianc |                      |\n",
            "|                      |                        | e](/en/lp/formulaire |          \\-          |\n",
            "|                      |                        |       -acces)        |                      |\n",
            "|                      |                        | [Lifelong Learning]( |        Option        |\n",
            "|                      |                        |  /en/lp/formulaire-  |                      |\n",
            "|                      |                        |        acces)        |        Option        |\n",
            "|                      |                        | [Future of Work](/en |                      |\n",
            "|                      |                        |   /lp/formulaire-    |  Webinar start (1h)  |\n",
            "|                      |                        |        acces)        |                      |\n",
            "|                      |                        |                      |   Getting Started    |\n",
            "|                      |                        | [Pricing](/en/tarifs |    Webinar Ickoff    |\n",
            "|                      |                        |          )           |       Meeting        |\n",
            "|                      |                        |                      |  \\+ 1 Workshop (1h)  |\n",
            "|                      |                        |      Resources       |      for annual      |\n",
            "|                      |                        |                      |     commitments      |\n",
            "|                      |                        |    [White Paper •    |                      |\n",
            "|                      |                        |     **NEW**](/en     |   Getting Started    |\n",
            "|                      |                        |                      |       Webinar        |\n",
            "|                      |                        |                      | \\+ 4 Workshops (1h)  |\n",
            "|                      |                        |                      |      for annual      |\n",
            "|                      |                        |                      |     commitments      |\n",
            "|                      |                        |                      |                      |\n",
            "+----------------------+------------------------+----------------------+----------------------+\n",
            "\n",
            "Cost Table:\n",
            "This is how much it would cost to use gpt-4o to parse this content for extraction.\n",
            "+----------------------+---------------------+----------------+--------------+\n",
            "|      Site Name       | Beautiful Soup cost | Firecrawl cost | Jina AI cost |\n",
            "+----------------------+---------------------+----------------+--------------+\n",
            "|  Articulate 360 by   |      $0.000210      |   $0.019450    |  $0.008025   |\n",
            "|        Adobe         |                     |                |              |\n",
            "+----------------------+---------------------+----------------+--------------+\n",
            "|        7taps         |      $0.071940      |   $0.013890    |  $0.005745   |\n",
            "+----------------------+---------------------+----------------+--------------+\n",
            "|     Mindsmith AI     |      $0.205140      |   $0.022055    |  $0.018445   |\n",
            "+----------------------+---------------------+----------------+--------------+\n",
            "| Cards-microlearning  |      $0.136780      |   $0.034135    |  $0.000735   |\n",
            "+----------------------+---------------------+----------------+--------------+\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Now let's use OpenAI and extract just the information we need\n",
        "\n",
        "Let's see how accurate the extraction task is between each provider.\n",
        "\n"
      ],
      "metadata": {
        "id": "cH3WfAO3LAoP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "First, we create an extraction function using OpenAI's gpt-4o to get only the pricing content from each scraped website from each provider."
      ],
      "metadata": {
        "id": "fee4SzR9qMk0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install openai --quiet"
      ],
      "metadata": {
        "id": "tyWh8eUVLQ1n",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2ad07702-df4c-451e-cf96-c2ad571eea11"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/320.6 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m317.4/320.6 kB\u001b[0m \u001b[31m10.7 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m320.6/320.6 kB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m9.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m8.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import getpass\n",
        "from openai import OpenAI\n",
        "\n",
        "OPENAI_API_KEY = getpass.getpass('Enter your OpenAI API key: ')\n",
        "\n",
        "client = OpenAI(api_key=OPENAI_API_KEY)\n",
        "\n",
        "def extract(user_input: str):\n",
        "  entity_extraction_system_message = {\"role\": \"system\", \"content\": \"Get me the three pricing tiers from this website's content, and return as a JSON with three keys: {cheapest: {name: str, price: float}, middle: {name: str, price: float}, most_expensive: {name: str, price: float}}\"}\n",
        "\n",
        "  messages = [entity_extraction_system_message]\n",
        "  messages.append({\"role\": \"user\", \"content\": user_input})\n",
        "\n",
        "  response = client.chat.completions.create(\n",
        "        model=\"gpt-4o\",\n",
        "        messages=messages,\n",
        "        stream=False,\n",
        "        response_format={\"type\": \"json_object\"}\n",
        "    )\n",
        "\n",
        "  return response.choices[0].message.content"
      ],
      "metadata": {
        "id": "F9CfGvaYNi7F",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "49e710c0-0045-4e86-ca5a-0fffaa47c764"
      },
      "execution_count": 16,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your OpenAI API key: ··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Then, we create a utility function to display that content in a table."
      ],
      "metadata": {
        "id": "ermtSvxSra1I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def display_extracted_content(results: List[Dict[str, any]], num_objects: int):\n",
        "    table = PrettyTable()\n",
        "    table.field_names = [\"Site\", \"Provider Name\", \"Extracted Content\"]\n",
        "\n",
        "    # Ensure num_objects does not exceed the length of the results list\n",
        "    num_objects = min(num_objects, len(results))\n",
        "\n",
        "    # Process the specified number of items from the results list with a progress bar\n",
        "    for result in tqdm(results[:num_objects], desc=\"Processing results\"):\n",
        "        provider_name = result[\"provider\"]\n",
        "\n",
        "        for site in result[\"sites\"]:\n",
        "            function_name = site[\"name\"]\n",
        "            content = site[\"content\"]\n",
        "\n",
        "            # Progress bar for each function\n",
        "            for _ in tqdm(range(1), desc=f\"Extracting content with {provider_name} for {function_name}\"):\n",
        "                extracted_content = extract(content)\n",
        "                table.add_row([provider_name, function_name, extracted_content])\n",
        "\n",
        "    table.max_width = 50  # Set the maximum width for better display\n",
        "    table.hrules = ALL\n",
        "\n",
        "    print(\"Extracted Content Table:\")\n",
        "    print(table)"
      ],
      "metadata": {
        "id": "9XIcSACXkIvn"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "display_extracted_content(all_content, num_objects=9)"
      ],
      "metadata": {
        "id": "KkgYywhEqF-V",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4eeb9031-4716-4105-f848-659c6707992c"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Processing results:   0%|          | 0/4 [00:00<?, ?it/s]\n",
            "Extracting content with Articulate 360 by Adobe for Beautiful Soup:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with Articulate 360 by Adobe for Beautiful Soup: 100%|██████████| 1/1 [00:01<00:00,  1.42s/it]\n",
            "\n",
            "Extracting content with Articulate 360 by Adobe for Firecrawl:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with Articulate 360 by Adobe for Firecrawl: 100%|██████████| 1/1 [00:01<00:00,  1.72s/it]\n",
            "\n",
            "Extracting content with Articulate 360 by Adobe for Jina AI:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with Articulate 360 by Adobe for Jina AI: 100%|██████████| 1/1 [00:01<00:00,  1.55s/it]\n",
            "Processing results:  25%|██▌       | 1/4 [00:04<00:14,  4.72s/it]\n",
            "Extracting content with 7taps for Beautiful Soup:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with 7taps for Beautiful Soup: 100%|██████████| 1/1 [00:02<00:00,  2.44s/it]\n",
            "\n",
            "Extracting content with 7taps for Firecrawl:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with 7taps for Firecrawl: 100%|██████████| 1/1 [00:01<00:00,  1.27s/it]\n",
            "\n",
            "Extracting content with 7taps for Jina AI:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with 7taps for Jina AI: 100%|██████████| 1/1 [00:01<00:00,  1.29s/it]\n",
            "Processing results:  50%|█████     | 2/4 [00:09<00:09,  4.90s/it]\n",
            "Extracting content with Mindsmith AI for Beautiful Soup:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with Mindsmith AI for Beautiful Soup: 100%|██████████| 1/1 [00:41<00:00, 41.91s/it]\n",
            "\n",
            "Extracting content with Mindsmith AI for Firecrawl:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with Mindsmith AI for Firecrawl: 100%|██████████| 1/1 [00:06<00:00,  6.14s/it]\n",
            "\n",
            "Extracting content with Mindsmith AI for Jina AI:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with Mindsmith AI for Jina AI: 100%|██████████| 1/1 [00:05<00:00,  5.59s/it]\n",
            "Processing results:  75%|███████▌  | 3/4 [01:03<00:27, 27.17s/it]\n",
            "Extracting content with Cards-microlearning for Beautiful Soup:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with Cards-microlearning for Beautiful Soup: 100%|██████████| 1/1 [00:46<00:00, 46.06s/it]\n",
            "\n",
            "Extracting content with Cards-microlearning for Firecrawl:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with Cards-microlearning for Firecrawl: 100%|██████████| 1/1 [00:09<00:00,  9.29s/it]\n",
            "\n",
            "Extracting content with Cards-microlearning for Jina AI:   0%|          | 0/1 [00:00<?, ?it/s]\u001b[A\n",
            "Extracting content with Cards-microlearning for Jina AI: 100%|██████████| 1/1 [00:01<00:00,  1.66s/it]\n",
            "Processing results: 100%|██████████| 4/4 [02:00<00:00, 30.12s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Extracted Content Table:\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "|           Site          | Provider Name  |                 Extracted Content                  |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "| Articulate 360 by Adobe | Beautiful Soup |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |                      \"name\": \"\",                   |\n",
            "|                         |                |                      \"price\": 0.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |                      \"name\": \"\",                   |\n",
            "|                         |                |                      \"price\": 0.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |                      \"name\": \"\",                   |\n",
            "|                         |                |                      \"price\": 0.0                  |\n",
            "|                         |                |                          }                         |\n",
            "|                         |                |                         }                          |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "| Articulate 360 by Adobe |   Firecrawl    |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |                \"name\": \"Personal Plan\",            |\n",
            "|                         |                |                    \"price\": 1099.0                 |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |                 \"name\": \"Teams Plan\",              |\n",
            "|                         |                |                    \"price\": 1399.0                 |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |                \"name\": \"Reach 360 Pro\",            |\n",
            "|                         |                |          \"price\": \"Contact sales for pricing\"      |\n",
            "|                         |                |                          }                         |\n",
            "|                         |                |                         }                          |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "| Articulate 360 by Adobe |    Jina AI     |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |                \"name\": \"Personal Plan\",            |\n",
            "|                         |                |                    \"price\": 1099.0                 |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |                 \"name\": \"Teams Plan\",              |\n",
            "|                         |                |                    \"price\": 1399.0                 |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |             \"name\": \"Reach 360 Pro Plan\",          |\n",
            "|                         |                |                  \"price\": \"Variable\"               |\n",
            "|                         |                |                          }                         |\n",
            "|                         |                |                         }                          |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "|          7taps          | Beautiful Soup |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |                    \"name\": \"Free\",                 |\n",
            "|                         |                |                      \"price\": 0.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |                     \"name\": \"Pro\",                 |\n",
            "|                         |                |                     \"price\": 99.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |                 \"name\": \"Enterprise\",              |\n",
            "|                         |                |                              \"price\": 0.0 \t}                           |\n",
            "|                         |                |                         }                          |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "|          7taps          |   Firecrawl    |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |                    \"name\": \"Free\",                 |\n",
            "|                         |                |                      \"price\": 0.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |                   \"name\": \"Starter\",               |\n",
            "|                         |                |                     \"price\": 20.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |                     \"name\": \"Pro\",                 |\n",
            "|                         |                |                     \"price\": 99.0                  |\n",
            "|                         |                |                          }                         |\n",
            "|                         |                |                         }                          |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "|          7taps          |    Jina AI     |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |                    \"name\": \"Free\",                 |\n",
            "|                         |                |                     \"price\": 0.00                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |                     \"name\": \"Pro\",                 |\n",
            "|                         |                |                    \"price\": 199.00                 |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |                 \"name\": \"Enterprise\",              |\n",
            "|                         |                |                    \"price\": 499.00                 |\n",
            "|                         |                |                          }                         |\n",
            "|                         |                |                         }                          |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "|       Mindsmith AI      | Beautiful Soup |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |                    \"name\": \"Free\",                 |\n",
            "|                         |                |                      \"price\": 0.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |                \"name\": \"Professional\",             |\n",
            "|                         |                |                     \"price\": 39.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |                    \"name\": \"Team\",                 |\n",
            "|                         |                |                     \"price\": null                  |\n",
            "|                         |                |                          }                         |\n",
            "|                         |                |                         }                          |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "|       Mindsmith AI      |   Firecrawl    |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |                    \"name\": \"Free\",                 |\n",
            "|                         |                |                      \"price\": 0.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |                \"name\": \"Professional\",             |\n",
            "|                         |                |                     \"price\": 31.20                 |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |                    \"name\": \"Team\",                 |\n",
            "|                         |                |                     \"price\": 39.0                  |\n",
            "|                         |                |                          }                         |\n",
            "|                         |                |                         }                          |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "|       Mindsmith AI      |    Jina AI     |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |                    \"name\": \"Free\",                 |\n",
            "|                         |                |                      \"price\": 0.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |                \"name\": \"Professional\",             |\n",
            "|                         |                |                     \"price\": 31.2                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |                    \"name\": \"Team\",                 |\n",
            "|                         |                |                     \"price\": 39.0                  |\n",
            "|                         |                |                          }                         |\n",
            "|                         |                |                         }                          |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "|   Cards-microlearning   | Beautiful Soup |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |                    \"name\": \"Basic\",                |\n",
            "|                         |                |                     \"price\": 99.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |                   \"name\": \"Starter\",               |\n",
            "|                         |                |                    \"price\": 4400.0                 |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |                  \"name\": \"Business\",               |\n",
            "|                         |                |                    \"price\": 10500.0                |\n",
            "|                         |                |                          }                         |\n",
            "|                         |                |                         }                          |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "|   Cards-microlearning   |   Firecrawl    |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |                 \"name\": \"BASIC OFFER\",             |\n",
            "|                         |                |                     \"price\": 99.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |                   \"name\": \"Starter\",               |\n",
            "|                         |                |                    \"price\": 4400.0                 |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |                  \"name\": \"Business\",               |\n",
            "|                         |                |                    \"price\": 10500.0                |\n",
            "|                         |                |                          }                         |\n",
            "|                         |                |                         }                          |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n",
            "|   Cards-microlearning   |    Jina AI     |                         {                          |\n",
            "|                         |                |                    \"cheapest\": {                   |\n",
            "|                         |                |         \"name\": \"HT cost/ additional license\",     |\n",
            "|                         |                |                      \"price\": 3.6                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                     \"middle\": {                    |\n",
            "|                         |                |        \"name\": \"Getting Started Webinar Ickoff     |\n",
            "|                         |                | Meeting + 1 Workshop (1h) for annual commitments\", |\n",
            "|                         |                |                     \"price\": 89.0                  |\n",
            "|                         |                |                          },                        |\n",
            "|                         |                |                 \"most_expensive\": {                |\n",
            "|                         |                |     \"name\": \"Getting Started Webinar + 4 Workshops |\n",
            "|                         |                |           (1h) for annual commitments\",            |\n",
            "|                         |                |                     \"price\": null                  |\n",
            "|                         |                |                          }                         |\n",
            "|                         |                |                         }                          |\n",
            "|                         |                |                                                    |\n",
            "|                         |                |                                                    |\n",
            "+-------------------------+----------------+----------------------------------------------------+\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "JIEQytLaN-vm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Bonus: Scrapegraph-ai"
      ],
      "metadata": {
        "id": "_fYI3_sKJi2M"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Scrapegraph-ai\n",
        "\n",
        "Scrapegraph-ai takes care of the entire flow (from scrape to extraction). It's also interesting that it's node-based, and can run off of local models (Ollama supported). But I couldn't find a way to get cost estimates based on tokens used.\n",
        "\n",
        "Demo link: https://scrapegraph-ai-demo.streamlit.app/\n"
      ],
      "metadata": {
        "id": "-hW7r5VJctoo"
      }
    }
  ]
}