第六章：数据编码和处理.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 读写CSV数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['AA', '39.48', '6/11/2007', '9:36am', '-0.18', '181800']\n",
      "['AIG', '71.38', '6/11/2007', '9:36am', '-0.15', '195500']\n",
      "['AXP', '62.58', '6/11/2007', '9:36am', '-0.46', '935000']\n",
      "['BA', '98.31', '6/11/2007', '9:36am', '+0.12', '104800']\n",
      "['C', '53.08', '6/11/2007', '9:36am', '-0.25', '360900']\n",
      "['CAT', '78.29', '6/11/2007', '9:36am', '-0.23', '225400']\n"
     ]
    }
   ],
   "source": [
    "# 对于大多数的CSV格式的数据读写问题，都可以使用csv库\n",
    "import csv\n",
    "\n",
    "with open('stocks.csv') as f:\n",
    "    f_csv = csv.reader(f)\n",
    "    headers = next(f_csv)\n",
    "    for row in f_csv:\n",
    "        print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">> 在上面的代码中，row会是一个列表。因此，为了访问某个字段，你需要使用下标，如row[0]访问Symbol,row[4]访问Change。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row(Symbol='AA', Price='39.48', Date='6/11/2007', Time='9:36am', Change='-0.18', Volume='181800')\n",
      "Row(Symbol='AIG', Price='71.38', Date='6/11/2007', Time='9:36am', Change='-0.15', Volume='195500')\n",
      "Row(Symbol='AXP', Price='62.58', Date='6/11/2007', Time='9:36am', Change='-0.46', Volume='935000')\n",
      "Row(Symbol='BA', Price='98.31', Date='6/11/2007', Time='9:36am', Change='+0.12', Volume='104800')\n",
      "Row(Symbol='C', Price='53.08', Date='6/11/2007', Time='9:36am', Change='-0.25', Volume='360900')\n",
      "Row(Symbol='CAT', Price='78.29', Date='6/11/2007', Time='9:36am', Change='-0.23', Volume='225400')\n"
     ]
    }
   ],
   "source": [
    "# 由于这种下标访问通常会引起混淆，你可以考虑使用命名元组\n",
    "from collections import namedtuple\n",
    "\n",
    "with open('stocks.csv') as f:\n",
    "    f_csv = csv.reader(f)\n",
    "    headings = next(f_csv)\n",
    "    Row = namedtuple('Row',headings)\n",
    "    for r in f_csv:\n",
    "        row = Row(*r)\n",
    "        print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">> 它允许你使用列名如row.Symbola和row.Change代替下标访问。需要注意的是这个只有在列名是合法的Python标识符的时候才生效。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AA\n",
      "AIG\n",
      "AXP\n",
      "BA\n",
      "C\n",
      "CAT\n"
     ]
    }
   ],
   "source": [
    "# 另外一个选择就是将数据读取到一个字典序列中去。\n",
    "import csv\n",
    "\n",
    "with open('stocks.csv') as f:\n",
    "    f_csv = csv.DictReader(f)\n",
    "    for row in f_csv:\n",
    "        # process row\n",
    "        print(row['Symbol'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">> 在这个版本中，你可以使用列名去访问每一行的数据，比如：row['Symbol']或者row['change']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 为了写入CSV数据，你仍然可以使用csv模块，不过这时候先创建一个writer对象。\n",
    "headers = ['Symbol','Price','Date','Time','Change','Volume']\n",
    "rows = [('AA','39.38','6/11/2007','9:36am',-0.18,181800),\n",
    "       ('AIG',71.38,'6/11/2007','9:36am',-0.15,195500),\n",
    "       ('AXP',62.58,'6/11/2007','9:36am',-0.46,935000)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('Stocks.csv','w') as f:\n",
    "    f_csv = csv.writer(f)\n",
    "    f_csv.writerow(headers)\n",
    "    f_csv.writerows(rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Symbol', 'Price', 'Date', 'Time', 'Change', 'Volume\\n']\n",
      "['AA', '39.38', '6/11/2007', '9:36am', '-0.18', '181800\\n']\n",
      "['AIG', '71.38', '6/11/2007', '9:36am', '-0.15', '195500\\n']\n",
      "['AXP', '62.58', '6/11/2007', '9:36am', '-0.46', '935000\\n']\n"
     ]
    }
   ],
   "source": [
    "# 你应该总是优先选择csv模块分割或解析csv数据。例如，你可能会像编写类似下面这样的代码：\n",
    "with open('stocks.csv') as f:\n",
    "    for line in f:\n",
    "        row = line.split(',')\n",
    "        # process row\n",
    "        print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用这种方式的一个缺点就是你仍然需要去处理一些棘手的细节问题。 比如，如果某些字段值被引号包围，你不得不去除这些引号。 另外，如果一个被引号包围的字段碰巧含有一个逗号，那么程序就会因为产生一个错误大小的行而出错。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "默认情况下，csv 库可识别Microsoft Excel所使用的CSV编码规则。 这或许也是最常见的形式，并且也会给你带来最好的兼容性。 然而，如果你查看csv的文档，就会发现有很多种方法将它应用到其他编码格式上(如修改分割字符等)。 例如，如果你想读取以tab分割的数据，可以这样做：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Symbol', 'Price', 'Date', 'Time', 'Change', 'Volume']\n",
      "['AA', '39.48', '6/11/2007', '9:36am', '-0.18', '181800']\n",
      "['AIG', '71.38', '6/11/2007', '9:36am', '-0.15', '195500']\n",
      "['AXP', '62.58', '6/11/2007', '9:36am', '-0.46', '935000']\n",
      "['BA', '98.31', '6/11/2007', '9:36am', '+0.12', '104800']\n",
      "['C', '53.08', '6/11/2007', '9:36am', '-0.25', '360900']\n",
      "['CAT', '78.29', '6/11/2007', '9:36am', '-0.23', '225400']\n"
     ]
    }
   ],
   "source": [
    "# Example of reading tab-separeted values\n",
    "with open('stocks.tsv') as f:\n",
    "    f_tsv = csv.reader(f,delimiter='\\t')\n",
    "    for row in f_tsv:\n",
    "        # process row\n",
    "        print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">> 最后，如果你读取CSV数据的目的是做数据分析和统计的话， 你可能需要看一看 Pandas 包。Pandas 包含了一个非常方便的函数叫 pandas.read_csv() ， 它可以加载CSV数据到一个 DataFrame 对象中去。 然后利用这个对象你就可以生成各种形式的统计、过滤数据以及执行其他高级操作了。 在6.13小节中会有这样一个例子。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 读写JSON数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# json模块提供了一种很简单的方式来编码和解码JSON数据。其中两个主要的函数是json.dumps()\n",
    "# 和 json.loads(),要比其它序列化函数库如pickle的接口少很多。\n",
    "import json\n",
    "\n",
    "data = {\n",
    "    'name':'ACME',\n",
    "    'shares':100,\n",
    "    'price':542.3\n",
    "}\n",
    "\n",
    "json_str = json.dumps(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'ACME', 'shares': 100, 'price': 542.3}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 下面演示如何将一个JSON编码的字符串转换回一个Python数据结构\n",
    "data = json.loads(json_str)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">> 如果你要处理的是文件而不是字符串，你可以使用json.dump()和json.load()来编码和解码JSON数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Writing JSON data\n",
    "with open('data.json','w') as f:\n",
    "    json.dump(data,f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'ACME', 'shares': 100, 'price': 542.3}\n"
     ]
    }
   ],
   "source": [
    "# Reading data back\n",
    "with open('data.json','r') as f:\n",
    "    data = json.load(f)\n",
    "    print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JSON编码支持的基本数据类型为 None ， bool ， int ， float 和 str ， 以及包含这些类型数据的lists，tuples和dictionaries。 对于dictionaries，keys需要是字符串类型(字典中任何非字符串类型的key在编码时会先转换为字符串)。 为了遵循JSON规范，你应该只编码Python的lists和dictionaries。 而且，在web应用程序中，顶层对象被编码为一个字典是一个标准做法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JSON编码的格式对于Python语法而已几乎是完全一样的，除了一些小的差异之外。 比如，True会被映射为true，False被映射为false，而None会被映射为null。 下面是一个例子，演示了编码后的字符串效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'false'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json.dumps(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = {\n",
    "    'a':True,\n",
    "    'b':'Hello',\n",
    "    'c':None\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"a\": true, \"b\": \"Hello\", \"c\": null}'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json.dumps(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你试着去检查JSON解码后的数据，你通常很难通过简单的打印来确定它的结构， 特别是当数据的嵌套结构层次很深或者包含大量的字段时。 为了解决这个问题，可以考虑使用pprint模块的 pprint() 函数来代替普通的 print() 函数。 它会按照key的字母顺序并以一种更加美观的方式输出。 下面是一个演示如何漂亮的打印输出Twitter上搜索结果的例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlopen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "ename": "HTTPError",
     "evalue": "HTTP Error 410: Gone",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mHTTPError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-30-079ebf338379>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mu\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0murlopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'http://search.twitter.com/search.json?q=python&rpp=5'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(url, data, timeout, cafile, capath, cadefault, context)\u001b[0m\n\u001b[1;32m    220\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    221\u001b[0m         \u001b[0mopener\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_opener\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 222\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mopener\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    223\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    224\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0minstall_opener\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopener\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(self, fullurl, data, timeout)\u001b[0m\n\u001b[1;32m    529\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mprocessor\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprocess_response\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprotocol\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    530\u001b[0m             \u001b[0mmeth\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprocessor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmeth_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 531\u001b[0;31m             \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmeth\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    532\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    533\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36mhttp_response\u001b[0;34m(self, request, response)\u001b[0m\n\u001b[1;32m    639\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m200\u001b[0m \u001b[0;34m<=\u001b[0m \u001b[0mcode\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;36m300\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    640\u001b[0m             response = self.parent.error(\n\u001b[0;32m--> 641\u001b[0;31m                 'http', request, response, code, msg, hdrs)\n\u001b[0m\u001b[1;32m    642\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    643\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36merror\u001b[0;34m(self, proto, *args)\u001b[0m\n\u001b[1;32m    561\u001b[0m             \u001b[0mhttp_err\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    562\u001b[0m         \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mdict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mproto\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmeth_name\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 563\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_chain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    564\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    565\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36m_call_chain\u001b[0;34m(self, chain, kind, meth_name, *args)\u001b[0m\n\u001b[1;32m    501\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhandler\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mhandlers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    502\u001b[0m             \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhandler\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmeth_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 503\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    504\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    505\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36mhttp_error_302\u001b[0;34m(self, req, fp, code, msg, headers)\u001b[0m\n\u001b[1;32m    753\u001b[0m         \u001b[0mfp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    754\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 755\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnew\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreq\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    756\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    757\u001b[0m     \u001b[0mhttp_error_301\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhttp_error_303\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhttp_error_307\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhttp_error_302\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(self, fullurl, data, timeout)\u001b[0m\n\u001b[1;32m    529\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mprocessor\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprocess_response\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprotocol\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    530\u001b[0m             \u001b[0mmeth\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprocessor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmeth_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 531\u001b[0;31m             \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmeth\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    532\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    533\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36mhttp_response\u001b[0;34m(self, request, response)\u001b[0m\n\u001b[1;32m    639\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m200\u001b[0m \u001b[0;34m<=\u001b[0m \u001b[0mcode\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;36m300\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    640\u001b[0m             response = self.parent.error(\n\u001b[0;32m--> 641\u001b[0;31m                 'http', request, response, code, msg, hdrs)\n\u001b[0m\u001b[1;32m    642\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    643\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36merror\u001b[0;34m(self, proto, *args)\u001b[0m\n\u001b[1;32m    567\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mhttp_err\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    568\u001b[0m             \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mdict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'default'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'http_error_default'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0morig_args\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 569\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call_chain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    570\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    571\u001b[0m \u001b[0;31m# XXX probably also want an abstract factory that knows when it makes\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36m_call_chain\u001b[0;34m(self, chain, kind, meth_name, *args)\u001b[0m\n\u001b[1;32m    501\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhandler\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mhandlers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    502\u001b[0m             \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhandler\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmeth_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 503\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    504\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    505\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/urllib/request.py\u001b[0m in \u001b[0;36mhttp_error_default\u001b[0;34m(self, req, fp, code, msg, hdrs)\u001b[0m\n\u001b[1;32m    647\u001b[0m \u001b[0;32mclass\u001b[0m \u001b[0mHTTPDefaultErrorHandler\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mBaseHandler\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    648\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mhttp_error_default\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreq\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhdrs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 649\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mHTTPError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfull_url\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhdrs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    650\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    651\u001b[0m \u001b[0;32mclass\u001b[0m \u001b[0mHTTPRedirectHandler\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mBaseHandler\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mHTTPError\u001b[0m: HTTP Error 410: Gone"
     ]
    }
   ],
   "source": [
    "u = urlopen('http://search.twitter.com/search.json?q=python&rpp=5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般来讲，JSON解码会根据提供的数据创建dicts或lists。 如果你想要创建其他类型的对象，可以给 json.loads() 传递object_pairs_hook或object_hook参数。 例如，下面是演示如何解码JSON数据并在一个OrderedDict中保留其顺序的例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = '{\"name\":\"ACME\",\"shares\":50,\"price\":490.1}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import OrderedDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = json.loads(s,object_pairs_hook=OrderedDict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('name', 'ACME'), ('shares', 50), ('price', 490.1)])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 下面是如何将一个JSON字典转换为一个Python对象例子\n",
    "class JSONObject:\n",
    "    def __init__(self,d):\n",
    "        self.__dict__ = d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = json.loads(s,object_hook=JSONObject)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ACME'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shares"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "490.1"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"name\": \"ACME\", \"shares\": 100, \"price\": 542.23}\n"
     ]
    }
   ],
   "source": [
    "# 在编码json的时候，还有一些选项很有用。如果你想获得漂亮的格式化字符串后输出，\n",
    "# 可以使用json.dumps()的indent参数。\n",
    "data = {\n",
    "    'name' : 'ACME',\n",
    "    'shares' : 100,\n",
    "    'price' : 542.23\n",
    "}\n",
    "print(json.dumps(data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"name\": \"ACME\",\n",
      "    \"shares\": 100,\n",
      "    \"price\": 542.23\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(data,indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 解析简单的XML数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 可以使用xml.etree.ElementTree模块从简单的XML文档中提取数据\n",
    "from urllib.request import urlopen\n",
    "from xml.etree.ElementTree import parse\n",
    "\n",
    "# Download the RSS feed and parse it\n",
    "u = urlopen('http://planet.python.org/rss20.xml')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = parse(u)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ListenData: Create Dummy Data in Python\n",
      "Sat, 27 Apr 2019 13:38:35 +0000\n",
      "https://www.listendata.com/2019/04/create-dummy-data-in-python.html\n",
      "\n",
      "A. Jesse Jiryu Davis: PyCon Canada Video: API Evolution the Right Way\n",
      "Sat, 27 Apr 2019 12:07:22 +0000\n",
      "https://emptysqua.re/blog/api-evolution-pycon-canada-video/\n",
      "\n",
      "Weekly Python StackOverflow Report: (clxxv) stackoverflow python report\n",
      "Sat, 27 Apr 2019 06:30:00 +0000\n",
      "http://python-weekly.blogspot.com/2019/04/clxxv-stackoverflow-python-report.html\n",
      "\n",
      "Codementor: Deploy your distributed system efficiently with fabric\n",
      "Sat, 27 Apr 2019 02:18:58 +0000\n",
      "https://www.codementor.io/tonywang/deploy-your-distributed-system-efficiently-with-fabric-uf00dvvwk\n",
      "\n",
      "Catalin George Festila: Django REST framework - part 001.\n",
      "Sat, 27 Apr 2019 00:27:10 +0000\n",
      "http://python-catalin.blogspot.com/2019/04/django-rest-framework-part-001.html\n",
      "\n",
      "Catalin George Festila: Python 3.7.3 and Django REST framework.\n",
      "Sat, 27 Apr 2019 00:23:12 +0000\n",
      "http://python-catalin.blogspot.com/2019/04/python-373-and-django-rest-framework.html\n",
      "\n",
      "Yasoob Khalid: Python dis module and constant folding\n",
      "Fri, 26 Apr 2019 23:16:14 +0000\n",
      "https://pythontips.com/2019/02/26/python-dis-module-and-constant-folding/\n",
      "\n",
      "Stories in My Pocket: Don't be afraid of Test-Driven Development\n",
      "Fri, 26 Apr 2019 16:46:22 +0000\n",
      "https://storiesinmypocket.com/articles/dont-be-afraid-test-driven-development/\n",
      "\n",
      "PyCharm: Webinar Recording: “Effective Data Science with PyCharm” with Dan Tofan\n",
      "Fri, 26 Apr 2019 14:22:05 +0000\n",
      "http://feedproxy.google.com/~r/Pycharm/~3/yq1eNdOFlzw/\n",
      "\n",
      "Stack Abuse: Python for NLP: Introduction to the Pattern Library\n",
      "Fri, 26 Apr 2019 13:45:00 +0000\n",
      "https://stackabuse.com/python-for-nlp-introduction-to-the-pattern-library/\n",
      "\n",
      "PyCon: An update regarding PyCon 2019 sponsor DataCamp\n",
      "Thu, 25 Apr 2019 21:59:27 +0000\n",
      "https://pycon.blogspot.com/2019/04/an-update-regarding-pycon-2019-sponsor.html\n",
      "\n",
      "Continuum Analytics Blog: TensorFlow CPU optimizations in Anaconda\n",
      "Thu, 25 Apr 2019 20:54:52 +0000\n",
      "https://www.anaconda.com/tensorflow-cpu-optimizations-in-anaconda/\n",
      "\n",
      "Stack Abuse: Deep vs Shallow Copies in Python\n",
      "Thu, 25 Apr 2019 19:28:27 +0000\n",
      "https://stackabuse.com/deep-vs-shallow-copies-in-python/\n",
      "\n",
      "Python Engineering at Microsoft: Come meet Microsoft at PyCon 2019!\n",
      "Thu, 25 Apr 2019 17:00:21 +0000\n",
      "https://devblogs.microsoft.com/python/come-meet-microsoft-at-pycon-2019/\n",
      "\n",
      "Neckbeard Republic: Conditional Statements in Python (if/elif/else)\n",
      "Thu, 25 Apr 2019 14:00:00 +0000\n",
      "https://realpython.com/courses/python-conditional-statements/\n",
      "\n",
      "Evennia: Steaming on, eating jam\n",
      "Thu, 25 Apr 2019 10:39:35 +0000\n",
      "http://evennia.blogspot.com/2019/04/steaming-on-eating-jam.html\n",
      "\n",
      "Python Bytes: #127 That Python code is on fire!\n",
      "Thu, 25 Apr 2019 08:00:00 +0000\n",
      "https://pythonbytes.fm/episodes/show/127/that-python-code-is-on-fire\n",
      "\n",
      "EuroPython Society: EuroPython 2019: Call for Proposals\n",
      "Thu, 25 Apr 2019 07:35:35 +0000\n",
      "https://www.europython-society.org/post/184430536360\n",
      "\n",
      "EuroPython: EuroPython 2019: Call for Proposals\n",
      "Thu, 25 Apr 2019 07:27:24 +0000\n",
      "https://blog.europython.eu/post/184430444152\n",
      "\n",
      "EuroPython Society: EuroPython 2019: Launching our website\n",
      "Thu, 25 Apr 2019 07:01:52 +0000\n",
      "https://www.europython-society.org/post/184430136620\n",
      "\n",
      "EuroPython: EuroPython 2019: Launching our website\n",
      "Thu, 25 Apr 2019 06:59:30 +0000\n",
      "https://blog.europython.eu/post/184430105792\n",
      "\n",
      "Django Weblog: Paid Internship Opportunity: Build an App for the DSF\n",
      "Wed, 24 Apr 2019 18:43:21 +0000\n",
      "https://www.djangoproject.com/weblog/2019/apr/24/internship-opportunity-dsf-app/\n",
      "\n",
      "Real Python: Python KeyError Exceptions and How to Handle Them\n",
      "Wed, 24 Apr 2019 14:00:00 +0000\n",
      "https://realpython.com/python-keyerror/\n",
      "\n",
      "Catalin George Festila: Google's Python Class - another step.\n",
      "Wed, 24 Apr 2019 13:55:21 +0000\n",
      "http://python-catalin.blogspot.com/2019/04/googles-python-class-another-step.html\n",
      "\n",
      "Stack Abuse: Working with PDFs in Python: Reading and Splitting\n",
      "Wed, 24 Apr 2019 13:18:00 +0000\n",
      "https://stackabuse.com/working-with-pdfs-in-python-reading-and-splitting/\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Ectract and output tags of interset\n",
    "for item in doc.iterfind('channel/item'):\n",
    "    title = item.findtext('title')\n",
    "    date = item.findtext('pubDate')\n",
    "    link = item.findtext('link')\n",
    "    \n",
    "    print(title)\n",
    "    print(date)\n",
    "    print(link)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xml.etree.ElementTree.ElementTree at 0x10ea92e80>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "e = doc.find('channel/title')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Element 'title' at 0x10cab28b8>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'title'"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Planet Python'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "e.get('some_attribute')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有一点要强调的是 xml.etree.ElementTree 并不是XML解析的唯一方法。 对于更高级的应用程序，你需要考虑使用 lxml 。 它使用了和ElementTree同样的编程接口，因此上面的例子同样也适用于lxml。 你只需要将刚开始的import语句换成 from lxml.etree import parse 就行了。 lxml 完全遵循XML标准，并且速度也非常快，同时还支持验证，XSLT，和XPath等特性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 增量式解析大型XML文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 任何时候只要你遇到增量式的数据处理时，第一时间就应该想到迭代器和生成器。下面是一个很简单\n",
    "# 的函数，只使用很少的内存就能增量式的处理一个大型XML文件\n",
    "from xml.etree.ElementTree import iterparse\n",
    "\n",
    "def parse_and_remove(filename,path):\n",
    "    path_parts = path.split('/')\n",
    "    doc = iterparse(filename,('start','end'))\n",
    "    # skip the root element\n",
    "    next(doc)\n",
    "    \n",
    "    tag_stack = []\n",
    "    elem_stack =[]\n",
    "    for event,elem in doc:\n",
    "        if event == 'start':\n",
    "            tag_stack.append(elem.tag)\n",
    "            elem_stack.append(elem)\n",
    "        elif event == 'end':\n",
    "            if tag_stack == path_parts:\n",
    "                yield elem\n",
    "                elem_stack[-2].remove(elem)\n",
    "            try:\n",
    "                tag_stack.pop()\n",
    "                elem_stack.pop()\n",
    "            except IndexError:\n",
    "                pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60617 13\n",
      "60626 8\n",
      "60651 7\n",
      "60647 6\n",
      "60623 6\n",
      "60613 4\n",
      "60636 4\n",
      "60625 4\n",
      "60628 4\n",
      "60609 4\n",
      "60622 3\n",
      "60657 3\n",
      "60619 3\n",
      "60629 3\n",
      "60641 3\n",
      "60618 2\n",
      "60644 2\n",
      "60654 2\n",
      "60649 2\n",
      "60638 2\n",
      "60656 2\n",
      "60660 1\n",
      "60643 1\n",
      "60634 1\n",
      "60632 1\n",
      "60639 1\n",
      "60630 1\n",
      "60612 1\n",
      "60616 1\n",
      "60614 1\n",
      "60652 1\n",
      "60707 1\n",
      "60631 1\n",
      "60637 1\n"
     ]
    }
   ],
   "source": [
    "from xml.etree.ElementTree import parse\n",
    "from collections import Counter\n",
    "\n",
    "potholes_by_zip = Counter()\n",
    "\n",
    "doc = parse('potholes.xml')\n",
    "for pothole in doc.iterfind('row/row'):\n",
    "    potholes_by_zip[pothole.findtext('zip')] += 1\n",
    "for zipcode,num in potholes_by_zip.most_common():\n",
    "    print(zipcode,num)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个脚本唯一的问题是它会先将整个XML文件加载到内存中然后解析。 在我的机器上，为了运行这个程序需要用到450MB左右的内存空间。 如果使用如下代码，程序只需要修改一点点："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60617 13\n",
      "60626 8\n",
      "60651 7\n",
      "60647 6\n",
      "60623 6\n",
      "60613 4\n",
      "60636 4\n",
      "60625 4\n",
      "60628 4\n",
      "60609 4\n",
      "60622 3\n",
      "60657 3\n",
      "60619 3\n",
      "60629 3\n",
      "60641 3\n",
      "60618 2\n",
      "60644 2\n",
      "60654 2\n",
      "60649 2\n",
      "60638 2\n",
      "60656 2\n",
      "60660 1\n",
      "60643 1\n",
      "60634 1\n",
      "60632 1\n",
      "60639 1\n",
      "60630 1\n",
      "60612 1\n",
      "60616 1\n",
      "60614 1\n",
      "60652 1\n",
      "60707 1\n",
      "60631 1\n",
      "60637 1\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "potholes_by_zip = Counter()\n",
    "\n",
    "data = parse_and_remove('potholes.xml','row/row')\n",
    "for pothole in data:\n",
    "    potholes_by_zip[pothole.findtext('zip')] += 1\n",
    "for zipcode,num in potholes_by_zip.most_common():\n",
    "    print(zipcode,num)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这一节的技术会依赖 ElementTree 模块中的两个核心功能。 第一，iterparse() 方法允许对XML文档进行增量操作。 使用时，你需要提供文件名和一个包含下面一种或多种类型的事件列表： start , end, start-ns 和 end-ns 。 由 iterparse() 创建的迭代器会产生形如 (event, elem) 的元组， 其中 event 是上述事件列表中的某一个，而 elem 是相应的XML元素。例如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = iterparse('potholes.xml',('start','end'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('start', <Element 'response' at 0x10c74fb38>)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('start', <Element 'row' at 0x10c74f908>)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('start', <Element 'row' at 0x10c7587c8>)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('start', <Element 'creation_date' at 0x10c758a48>)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('end', <Element 'creation_date' at 0x10c758a48>)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "start 事件在某个元素第一次被创建并且还没有被插入其他数据(如子元素)时被创建。 而 end 事件在某个元素已经完成时被创建。 尽管没有在例子中演示， start-ns 和 end-ns 事件被用来处理XML文档命名空间的声明。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 将字典转换为XML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 尽管xml.etree.ElementTree库通常用来解析工作，其实它也可以创建XML文档\n",
    "from xml.etree.ElementTree import Element\n",
    "\n",
    "def dict_to_xml(tag,d):\n",
    "    '''\n",
    "    Turn a simple dict of key/value pairs into XML\n",
    "    '''\n",
    "    elem = Element(tag)\n",
    "    for key,val in d.items():\n",
    "        child = Element(key)\n",
    "        child.text = str(val)\n",
    "        elem.append(child)\n",
    "    return elem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = {'name':\"GOOG\",'shares':100,'price':490.1}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Element 'stock' at 0x10ecfccc8>"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e = dict_to_xml('stock',s)\n",
    "e"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xml.etree.ElementTree import tostring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'<stock><name>GOOG</name><shares>100</shares><price>490.1</price></stock>'"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tostring(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 如果你想给某个元素添加属性，可以使用set()\n",
    "e.set('_id','1234')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'<stock _id=\"1234\"><name>GOOG</name><shares>100</shares><price>490.1</price></stock>'"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tostring(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. 解析和修改XML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用xml.etree.ElementTree模块可以很容易的处理这些任务。第一步是以通常的方式来解析这个文档。例如，假设你有一个名为pred.xml的文档。\n",
    "\n",
    "```\n",
    "<?xml version=\"1.0\"?>\n",
    "<stop>\n",
    "    <id>14791</id>\n",
    "    <nm>Clark &amp; Balmoral</nm>\n",
    "    <sri>\n",
    "        <rt>22</rt>\n",
    "        <d>North Bound</d>\n",
    "        <dd>North Bound</dd>\n",
    "    </sri>\n",
    "    <cr>22</cr>\n",
    "    <pre>\n",
    "        <pt>5 MIN</pt>\n",
    "        <fd>Howard</fd>\n",
    "        <v>1378</v>\n",
    "        <rn>22</rn>\n",
    "    </pre>\n",
    "    <pre>\n",
    "        <pt>15 MIN</pt>\n",
    "        <fd>Howard</fd>\n",
    "        <v>1867</v>\n",
    "        <rn>22</rn>\n",
    "    </pre>\n",
    "</stop>\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Element 'stop' at 0x1108110e8>"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 下面是一个利用ElementTree来读取这个文档并对它做一些修改的例子\n",
    "from xml.etree.ElementTree import parse,Element\n",
    "\n",
    "doc = parse('pred.xml')\n",
    "root = doc.getroot()\n",
    "root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove a few elements\n",
    "root.remove(root.find('sri'))\n",
    "root.remove(root.find('cr'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/lidianxiang/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:2: DeprecationWarning: This method will be removed in future versions.  Use 'list(elem)' or iteration over elem instead.\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Insert a new element after <nm>...</nm>\n",
    "root.getchildren().index(root.find('nm'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "e = Element('spam')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "e.text = 'This is a test'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "root.insert(2,e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write back to a file\n",
    "doc.write('newpred.xml',xml_declaration=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">> 修改一个XML文档结构是很容易的，但是你必须牢记的是所有的修改都是针对父节点元素， 将它作为一个列表来处理。例如，如果你删除某个元素，通过调用父节点的 remove() 方法从它的直接父节点中删除。 如果你插入或增加新的元素，你同样使用父节点元素的 insert() 和 append() 方法。 还能对元素使用索引和切片操作，比如 element[i] 或 element[i:j]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. 利用命名空间解析XML文档"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "考虑下面这个使用了命名空间的文档：\n",
    "\n",
    "```\n",
    "\n",
    "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
    "<top>\n",
    "    <author>David Beazley</author>\n",
    "    <content>\n",
    "        <html xmlns=\"http://www.w3.org/1999/xhtml\">\n",
    "            <head>\n",
    "                <title>Hello World</title>\n",
    "            </head>\n",
    "            <body>\n",
    "                <h1>Hello World!</h1>\n",
    "            </body>\n",
    "        </html>\n",
    "    </content>\n",
    "</top>\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 通过将命名空间处理逻辑包装为一个工具类来简化这个过程\n",
    "class XMLNamespaces:\n",
    "    def __init__(self,**kwargs):\n",
    "        self.namespaces = {}\n",
    "        for name,uri in kwargs.items():\n",
    "            self.register(name,uri)\n",
    "    def register(self,name,uri):\n",
    "        self.namespaces[name] = '{'+uri+'}'\n",
    "    def __call__(self,path):\n",
    "        return path.format_map(self.namespaces)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "ns = XMLNamespaces(html='http://www.w3.org/1999/xhtml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc.find(ns('content/{html}html'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc.findtext(ns('content/{html}html/{html}head/{html}title'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后一点，如果你要处理的XML文本除了要使用到其他高级XML特性外，还要使用到命名空间， 建议你最好是使用 lxml 函数库来代替 ElementTree 。 例如，lxml 对利用DTD验证文档、更好的XPath支持和一些其他高级XML特性等都提供了更好的支持。 这一小节其实只是教你如何让XML解析稍微简单一点。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. 与关系型数据库的交互"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 第一步是连接到数据库。通常你要执行 connect() 函数， 给它提供一些数据库名、主机、用户\n",
    "# 名、密码和其他必要的一些参数。例如：\n",
    "import sqlite3\n",
    "\n",
    "db = sqlite3.connect('database.db')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlite3.Cursor at 0x11080c030>"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = db.cursor()\n",
    "\n",
    "c.execute('create table portfolio (symbol text, shares integer, price real)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "db.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlite3.Cursor at 0x11080c030>"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 为了向数据库表中插入多条记录，使用类似下面这样的语句：\n",
    "stocks =  [\n",
    "    ('GOOG', 100, 490.1),\n",
    "    ('AAPL', 50, 545.75),\n",
    "    ('FB', 150, 7.45),\n",
    "    ('HPQ', 75, 33.2),\n",
    "]\n",
    "\n",
    "c.executemany('insert into portfolio values (?,?,?)',stocks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "db.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('GOOG', 100, 490.1)\n",
      "('AAPL', 50, 545.75)\n",
      "('FB', 150, 7.45)\n",
      "('HPQ', 75, 33.2)\n"
     ]
    }
   ],
   "source": [
    "# 为了执行某个查询，使用像下面这样的语句\n",
    "for row in db.execute('select * from portfolio'):\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 如果你想接受用户输入作为参数来执行查询操作，必须确保你使用下面这样的占位符''?''来进行引用参数\n",
    "min_price =100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('GOOG', 100, 490.1)\n",
      "('AAPL', 50, 545.75)\n"
     ]
    }
   ],
   "source": [
    "for row in db.execute('select * from portfolio where price >= ?',(min_price,)):\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. 编码和解码十六进制数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 如果你只是简单的解码或编码一个十六进制的原始字符串，可以使用binascii模块。\n",
    "# Initial byte string\n",
    "s = b'hello'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'68656c6c6f'"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Encode as hex\n",
    "import binascii\n",
    "\n",
    "h = binascii.b2a_hex(s)\n",
    "h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'hello'"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Decode back to bytes\n",
    "binascii.a2b_hex(h)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'68656C6C6F'"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 类似的功能可以在base64模块中找到。\n",
    "import base64\n",
    "\n",
    "h = base64.b16encode(s)\n",
    "h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'hello'"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "base64.b16decode(h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "大部分情况下，通过使用上述的函数来转换十六进制是很简单的。 上面两种技术的主要不同在于大小写的处理。 函数 base64.b16decode() 和 base64.b16encode() 只能操作大写形式的十六进制字母， 而 binascii 模块中的函数大小写都能处理。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. 编码解码Base64数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "# base64模块中有两个函数b64encode()和b64decode()可以帮你解决这个问题\n",
    "# some byte data \n",
    "s = b'hello'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'aGVsbG8='"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import base64\n",
    "\n",
    "a = base64.b64encode(s)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'hello'"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Decode from Base64\n",
    "base64.b64decode(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Base64编码仅仅用于面向字节的数据比如字节字符串和字节数组。 此外，编码处理的输出结果总是一个字节字符串。 如果你想混合使用Base64编码的数据和Unicode文本，你必须添加一个额外的解码步骤。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'aGVsbG8='"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = base64.b64encode(s).decode('ascii')\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. 读写二进制数组数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用struct模块处理二进制数据\n",
    "from struct import Struct\n",
    "\n",
    "def write_records(records,format,f):\n",
    "    '''\n",
    "    Write a sequence of tuples to a binary file of structures\n",
    "    '''\n",
    "    record_struct = Struct(format)\n",
    "    for r in records:\n",
    "        f.write(record_struct.pack(*r))\n",
    "        \n",
    "# Example \n",
    "if __name__ == '__main__':\n",
    "    records = [\n",
    "        (1,2.3,4.5),\n",
    "        (6,7.8,9.0),\n",
    "        (12,13.4,56.7)\n",
    "    ]\n",
    "    with open('data.b','wb') as f:\n",
    "        write_records(records,'<idd',f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. 读取嵌套和可变长的二进制数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "# struct模块可被用来编码/解码几乎所有类型的二进制数据结构。\n",
    "import struct\n",
    "import itertools\n",
    "\n",
    "def write_polys(filename, polys):\n",
    "    # Determine bounding box\n",
    "    flattened = list(itertools.chain(*polys))\n",
    "    min_x = min(x for x, y in flattened)\n",
    "    max_x = max(x for x, y in flattened)\n",
    "    min_y = min(y for x, y in flattened)\n",
    "    max_y = max(y for x, y in flattened)\n",
    "    with open(filename, 'wb') as f:\n",
    "        f.write(struct.pack('<iddddi', 0x1234,\n",
    "                            min_x, min_y,\n",
    "                            max_x, max_y,\n",
    "                            len(polys)))\n",
    "        for poly in polys:\n",
    "            size = len(poly) * struct.calcsize('<dd')\n",
    "            f.write(struct.pack('<i', size + 4))\n",
    "            for pt in poly:\n",
    "                f.write(struct.pack('<dd', *pt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将数据读取回来的时候，可以利用函数 struct.unpack() ，代码很相似，基本就是上面写操作的逆序。如下：\n",
    "def read_polys(filename):\n",
    "    with open(filename, 'rb') as f:\n",
    "        # Read the header\n",
    "        header = f.read(40)\n",
    "        file_code, min_x, min_y, max_x, max_y, num_polys = \\\n",
    "            struct.unpack('<iddddi', header)\n",
    "        polys = []\n",
    "        for n in range(num_polys):\n",
    "            pbytes, = struct.unpack('<i', f.read(4))\n",
    "            poly = []\n",
    "            for m in range(pbytes // 16):\n",
    "                pt = struct.unpack('<dd', f.read(16))\n",
    "                poly.append(pt)\n",
    "            polys.append(poly)\n",
    "    return polys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13. 数据的累加与统计操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/lidianxiang/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:4: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support skipfooter; you can avoid this warning by specifying engine='python'.\n",
      "  after removing the cwd from sys.path.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Creation Date</th>\n",
       "      <th>Status</th>\n",
       "      <th>Completion Date</th>\n",
       "      <th>Service Request Number</th>\n",
       "      <th>Type of Service Request</th>\n",
       "      <th>Number of Premises Baited</th>\n",
       "      <th>Number of Premises with Garbage</th>\n",
       "      <th>Number of Premises with Rats</th>\n",
       "      <th>Current Activity</th>\n",
       "      <th>Most Recent Action</th>\n",
       "      <th>...</th>\n",
       "      <th>Police District</th>\n",
       "      <th>Community Area</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Location</th>\n",
       "      <th>Historical Wards 2003-2015</th>\n",
       "      <th>Zip Codes</th>\n",
       "      <th>Community Areas</th>\n",
       "      <th>Census Tracts</th>\n",
       "      <th>Wards</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12/18/2018</td>\n",
       "      <td>Open</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18-03388501</td>\n",
       "      <td>Rodent Baiting/Rat Complaint</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>41.849080</td>\n",
       "      <td>-87.714923</td>\n",
       "      <td>(41.849080332575, -87.714922751048)</td>\n",
       "      <td>14.0</td>\n",
       "      <td>21569.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>755.0</td>\n",
       "      <td>28.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12/18/2018</td>\n",
       "      <td>Open</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18-03386546</td>\n",
       "      <td>Rodent Baiting/Rat Complaint</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>41.928771</td>\n",
       "      <td>-87.668625</td>\n",
       "      <td>(41.928771396163, -87.668625093921)</td>\n",
       "      <td>16.0</td>\n",
       "      <td>21190.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>743.0</td>\n",
       "      <td>40.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12/18/2018</td>\n",
       "      <td>Open</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18-03388055</td>\n",
       "      <td>Rodent Baiting/Rat Complaint</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>17.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>41.963051</td>\n",
       "      <td>-87.727885</td>\n",
       "      <td>(41.963051420227, -87.727885158144)</td>\n",
       "      <td>28.0</td>\n",
       "      <td>21869.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>257.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>12/18/2018</td>\n",
       "      <td>Open</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18-03388235</td>\n",
       "      <td>Rodent Baiting/Rat Complaint</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>41.952819</td>\n",
       "      <td>-87.667465</td>\n",
       "      <td>(41.952819075577, -87.667464758026)</td>\n",
       "      <td>13.0</td>\n",
       "      <td>21186.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>724.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12/18/2018</td>\n",
       "      <td>Open</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18-03388510</td>\n",
       "      <td>Rodent Baiting/Rat Complaint</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>8.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>41.791652</td>\n",
       "      <td>-87.798980</td>\n",
       "      <td>(41.791652100128, -87.798980091442)</td>\n",
       "      <td>35.0</td>\n",
       "      <td>22268.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>589.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  Creation Date Status Completion Date Service Request Number  \\\n",
       "0    12/18/2018   Open             NaN            18-03388501   \n",
       "1    12/18/2018   Open             NaN            18-03386546   \n",
       "2    12/18/2018   Open             NaN            18-03388055   \n",
       "3    12/18/2018   Open             NaN            18-03388235   \n",
       "4    12/18/2018   Open             NaN            18-03388510   \n",
       "\n",
       "        Type of Service Request  Number of Premises Baited  \\\n",
       "0  Rodent Baiting/Rat Complaint                        NaN   \n",
       "1  Rodent Baiting/Rat Complaint                        NaN   \n",
       "2  Rodent Baiting/Rat Complaint                        NaN   \n",
       "3  Rodent Baiting/Rat Complaint                        NaN   \n",
       "4  Rodent Baiting/Rat Complaint                        NaN   \n",
       "\n",
       "   Number of Premises with Garbage  Number of Premises with Rats  \\\n",
       "0                              NaN                           NaN   \n",
       "1                              NaN                           NaN   \n",
       "2                              NaN                           NaN   \n",
       "3                              NaN                           NaN   \n",
       "4                              NaN                           NaN   \n",
       "\n",
       "  Current Activity Most Recent Action  ... Police District  Community Area  \\\n",
       "0              NaN                NaN  ...            10.0            30.0   \n",
       "1              NaN                NaN  ...            19.0             7.0   \n",
       "2              NaN                NaN  ...            17.0            14.0   \n",
       "3              NaN                NaN  ...            19.0             6.0   \n",
       "4              NaN                NaN  ...             8.0            56.0   \n",
       "\n",
       "    Latitude  Longitude                             Location  \\\n",
       "0  41.849080 -87.714923  (41.849080332575, -87.714922751048)   \n",
       "1  41.928771 -87.668625  (41.928771396163, -87.668625093921)   \n",
       "2  41.963051 -87.727885  (41.963051420227, -87.727885158144)   \n",
       "3  41.952819 -87.667465  (41.952819075577, -87.667464758026)   \n",
       "4  41.791652 -87.798980  (41.791652100128, -87.798980091442)   \n",
       "\n",
       "   Historical Wards 2003-2015  Zip Codes  Community Areas  Census Tracts Wards  \n",
       "0                        14.0    21569.0             32.0          755.0  28.0  \n",
       "1                        16.0    21190.0             68.0          743.0  40.0  \n",
       "2                        28.0    21869.0             14.0          257.0  12.0  \n",
       "3                        13.0    21186.0             57.0          724.0  18.0  \n",
       "4                        35.0    22268.0             53.0          589.0   6.0  \n",
       "\n",
       "[5 rows x 25 columns]"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 对于任何涉及统计、时间序列以及其它相关技术的数据分析问题，都可以考虑使用Pandas库。\n",
    "import pandas\n",
    "\n",
    "rats = pandas.read_csv('311_Service_Requests_-_Rodent_Baiting_-_Historical.csv',skipfooter=1)\n",
    "rats.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(319186, 25)"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rats.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([nan, 'Dispatch Crew', 'Request Sanitation Inspector',\n",
       "       'FVI - Outcome', 'Inspect for Violation'], dtype=object)"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rats['Current Activity'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter the data\n",
    "crew_dispatched = rats[rats['Current Activity'] == 'Dispatch Crew']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "297375"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(crew_dispatched)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60618.0    17023\n",
       "60647.0    16153\n",
       "60629.0    12497\n",
       "60614.0    12061\n",
       "60657.0    10608\n",
       "60641.0     9803\n",
       "60636.0     9105\n",
       "60623.0     8896\n",
       "60609.0     8760\n",
       "60645.0     8673\n",
       "60634.0     8597\n",
       "60651.0     8573\n",
       "60638.0     8440\n",
       "60625.0     8246\n",
       "60622.0     8239\n",
       "60632.0     7972\n",
       "60620.0     7827\n",
       "60659.0     7578\n",
       "60639.0     7527\n",
       "60630.0     7146\n",
       "60608.0     6624\n",
       "60612.0     6602\n",
       "60624.0     6295\n",
       "60613.0     5973\n",
       "60621.0     5570\n",
       "60628.0     5538\n",
       "60640.0     5174\n",
       "60644.0     5078\n",
       "60619.0     4846\n",
       "60652.0     4223\n",
       "           ...  \n",
       "60660.0     4072\n",
       "60617.0     3862\n",
       "60643.0     3018\n",
       "60642.0     2853\n",
       "60616.0     2849\n",
       "60637.0     2722\n",
       "60610.0     2662\n",
       "60655.0     2328\n",
       "60646.0     2158\n",
       "60615.0     2085\n",
       "60607.0     1983\n",
       "60653.0     1831\n",
       "60649.0     1761\n",
       "60707.0     1664\n",
       "60631.0     1592\n",
       "60656.0     1355\n",
       "60605.0     1023\n",
       "60611.0      738\n",
       "60654.0      601\n",
       "60601.0      276\n",
       "60606.0      218\n",
       "60633.0      207\n",
       "60602.0      178\n",
       "60604.0      166\n",
       "60661.0      140\n",
       "60603.0      138\n",
       "60827.0       80\n",
       "60666.0       13\n",
       "0.0            3\n",
       "60635.0        2\n",
       "Name: ZIP Code, Length: 61, dtype: int64"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "crew_dispatched['ZIP Code'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "dates = crew_dispatched.groupby('Completion Date')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1985"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "date_counts = dates.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Completion Date\n",
       "01/01/2014      7\n",
       "01/02/2013     20\n",
       "01/02/2014     96\n",
       "01/02/2015      5\n",
       "01/02/2018     71\n",
       "01/03/2011      4\n",
       "01/03/2012    125\n",
       "01/03/2013     46\n",
       "01/03/2014     59\n",
       "01/03/2017    212\n",
       "dtype: int64"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date_counts[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Completion Date\n",
       "02/18/2013      1\n",
       "09/19/2011      1\n",
       "08/11/2018      1\n",
       "07/24/2016      1\n",
       "03/31/2014      1\n",
       "01/07/2011      1\n",
       "01/27/2015      1\n",
       "02/19/2014      1\n",
       "02/14/2014      1\n",
       "02/01/2011      1\n",
       "02/13/2017      1\n",
       "07/24/2012      2\n",
       "01/14/2011      2\n",
       "01/21/2013      2\n",
       "12/22/2015      3\n",
       "02/27/2014      3\n",
       "01/03/2011      4\n",
       "01/21/2015      4\n",
       "02/09/2018      4\n",
       "12/09/2011      4\n",
       "01/26/2015      5\n",
       "03/09/2015      5\n",
       "02/24/2016      5\n",
       "01/27/2011      5\n",
       "03/30/2015      5\n",
       "01/02/2015      5\n",
       "10/02/2014      6\n",
       "04/07/2015      7\n",
       "01/01/2014      7\n",
       "12/13/2011      7\n",
       "             ... \n",
       "09/16/2016    339\n",
       "07/18/2017    341\n",
       "08/18/2017    346\n",
       "10/03/2016    346\n",
       "09/29/2016    348\n",
       "09/05/2017    348\n",
       "09/26/2016    349\n",
       "12/09/2013    357\n",
       "09/08/2016    358\n",
       "11/08/2017    359\n",
       "06/16/2016    360\n",
       "08/08/2017    361\n",
       "11/18/2013    362\n",
       "11/13/2013    365\n",
       "09/15/2016    365\n",
       "11/07/2017    366\n",
       "11/06/2017    371\n",
       "08/16/2017    373\n",
       "10/13/2011    378\n",
       "11/28/2014    384\n",
       "06/07/2016    384\n",
       "10/14/2011    391\n",
       "08/17/2017    392\n",
       "10/11/2017    392\n",
       "11/12/2013    401\n",
       "10/14/2016    412\n",
       "10/07/2011    457\n",
       "07/06/2016    461\n",
       "11/01/2013    488\n",
       "09/09/2016    492\n",
       "Length: 1985, dtype: int64"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sort the counts\n",
    "date_counts.sort_values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Completion Date\n",
       "12/29/2017     40\n",
       "12/30/2011    150\n",
       "12/30/2013    107\n",
       "12/30/2014    168\n",
       "12/30/2015     31\n",
       "12/30/2016    142\n",
       "12/31/2012     59\n",
       "12/31/2013    104\n",
       "12/31/2014    105\n",
       "12/31/2015     53\n",
       "dtype: int64"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "date_counts[-10:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">> Pandas是一个拥有很多特性的大型函数库，我在这里不可能介绍完。 但是只要你需要去分析大型数据集合、对数据分组、计算各种统计量或其他类似任务的话，这个函数库真的值得你去看一看。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}