day06(동적 크롤링) 복습.ipynb

{"cells":[{"cell_type":"code","execution_count":98,"metadata":{},"outputs":[],"source":["import warnings\n","warnings.filterwarnings('ignore')"]},{"cell_type":"markdown","metadata":{"id":"JM5N_uD9lmAY"},"source":["# 4.Crawling\n","\n","스크레이핑(Scraping)이라고도 하며 웹 페이지 내의 데이터를 추출하는 것을 의미 <br>\n","데이터를 수집하기 위한 방법으로 많이 사용 <br>\n","크게 두 가지의 방법이 존재\n","1. 정적크롤링 <br>\n","정적 데이터를 수집하는 방법 <br>\n","정적 데이터란 페이지 내에 원하는 정보가 모두 들어남\n","\n","2. 동적크롤링 <br>\n","동적 데이터를 수집하는 방법 <br>\n","동적 데이터란 클릭, 로그인 등의 행위를 통해 원하는 데이터에 접근 가능 <br>\n","\n","\n","|    |정적 크롤링|동적 크롤링|\n","|----|--------|--------|\n","|방법 |주소 사용  |브라우저 사용|\n","|수집 범위|제한적  |제한 없음|\n","|속도|매우 빠름|매우 \n","<br>\n","\n","크롤링 시 사이트에서 크롤링을 허용하는지를 반드시 확인해야 함 <br>\n","robots.txt를 뒤에 붙여 확인 가능 <br>\n","강제는 아니나 이를 무시하면 추후 법률적 문제가 생길 수 있음\n","```\n","www.daum.net/robots.txt\n","User-agent: *\n","Disallow: /\n","```\n","*: All <br>\n","/: All Directories\n"]},{"cell_type":"markdown","metadata":{"id":"4JLYuAdAtoTc"},"source":["## 4.1 HTTP\n","\n","WWW(World Wide Web, W3) 상에서 정보를 주고받을 수 있는 프로토콜 <br>\n","클라이언트와 서버 사이에 이루어지는 요청/응답 프로토콜"]},{"cell_type":"markdown","metadata":{"id":"VFv-YAqLvQ4j"},"source":["### API\n","API(Application Programming Interface) <br>\n","- Application: 고유한 기능을 가진 모든 소프트웨어\n","- Interface: 두 애플리케이션 간의 규약 <br>\n","이 계약은 요청과 응답을 사용하여 두 애플리케이션이 서로 통신하는 방법을 정의합니다."]},{"cell_type":"markdown","metadata":{"id":"8MhlEUlxuRTs"},"source":["### REST\n","REST(Representational State Transfer): 자원을 이름으로 구분하여 해당 자원의 상태를 주고받는 것 <br>\n","\n","REST 구성\n","1. 자원(Resource): HTTP URI\n","2. 자원에 대한 행위(Verb): HTTP Method\n","3. 자원에 대한 행위의 내용(Representations): HTTP Message Pay Load\n","\n","\n","- HTTP URI(Uniform Resource Identifier)를 통해 자원(Resource)을 명시\n","- HTTP Method(POST, GET, PUT, DELETE)를 사용하여 URI에 대한 CRUD Operation을 적용 <br>\n","<br>\n","\n","HTTP Methods\n","- GET: 자원 검색\n","- POST: 자원 작성\n","- PUT: 자원 업데이트\n","- DELETE: 데이터 삭제\n","- HEAD: 자원 검색 (GET과 유사하나 상태 줄과 헤더만 반환)\n","- OPTIONS: 자원이 지원하고 있는 메소드의 취득\n","- PATCH: 자원 일부 수정 (PUT과 유사하나 일부만 수정)\n","- CONNECT: 자원의 터널 접속을 변경\n","- TRACE: 리소스에 대한 경로를 따라 메시지 루프백 테스트를 수행\n","<br>\n","\n","HTTP Status\n","1. 1xx(Informational): 요청 처리중\n","2. 2xx(Successful): 요청 정상 처리 <br>\n","200: 요청 성공\n","3. 3xx(Redirection): 요청을 완료하려면 추가 행동이 필요\n","4. 4xx(Client Error): 클라이언트 오류, 잘못된 문법등으로 요청을 수행할 수 없음 <br>\n","400: Bad Request, 클라이언트의 잘못된 요청으로 서버가 요청을 처리할 수 없음 <br>\n","401: Unauthorized, 해당 리소스에 대한 인증이 필요함 <br>\n","403: Forbidden, 서버가 요청을 이해했지만 승인을 거부함 <br>\n","404: Not Found, 리소스를 찾을 수 없음 <br>\n","5. 5xx(Server Error): 서버 오류"]},{"cell_type":"markdown","metadata":{"id":"QtWV6XNfwm65"},"source":["### REST API\n","REST의 원리를 따르는 API <br>\n","※ RESTful: REST의 원리를 따르는 시스템"]},{"cell_type":"markdown","metadata":{"id":"gIr-wYY_-L_r"},"source":["## 4.2 HTML\n","HTML(HyperText Markup Language)은 웹 페이지 표시를 위해 개발된 지배적인 마크업 언어 <br>\n","HTML은 웹 페이지 콘텐츠 안의 꺾쇠 괄호에 둘러싸인 \"태그\"로 되어있는 HTML 요소 형태로 작성 <br>\n","HTML은 웹 브라우저와 같은 HTML 처리 장치의 행동에 영향을 주는 자바스크립트, 본문과 그 밖의 항목의 외관과 배치를 정의하는 CSS 같은 스크립트를 포함하거나 불러올 수 있음 <br>\n","<br>\n","HTML 선택자: HTML에서는 다수의 동일한 태그가 존재하는데 각 태그를 구별할 수 있도록 선택자를 이용\n","```html\n","<div> \n","\t<div> \n","      <a> c </a> \n","      <span> c++ </span> \n","    </div> \n","    \n","    <div> \n","      <a> java </a> \n","      <span> python </span> \n","    </div> \n","</div>\n","```\n","\n","```html\n","<div id=\"contents\"> \n","\t<div class=\"data1\"> \n","      <span class=\"language\"> c++ </span> \n","      <span class=\"language\"> java </span> \n","      <span class=\"language\"> python </span> \n","  </div> \n","    \n","  <div class=\"data2\"> \n","      <a class=\"framework\"> tensorflow </a> \n","      <a class=\"framework\"> pytorch </a> \n","      <a class=\"framework\"> spring </a> \n","  </div> \n","</div>\n","```"]},{"cell_type":"markdown","metadata":{"id":"l5jkE-4Bltzz"},"source":["## 4.3 정적크롤링"]},{"cell_type":"markdown","metadata":{"id":"D3On4hA8-WD7"},"source":["#### 4.3.1 라이브러리"]},{"cell_type":"markdown","metadata":{"id":"d6joc_SRs3Lc"},"source":["##### 4.3.1.1 requests\n","\n","requests: 파이썬용 http 라이브러리 <br>\n","reference: https://requests.readthedocs.io/en/latest/\n","\n","메소드별 사용법\n","```python\n","GET: requests.get()\n","POST: requests.post()\n","PUT: requests.put()\n","DELETE: requests.delete()\n","```\n","\n","```python\n","import requests\n","\n","requests.get(\"https://jsonplaceholder.typicode.com/users/1\")\n","```"]},{"cell_type":"markdown","metadata":{"id":"KWYAKeA21UYG"},"source":["###### Response Body\n","\n","요청이 정상적으로 처리가 되면, response body에 요청한 데이터가 담겨져 옴. <br>\n","response body 크게 3가지 방식으로 읽을 수 있음 <br>\n","1. content: binary 원문을 읽음\n","```python\n","response.content\n","```\n","2. text: utf-8로 인코딩 된 문자열로 읽음\n","```python\n","response.text\n","```\n","3. json: 응답이 json이면 dict로 읽음\n","```python\n","response.json()\n","```\n","\n"]},{"cell_type":"markdown","metadata":{"id":"rTdgFrTr25Mq"},"source":["###### Request\n","\n","param: 주소에 포함된 변수를 담음<br>\n","ex) https://www.naver.com/post/12345 <br>\n","-> 12345 <br>\n","query: 주소 바깥? 이후의 주소를 담음<br>\n","ex) https://www.naver.com/post/post_id=12345&id=1 <br>\n","-> 12345, 1\n","body: XML, JSON 등의 데이터를 담음, 주소에서는 확인 불가<br>\n","<br>\n","\n","requests에서는 아래와 같이 사용\n","- get\n","```python\n","response = requests.get(\"https://naver.com/post\", params={\"post_id\": \"12345\", \"id\": \"1\"})\n","```\n","\n","- post, put: HTML 데이터 전송\n","```python\n","response = requests.get(\"https://naver.com/post\", data={\"post_id\": \"12345\", \"id\": \"1\"})\n","```\n","json 형태로도 요청 가능\n","```python\n","response = requests.get(\"https://naver.com/post\", json={\"post_id\": \"12345\", \"id\": \"1\"})\n","```"]},{"cell_type":"markdown","metadata":{"id":"NHAgX_3A5zg9"},"source":["###### headers\n","\n","일부 웹 사이트는 bot agent를 차단 <br>\n","이 경우 header의 user-agent를 아래와 같이 넘기면 해결 <br>\n","```python\n","requests.get(\"https://naver.com/post\", headers={'User-Agent': 'Mozilla 5.0'})\n","```"]},{"cell_type":"markdown","metadata":{"id":"wPAvkXkMtOfV"},"source":["##### 4.3.1.2 BeautifulSoup\n","\n","BeautifulSoup: html, xml 등으로부터 원하는 정보를 가지고 올 수 있도록 하는 라이브러리 <br>\n","reference: https://www.crummy.com/software/BeautifulSoup/bs4/doc/\n","\n","```python\n","import requests\n","from bs4 import BeautifulSoup\n","\n","response = requests.get(url)\n","bs = BeautifulSoup(response.text, 'lxml')\n","```"]},{"cell_type":"markdown","metadata":{"id":"ZFgq8Bon82_j"},"source":["###### Parser\n","\n","|parser|특징|설치|속도|사용방법|\n","|------|---|---|---|------|\n","|html.parser||기본|보통|BeautifulSoup(html_doc, 'html.parser')|\n","|lxml|xml 지원|lxml 필요|빠름|BeautifulSoup(html_doc, 'lxml')|\n","|xml|xml 지원|lxml 필요|빠름|BeautifulSoup(html_doc, 'xml')|\n","|html5lib|브라우저와 동일|html5lib 필요|느림|BeautifulSoup(html_doc, 'html5lib')|"]},{"cell_type":"markdown","metadata":{"id":"OBuY_GBRAb8t"},"source":["###### find\n","속성과 값을 이용하여 원하는 값을 찾음\n","\n","find: 매칭되는 값 중 상위 1개 반환\n","find_all: 매칭되는 전체 반환\n","\n","특정 태그 추출\n","```python\n","soup.find_all('p') # p 태그 추출\n","```\n","<br>\n","\n","특정 클래스 추출\n","```python\n","soup.find_all(class_='a') # a 클래스 추출\n","```\n","<br>\n","\n","특정 태그와 class 추출\n","```python\n","soup.find_all('p', attrs={'class': 'a'}) # p 태그와 a 클래스 모두를 갖는 값 추출\n","```\n","<br>\n","\n","특정 id 추출\n","```python\n","soup.find_all(id='b') # b id를 갖는 값 추출\n","```\n","<br>"]},{"cell_type":"markdown","metadata":{"id":"uP7Vzn9uAcB2"},"source":["###### select\n","CSS Selector로 태그를 찾아 반환 <br>\n","CSS에서 HTML을 태깅하는 방법을 활용 <br>\n","<br>\n","select_one: 매칭되는 값 중 상위 1개 반환 <br>\n","select: 매칭되는 전체 반환 <br>\n","<br>\n","\n","특정 태그 추출\n","```python\n","soup.select('p') # p 태그 추출\n","```\n","<br>\n","\n","특정 클래스 추출\n","```python\n","soup.select('.a') # a 클래스 추출\n","```\n","<br>\n","\n","특정 태그와 class 추출\n","```python\n","soup.select('p.a') # p 태그와 a 클래스 모두를 갖는 값 추출\n","```\n","<br>\n","\n","특정 id 추출\n","```python\n","soup.select('#b') # b id를 갖는 값 추출\n","```\n","<br>\n","\n","특정 태그와 id 추출\n","```python\n","soup.select('p#b') # p 태그와 b id 모두를 갖는 값 추출\n","```\n","<br>\n","\n","특정 태그와 class, id 모두 추출\n","```python\n","soup.select('p.a#b') # p 태그와 a 클래스 b id 모두 갖는 값 추출\n","```\n","<br>\n","\n","특정 태그 아래에 있는 태그 찾기\n","```python\n","soup.select('div p') # div 아래 p태그가 있는 값 추출\n","soup.select('div > p') # div 바로 아래 p태그가 있는 값 추출\n","soup.select(\"div > #link\") # div 바로 아래 link id가 있는 값 추출\n","```\n","<br>\n","\n","형제 태그 찾기\n","```python\n","soup.select(\"#link + .sister\") # link 태그와 형제 태그 중 바로 직후 1개\n","soup.select(\"#link ~ .sister\") # link 태그와 형제 태그 중 뒤에 태그 전부\n","```\n","<br>\n","\n","여러 태그 중 i번째 태그 추출\n","```python\n","soup.select('a:nth-of-type(i)') # 추출된 a태그 중 i번째 값 반환\n","```\n","<br>\n","<br>\n","\n","정규표현식 활용 <br>\n","```python\n","soup.select('[class~=a]') # class 속성 중 a를 포함하는 태그\n","soup.select('a[href]') # a 태그 중 href 속성이 존재하는 태그\n","soup.select('a[href=\"https://www.naver.com\"]') # a 태그 중 href 속성이 https://www.naver.com과 매칭되는 태그\n","soup.select('a[href^=\"https://\"]') # a 태그 중 href 속성이 https://로 시작하는 태그\n","soup.select('a[href$=\"ac.kr\"]') # a 태그 중 href 속성이 ac.kr로 끝나는 태그\n","soup.select('a[href*=\"naver\"]') # a 태그 중 href 속성 중 naver를 가지는 태그\n","```\n","\n","<br>\n","<br>\n","\n","출력\n","```python\n","soup.strings # 값 반환\n","soup.stripped_strings # 공백을 제거한 값 반환\n","```"]},{"cell_type":"markdown","metadata":{"id":"9Mbvujsc7lcB"},"source":["### 4.3.2 실습"]},{"cell_type":"markdown","metadata":{"id":"H-PeuwJaMk7r"},"source":["##### 예제\n","\n","```html\n","<div id=\"contents\"> \n","    <div class=\"data1\"> \n","      <span class=\"language\"> c++ </span> \n","      <span class=\"language\"> java </span> \n","      <span class=\"language\"> python </span> \n","  </div> \n","\n","  <div class=\"data2\"> \n","      <a class=\"framework\"> tensorflow </a> \n","      <a class=\"framework\"> pytorch </a> \n","      <a class=\"framework\"> spring </a> \n","  </div> \n","</div>\n","```"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["import pandas as pd\n","import requests\n","from bs4 import BeautifulSoup\n","from tqdm import tqdm"]},{"cell_type":"markdown","metadata":{"id":"KfuXXlLV7nSV"},"source":["#### 네이버 뉴스"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["response = requests.get(\n","    'https://news.naver.com/main/list.naver?mode=LPOD&mid=sec&oid=023&date=20221126',\n","    headers={\n","        'User-Agent': 'Mozilla 5.0'\n","    }\n","    )\n","bs = BeautifulSoup(response.text, 'lxml')    "]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["bs.select('dt > a')[0]"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# slicing odd index\n","titles_and_content_links = bs.select('dt > a')[1::2]"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# title\n","titles_and_content_links[0].text.strip()"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# content link\n","content_link = titles_and_content_links[0].get('href')"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# content\n","response = requests.get(content_link, headers={'User-Agent': 'Mozilla 5.0'})\n","bs = BeautifulSoup(response.text, 'lxml')\n","bs.select('#dic_area')[0].text.strip()"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# the only one article\n","pd.DataFrame({\n","    'date': '20221126',\n","    'press_name': '조선일보',\n","    'title': titles_and_content_links[0].text.strip(),\n","    'content': bs.select('#dic_area')[0].text.strip(),\n","    'content_link': content_link\n","}, index=[0])"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# titles\n","titles = [title_and_content_link.text.strip() for title_and_content_link in titles_and_content_links]\n","\n","# urls\n","urls = [title_and_content_link.get('href') for title_and_content_link in titles_and_content_links]"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# list comprehension 없이 본문 추출\n","contents = []\n","for url in tqdm(urls):\n","    response = requests.get(url, headers={'User-Agent': 'Mozilla 5.0'})\n","    bs = BeautifulSoup(response.text, 'lxml')\n","    content = bs.select('#dic_area')[0].text.strip()\n","\n","    contents.append(content)"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 함수화\n","def bs_object_from_url(url):\n","    response = requests.get(url, headers={'User-Agent': 'Mozilla 5.0'})\n","    bs = BeautifulSoup(response.text, 'lxml')\n","\n","    return bs"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# list comprehension으로 본문 추출\n","contents = [bs_object_from_url(url).select('#dic_area')[0].text.strip() for url in tqdm(urls)]"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["chosun = pd.DataFrame({\n","    'date': '20221126',\n","    'press_name': '조선일보',\n","    'title': titles,\n","    'content': contents,\n","    'content_url': urls\n","})"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["chosun"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 해당 날짜 모든 데이터 크롤링\n","url = 'https://news.naver.com/main/list.naver?mode=LPOD&mid=sec&oid=023&date=20221126&page=100000'\n","bs = bs_object_from_url(url)\n","# get last page num\n","last_page_num = int(bs.select('div.paging > strong')[0].text.strip())\n","\n","chosun = pd.DataFrame()\n","for page in tqdm(range(1, last_page_num + 1)):\n","    url = f'https://news.naver.com/main/list.naver?mode=LPOD&mid=sec&oid=023&date=20221126&page={page}'\n","    bs_page = bs_object_from_url(url)\n","\n","    title_and_content_links = bs_page.select('dt > a')[1::2]\n","    titles =  [title_and_content_link.text.strip() for title_and_content_link in title_and_content_links]\n","    content_urls = [title_and_content_link.get('href') for title_and_content_link in title_and_content_links]\n","    contents = [bs_object_from_url(url).select('#dic_area')[0].text.strip() for url in content_urls]\n","\n","    temp = pd.DataFrame({\n","        'date': '20221126',\n","        'press_name': '조선일보',\n","        'title': titles,\n","        'content': contents,\n","        'content_url': content_urls\n","    })\n","\n","    chosun = chosun.append(temp)"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 본문 태그 변화를 처리하기 위한 함수\n","def get_contents_from_rul(url):\n","    bs = bs_object_from_url(url)\n","\n","    if bs.select('#contents'):\n","        return bs_object_from_url(url).select('#contents')[0].text.strip()\n","\n","    if bs.select('.content'):\n","        return bs_object_from_url(url).select('.content')[0].text.strip()\n","\n","    return -1"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 주어진 날짜의 조선일보 데이터 수집\n","chosun = pd.DataFrame()\n","for date in tqdm(pd.date_range('20221125', '20221127')):\n","    date = date.strftime(\"%Y%m%d\")\n","\n","    for page in range(1, last_page_num + 1):\n","        url = f'https://news.naver.com/main/list.naver?mode=LPOD&mid=sec&oid=023&date={date}&page={page}'\n","        bs_page = bs_object_from_url(url)\n","\n","        title_and_content_links = bs_page.select('dt > a')[1::2]\n","        titles =  [title_and_content_link.text.strip() for title_and_content_link in title_and_content_links]\n","        content_urls = [title_and_content_link.get('href') for title_and_content_link in title_and_content_links]\n","        contents = [get_contents_from_rul(url) for url in content_urls]\n","\n","        temp = pd.DataFrame({\n","            'date': date,\n","            'press_name': '조선일보',\n","            'title': titles,\n","            'content': contents,\n","            'content_url': content_urls\n","        })\n","\n","        chosun = chosun.append(temp)"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["chosun"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 날짜, 신문사, 페이지 정보를 입력하여 해당 주소의 값 반환\n","def crawl_news(date, oid, page):\n","    url = f'https://news.naver.com/main/list.naver?mode=LPOD&mid=sec&oid={oid}&date={date}&page={page}'\n","    bs_page = bs_object_from_url(url)\n","\n","    title_and_content_links = bs_page.select('dt > a')[1::2]\n","    titles =  [title_and_content_link.text.strip() for title_and_content_link in title_and_content_links]\n","    content_urls = [title_and_content_link.get('href') for title_and_content_link in title_and_content_links]\n","    contents = [get_contents_from_rul(url) for url in content_urls]\n","\n","    temp = pd.DataFrame({\n","        'date': date,\n","        'press_name': oid,\n","        'title': titles,\n","        'content': contents,\n","        'content_url': content_urls\n","    })\n","\n","    return temp"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 11.26 ~ 11.27일 조선일보, 한국경제 뉴스 크롤링\n","news = pd.DataFrame()\n","for oid in ('023', '015'):    \n","    for date in pd.date_range('20221126', '20221127'):\n","        date = date.strftime(\"%Y%m%d\")\n","        \n","        url = f'https://news.naver.com/main/list.naver?mode=LPOD&mid=sec&oid={oid}&date={date}&page=100000'\n","        bs = bs_object_from_url(url)\n","        last_page_num = int(bs.select('div.paging > strong')[0].text.strip())\n","\n","        for page in range(1, last_page_num + 1):\n","            temp = crawl_news(date, oid, page)\n","\n","            news = news.append(temp)"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["news"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 숙제\n","#   - 뉴스 크롤링\n","# news (2021년 네이버 언론사 모바일 랭킹뉴스 매체별 점유율, 기자협회보)\n","# 2022-10-01 ~ 2022-11-30\n","# 중앙일보, 한국경제, 조선일보, 매일경제, 머니투데이"]},{"cell_type":"markdown","metadata":{"id":"P4-hafLg7ogb"},"source":["#### 네이버 증권"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["import pandas as pd\n","import requests\n","from bs4 import BeautifulSoup\n","from tqdm import tqdm"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 삼성, 카카오, 카카오뱅크, 이지케어택, 네이버, 하이닉스, 엘지에너지솔루션, 현대차, SPC삼립, 셀트리온의\n","# 종목명, 주식코드, 주가, 거래량, 2021 당기순익(연간)을 추출해 데이터프레임 생성\n","stock_name = '삼성전자'\n","stock_code = '005930'\n","\n","response = requests.get('https://finance.naver.com/item/main.naver?code=005930')\n","bs = BeautifulSoup(response.text, 'lxml')"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["stock_price = bs.select('p.no_today')\n","stock_price = int(stock_price[0].select('span.blind')[0].text.strip().replace(',', ''))\n","\n","stock_volume = bs.select('div.rate_info table tr')\n","stock_volume = int(stock_volume[0].select('td')[2].select('span.blind')[0].text.strip().replace(',', ''))\n","\n","operating_profit = bs.select('div.sub_section table.tb_type1.tb_num.tb_type1_ifrs tbody')\n","operating_profit = int(operating_profit[0].select('tr')[2].select('td')[2].text.strip().replace(',', ''))"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["stock_info = {\n","    'stock_name': stock_name,\n","    'stock_code': stock_code,\n","    'stock_price': stock_price,\n","    'stock_volume': stock_volume,\n","    'operating_profit': operating_profit\n","}\n","\n","pd.DataFrame(stock_info, index=[0])"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["f'https://finance.naver.com/item/main.naver?code=099750'"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 전체 주식\n","stocks = {\n","    '삼성전자': '005930', \n","    '카카오': '035720', \n","    '카카오뱅크': '323410', \n","    '이지케어텍': '099750',\n","    '네이버': '035420', \n","    '하이닉스': '000660', \n","    '엘지에너지솔루션': '373220', \n","    '현대차': '005380', \n","    'SPC삼립': '005610', \n","    '셀트리온': '068270', \n","}\n","\n","stock_df = pd.DataFrame()\n","for i, (stock_name, stock_code) in enumerate(stocks.items()):\n","    response = requests.get(f'https://finance.naver.com/item/main.naver?code={stock_code}')\n","    bs = BeautifulSoup(response.text, 'lxml')\n","\n","    stock_price = bs.select('p.no_today')\n","    stock_price = int(stock_price[0].select('span.blind')[0].text.strip().replace(',', ''))\n","\n","    stock_volume = bs.select('div.rate_info table tr')\n","    stock_volume = int(stock_volume[0].select('td')[2].select('span.blind')[0].text.strip().replace(',', ''))\n","\n","    operating_profit = bs.select('div.sub_section table.tb_type1.tb_num.tb_type1_ifrs tbody')\n","    operating_profit = int(operating_profit[0].select('tr')[2].select('td')[2].text.strip().replace(',', ''))\n","\n","    stock_info = {\n","        'stock_name': stock_name,\n","        'stock_code': stock_code,\n","        'stock_price': stock_price,\n","        'stock_volume': stock_volume,\n","        'operating_profit': operating_profit\n","    }\n","\n","    stock_df = stock_df.append(pd.DataFrame(stock_info, index=[i]))"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["stock_df"]},{"cell_type":"markdown","metadata":{"id":"RtbLMtDR7qoS"},"source":["#### 다음 증권"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["import json"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["response = requests.get('https://finance.daum.net/domestic/market_cap')\n","bs = BeautifulSoup(response.text, 'lxml')"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["response = requests.get('https://finance.daum.net/api/trend/market_capitalization?page=1&perPage=30&fieldName=marketCap&order=desc&market=KOSPI&pagination=true',\n","headers={\n","    'User-Agent': 'Mozilla 5.0',\n","    'referer': 'https://finance.daum.net/domestic/market_cap'})\n","bs = BeautifulSoup(response.text, 'lxml')"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["pd.DataFrame(json.loads(bs.text).get('data'))"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["import selenium"]},{"cell_type":"markdown","metadata":{"id":"-7C41NNpyFmk"},"source":["## 4.4 동적크롤링\n","\n","reference: https://selenium-python.readthedocs.io/index.html"]},{"cell_type":"markdown","metadata":{"id":"7bdVT9m-yMv_"},"source":["### 4.4.1 설치 방법\n","\n","```cmd\n","pip install selenium\n","```\n","\n","Chrome Driver: https://chromedriver.chromium.org/downloads"]},{"cell_type":"markdown","metadata":{"id":"85-KzW4Qyzbe"},"source":["### 4.4.2 사용법\n","\n","기본적인 사용 방법은 아래와 같음\n","```python\n","from selenium import webdriver\n"," \n","driver = webdriver.Chrome(executable_path='chromedriver') # executable_path: chromedriver path\n","# driver = webdriver.Firefox()\n","\n","driver.get(url=\"http://www.naver.com\")\n","\n","driver.close()\n","```"]},{"cell_type":"markdown","metadata":{"id":"wZ83Z28w5_CU"},"source":["#### 4.4.2.1 Locating Elements\n","\n","find_element: 조건에 해당하는 하나의 값 반환  \n","find_elements: 조건에 해당하는 다수의 값 반환\n","\n","element를 찾는 method는 아래와 같음\n","- by_ID\n","- by_xpath\n","- by_link_text\n","- by_partial_link_text\n","- by_name\n","- by_tag_name\n","- by_class_name\n","- by_css_selector\n","\n","```python\n","element = driver.find_element(By.XPATH, x_path)\n","element = driver.find_element_by_xpath(x_path)\n","\n","element = driver.find_element(By.CSS_SELECTOR, css)\n","element = driver.find_element_by_css_selector(css)\n","```\n","\n","해당 값을 못 찾은 경우 NoSuchElementException 발생"]},{"cell_type":"markdown","metadata":{"id":"yMknm_5O8naC"},"source":["#### 4.4.2.2 Navigating"]},{"cell_type":"markdown","metadata":{"id":"TjtDgEhn_Ode"},"source":["###### Send Keys\n","```python\n","driver.find_element_by_xpath(xpath).send_keys(id)\n","driver.find_element_by_xpath(xpath).send_keys(password)\n","driver.find_element_by_xpath(xpath).click()\n","```\n","\n","외에도 Keys 내의 다양한 커맨드 입력 가능\n","```\n","from selenium.webdriver.common.keys import Keys\n","```\n","|Command          |Action |\n","|-----------------|-------|\n","|Keys.ENTER       |엔터    |\n","|Keys.RRETURN     |       |\n","|Keys.SPACE       |스페이스 |\n","|Keys.ARROW_UP    |화살표   |\n","|Keys.ARROW_DOWN  |       |\n","|Keys.ARROW_LEFT  |       |\n","|Keys.ARROW_RIGHT |       |\n","|Keys.BACK_SPACE  |지우기   |\n","|Keys.DELETE      |       |\n","|Keys.CONTROL     |Ctrl   |\n","|Keys.ALT         |Alt    |\n","|Keys.SHIFT       |Shift  |\n","|Keys.TAB         |Tab    |\n","|Keys.PAGE_UP     |Page-Up|\n","|Keys.PAGE_DOWN   |Page-Down|\n","|Keys.TAB         |Tab    |\n","|Keys.F1 ~ Keys.F9|F1~F9  |\n","|Keys.ESCAPE      |ESC    |\n","|Keys.HOME        |Home   |\n","|Keys.INSERT      |Insert |\n","|...              |...    |"]},{"cell_type":"markdown","metadata":{"id":"-trPTDFYFizh"},"source":["##### Drag and Drop\n","\n","```python\n","from selenium.webdriver import ActionChains\n","\n","element = driver.find_element(By.NAME, \"source\")\n","target = driver.find_element(By.NAME, \"target\")\n","\n","action_chains = ActionChains(driver)\n","action_chains.drag_and_drop(element, target).perform()\n","```"]},{"cell_type":"markdown","metadata":{"id":"MM03w3DP_sbO"},"source":["##### Moving between Windows\n","```python\n","previous_window = driver.window_handles[0]\n","new_window = driver.window_handles[1]\n","driver.switch_to.window(new_window)\n","```\n"]},{"cell_type":"markdown","metadata":{"id":"tV8duA5x_ip6"},"source":["##### Moving\n","```python\n","driver.forward()\n","driver.back()\n","```"]},{"cell_type":"markdown","metadata":{"id":"ChaKIr0C1hXj"},"source":["#### 4.4.2.3 Waiting\n","\n","Selenium은 실행 중인 driver를 이용하여 정보를 추출  \n","driver에 데이터 로딩이 완료되지 않을 경우 데이터 수집이 불가  \n","따라서 페이지 이동 등의 작업이 들어가면 다음 작업이 완료될 때까지 대기해야만 함"]},{"cell_type":"markdown","metadata":{"id":"3GCUZ0VT2rq5"},"source":["##### Time\n","\n","가장 기본적인 방법으로 대기 시간을 time 모듈을 통해 명시\n","\n","```python\n","import time\n","from selenium import webdriver\n","\n","\n","driver = webdriver.Chrome(executable_path='chromedriver')\n","driver.get(url=\"http://www.naver.com\")\n","time.time(5)\n","driver.close()\n","```"]},{"cell_type":"markdown","metadata":{"id":"udDE0BlT2sio"},"source":["##### Implicit Wait\n","\n","위의 time에서는 2가지 문제가 존재\n","1. 모든 반응형 동작마다 time을 걸어 웹 로딩 대기\n","2. time을 5초로 설정했지만 실제 그보다 동작이 빨리 끝날 경우 불필요한 시간 대기\n","\n","위의 문제를 해결하고자 implicit wait을 이용  \n","이는 driver의 옵션 설정으로 반응형 동작에서 최대 허용 대기 시간을 설정  \n","만약 허용 시간 이내에 로드가 완료될 경우 다음 작업을 바로 진행\n","\n","```python\n","from selenium import webdriver\n","\n","\n","driver = webdriver.Chrome(executable_path='chromedriver')\n","driver.implicitly_wait(time_to_wait=5)\n","driver.get(url=\"http://www.naver.com\")\n","driver.close()\n","\n","```"]},{"cell_type":"markdown","metadata":{"id":"vL0p1lkR2wgD"},"source":["##### Explicit Wait\n","\n","implicit wait에서는 driver가 로딩을 대기하는 시간을 설정하여 5초를 설정하였어도 이전에 로드가 완료되면 다음 작업의 진행이 가능하였음  \n","하지만 위 경우도 문제가 있는데 5초가 지나도 내가 원하는 데이터가 로드가 안 되었을 경우 데이터를 수집할 수 없음  \n","따라서 특정 값이 로드될 때까지 기다렸다가 그 값이 로드되면 다음 작업을 진행할 필요성이 발생  \n","이 때 이용하는 게 explicit wait\n","\n","```python\n","from selenium import webdriver\n","from selenium.webdriver.common.by import By\n","from selenium.webdriver.support.ui import WebDriverWait\n","from selenium.webdriver.support import expected_conditions as EC\n","\n","driver = webdriver.Chrome(executable_path='chromedriver') \n","driver.get(url=\"http://www.naver.com\")\n","\n","try:\n","    element = WebDriverWait(driver, 5).until(\n","        EC.presence_of_element_located((By.CLASS_NAME , 'paging'))\n","    )\n","finally:\n","    driver.quit()\n","\n","driver.close()\n","\n","```\n","\n","until: 조건이 False인 동안에 계속 실행  \n","not_until: 조건이 True인 동안에 계속 실행"]},{"cell_type":"markdown","metadata":{"id":"cK4xweopF-EB"},"source":["#### 4.4.2.4 ActionChains\n","\n","연속 동작을 수행하기 위함\n","\n","ex) control + c\n","```python\n","ActionChains(driver).key_down(Keys.CONTROL).send_keys('c').key_up(Keys.CONTROL).perform()\n","```"]},{"cell_type":"markdown","metadata":{"id":"UG7rKFfjGUV-"},"source":["##### Drag and Drop\n","\n","```python\n","from selenium.webdriver import ActionChains\n","\n","element = driver.find_element(By.NAME, \"source\")\n","target = driver.find_element(By.NAME, \"target\")\n","\n","action_chains = ActionChains(driver)\n","action_chains.drag_and_drop(element, target).perform()\n","```"]},{"cell_type":"markdown","metadata":{"id":"W1ga1GbjETUK"},"source":["#### 4.4.2.5 Others"]},{"cell_type":"markdown","metadata":{"id":"mVq-Qtu8E5EL"},"source":["##### Options\n","```python\n","options = webdriver.ChromeOptions()\n","options.add_argument('window-size=1920,1080')\n","\n","driver = webdriver.Chrome(executable_path, options=options)\n","```"]},{"cell_type":"markdown","metadata":{"id":"z8noDRwzG_yJ"},"source":["##### Alert\n","\n","경고창 발생 시 이에 수락, 거절 등의 행동을 취할 수 있음\n","\n","```python\n","from selenium.webdriver.common.alert import Alert\n","\n","Alert(driver).accept() # 수락\n","Alert(driver).dismiss() # 거절\n","Alert(driver).send_keys(keysToSend=key) # 특정 키를 보낼 수 있음\n","```"]},{"cell_type":"markdown","metadata":{"id":"u3ELD7eoEUe6"},"source":["##### Scroll Down\n","페이지의 최하단으로 이동\n","```python\n","driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')\n","```\n","<br>\n","\n","특정 태그가 등장할 때까지 이동\n","```python\n","from selenium.webdriver import ActionChains\n","\n","some_tag = driver.find_element_by_id('gorio')\n","ActionChains(driver).move_to_element(some_tag).perform()\n","```"]},{"cell_type":"markdown","metadata":{"id":"393j4ATBEmCZ"},"source":["##### Minimize/Maximize\n","\n","```python\n","driver.minimize_window()\n","driver.maximize_window()\n","```"]},{"cell_type":"markdown","metadata":{"id":"yReb_-aMEtlZ"},"source":["##### Screen Shot\n","```python\n","driver.save_screenshot('screenshot.png')\n","```"]},{"cell_type":"code","execution_count":7,"metadata":{},"outputs":[{"name":"stderr","output_type":"stream","text":["C:\\Users\\user\\AppData\\Local\\Temp\\ipykernel_18504\\403980170.py:8: DeprecationWarning: executable_path has been deprecated, please pass in a Service object\n","  driver = webdriver.Chrome(executable_path='./utils/chromedriver') # executable_path: chromedriver path\n"]}],"source":["# 실습\n","from selenium import webdriver\n","from selenium.webdriver.common.by import By\n","from selenium.webdriver.support.ui import WebDriverWait\n","from selenium.webdriver.support import expected_conditions as EC\n","\n","driver = webdriver.Chrome(executable_path='./utils/chromedriver') # executable_path: chromedriver path"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["driver.get(url=\"http://www.naver.com\")"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 네이버에서 검색하기\n","# 검색어 입력\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[2]/div[2]/div[1]/div/div[3]/form/fieldset/div/input'\n","    ).send_keys('월드컵')\n",")"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 클릭하기\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[2]/div[2]/div[1]/div/div[3]/form/fieldset/button'\n","    ).click()\n",")"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 사이트 정보 가지고 오기\n","response = driver.page_source\n","bs = BeautifulSoup(response, 'lxml')"]},{"cell_type":"code","execution_count":8,"metadata":{},"outputs":[],"source":["# 기업집단포털 \n","driver.get(url=\"https://www.egroup.go.kr/egps/wi/mainPage.do\")"]},{"cell_type":"code","execution_count":9,"metadata":{},"outputs":[],"source":["(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[2]/div[7]/header/div/div/div[2]/div/a'\n","    ).click()\n",")"]},{"cell_type":"code","execution_count":10,"metadata":{},"outputs":[],"source":["(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[2]/div[7]/div/div/div/div/div[1]/ul/li[2]/ul/li[2]/a'\n","    ).click()\n",")"]},{"cell_type":"code","execution_count":11,"metadata":{},"outputs":[],"source":["(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[2]/div[3]/div[3]/div/div[2]/div/ul/li[2]/a'\n","    ).click()\n",")\n","response = driver.page_source"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["bs = BeautifulSoup(response, 'lxml')\n","bs.select('table')[0].select('th')"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 로드 완료 후 \n","response = driver.page_source\n","bs = BeautifulSoup(response, 'lxml')\n","bs.select('table')[0].select('th')"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 대기\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[2]/div[3]/div[3]/div/div[2]/div/ul/li[2]/a'\n","    ).click()\n",")\n","\n","(\n","    WebDriverWait(driver, 10)\n","    .until(\n","        EC.element_to_be_clickable((\n","            By.XPATH,\n","            '/html/body/div[2]/div[3]/div[3]/div/form/div/div[2]/div/div/div/table/tbody/tr[1286]/td[1]'\n","            ))\n","        )\n",")\n","\n","response = driver.page_source\n","bs = BeautifulSoup(response, 'lxml')\n","bs.select('table')[0].select('th')"]},{"cell_type":"markdown","metadata":{},"source":["dart"]},{"cell_type":"code","execution_count":48,"metadata":{},"outputs":[],"source":["from bs4 import BeautifulSoup\n","from tqdm import tqdm"]},{"cell_type":"code","execution_count":15,"metadata":{},"outputs":[{"name":"stderr","output_type":"stream","text":["C:\\Users\\user\\AppData\\Local\\Temp\\ipykernel_18504\\3866069677.py:1: DeprecationWarning: executable_path has been deprecated, please pass in a Service object\n","  driver = webdriver.Chrome(executable_path='./utils/chromedriver') # executable_path: chromedriver path\n"]}],"source":["driver = webdriver.Chrome(executable_path='./utils/chromedriver') # executable_path: chromedriver path"]},{"cell_type":"code","execution_count":41,"metadata":{},"outputs":[{"ename":"StaleElementReferenceException","evalue":"Message: stale element reference: element is not attached to the page document\n  (Session info: chrome=108.0.5359.95)\nStacktrace:\nBacktrace:\n\t(No symbol) [0x0109F243]\n\t(No symbol) [0x01027FD1]\n\t(No symbol) [0x00F1D04D]\n\t(No symbol) [0x00F1FD34]\n\t(No symbol) [0x00F1FBE5]\n\t(No symbol) [0x00F1FE80]\n\t(No symbol) [0x00F4DA74]\n\t(No symbol) [0x00F4311D]\n\t(No symbol) [0x00F6858C]\n\t(No symbol) [0x00F42BFF]\n\t(No symbol) [0x00F68804]\n\t(No symbol) [0x00F7C9EB]\n\t(No symbol) [0x00F68386]\n\t(No symbol) [0x00F4163C]\n\t(No symbol) [0x00F4269D]\n\tGetHandleVerifier [0x01339A22+2655074]\n\tGetHandleVerifier [0x0132CA24+2601828]\n\tGetHandleVerifier [0x01148C0A+619850]\n\tGetHandleVerifier [0x01147830+614768]\n\t(No symbol) [0x010305FC]\n\t(No symbol) [0x01035968]\n\t(No symbol) [0x01035A55]\n\t(No symbol) [0x0104051B]\n\tBaseThreadInitThunk [0x7762FEF9+25]\n\tRtlGetAppContainerNamedObjectPath [0x777F7BBE+286]\n\tRtlGetAppContainerNamedObjectPath [0x777F7B8E+238]\n","output_type":"error","traceback":["\u001b[1;31m---------------------------------------------------------------------------\u001b[0m","\u001b[1;31mStaleElementReferenceException\u001b[0m            Traceback (most recent call last)","\u001b[1;32mc:\\Users\\user\\Documents\\github\\2022_11_19_bigdata\\bigdata1\\day06.ipynb 셀 92\u001b[0m in \u001b[0;36m<cell line: 84>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=65'>66</a>\u001b[0m (\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=66'>67</a>\u001b[0m     driver\u001b[39m.\u001b[39mfind_element(\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=67'>68</a>\u001b[0m         By\u001b[39m.\u001b[39mXPATH,\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=68'>69</a>\u001b[0m         \u001b[39m'\u001b[39m\u001b[39m/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[2]/div[2]/div[2]/a[1]\u001b[39m\u001b[39m'\u001b[39m\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=69'>70</a>\u001b[0m     )\u001b[39m.\u001b[39mclick()\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=70'>71</a>\u001b[0m )\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=72'>73</a>\u001b[0m (\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=73'>74</a>\u001b[0m     WebDriverWait(driver, \u001b[39m10\u001b[39m)\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=74'>75</a>\u001b[0m     \u001b[39m.\u001b[39muntil(\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=79'>80</a>\u001b[0m         )\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=80'>81</a>\u001b[0m )\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=82'>83</a>\u001b[0m (\n\u001b[1;32m---> <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=83'>84</a>\u001b[0m     driver\u001b[39m.\u001b[39;49mfind_element(\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=84'>85</a>\u001b[0m         By\u001b[39m.\u001b[39;49mXPATH,\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=85'>86</a>\u001b[0m         \u001b[39m'\u001b[39;49m\u001b[39m/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[3]/div/div[1]/table/tbody/tr/td[3]/a\u001b[39;49m\u001b[39m'\u001b[39;49m\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=86'>87</a>\u001b[0m     )\u001b[39m.\u001b[39;49mclick()\n\u001b[0;32m     <a href='vscode-notebook-cell:/c%3A/Users/user/Documents/github/2022_11_19_bigdata/bigdata1/day06.ipynb#Y205sZmlsZQ%3D%3D?line=87'>88</a>\u001b[0m )\n","File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python39\\site-packages\\selenium\\webdriver\\remote\\webelement.py:88\u001b[0m, in \u001b[0;36mWebElement.click\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m     86\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mclick\u001b[39m(\u001b[39mself\u001b[39m) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m \u001b[39mNone\u001b[39;00m:\n\u001b[0;32m     87\u001b[0m     \u001b[39m\"\"\"Clicks the element.\"\"\"\u001b[39;00m\n\u001b[1;32m---> 88\u001b[0m     \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_execute(Command\u001b[39m.\u001b[39;49mCLICK_ELEMENT)\n","File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python39\\site-packages\\selenium\\webdriver\\remote\\webelement.py:396\u001b[0m, in \u001b[0;36mWebElement._execute\u001b[1;34m(self, command, params)\u001b[0m\n\u001b[0;32m    394\u001b[0m     params \u001b[39m=\u001b[39m {}\n\u001b[0;32m    395\u001b[0m params[\u001b[39m'\u001b[39m\u001b[39mid\u001b[39m\u001b[39m'\u001b[39m] \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_id\n\u001b[1;32m--> 396\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_parent\u001b[39m.\u001b[39;49mexecute(command, params)\n","File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python39\\site-packages\\selenium\\webdriver\\remote\\webdriver.py:428\u001b[0m, in \u001b[0;36mWebDriver.execute\u001b[1;34m(self, driver_command, params)\u001b[0m\n\u001b[0;32m    426\u001b[0m response \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mcommand_executor\u001b[39m.\u001b[39mexecute(driver_command, params)\n\u001b[0;32m    427\u001b[0m \u001b[39mif\u001b[39;00m response:\n\u001b[1;32m--> 428\u001b[0m     \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49merror_handler\u001b[39m.\u001b[39;49mcheck_response(response)\n\u001b[0;32m    429\u001b[0m     response[\u001b[39m'\u001b[39m\u001b[39mvalue\u001b[39m\u001b[39m'\u001b[39m] \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_unwrap_value(\n\u001b[0;32m    430\u001b[0m         response\u001b[39m.\u001b[39mget(\u001b[39m'\u001b[39m\u001b[39mvalue\u001b[39m\u001b[39m'\u001b[39m, \u001b[39mNone\u001b[39;00m))\n\u001b[0;32m    431\u001b[0m     \u001b[39mreturn\u001b[39;00m response\n","File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python39\\site-packages\\selenium\\webdriver\\remote\\errorhandler.py:243\u001b[0m, in \u001b[0;36mErrorHandler.check_response\u001b[1;34m(self, response)\u001b[0m\n\u001b[0;32m    241\u001b[0m         alert_text \u001b[39m=\u001b[39m value[\u001b[39m'\u001b[39m\u001b[39malert\u001b[39m\u001b[39m'\u001b[39m]\u001b[39m.\u001b[39mget(\u001b[39m'\u001b[39m\u001b[39mtext\u001b[39m\u001b[39m'\u001b[39m)\n\u001b[0;32m    242\u001b[0m     \u001b[39mraise\u001b[39;00m exception_class(message, screen, stacktrace, alert_text)  \u001b[39m# type: ignore[call-arg]  # mypy is not smart enough here\u001b[39;00m\n\u001b[1;32m--> 243\u001b[0m \u001b[39mraise\u001b[39;00m exception_class(message, screen, stacktrace)\n","\u001b[1;31mStaleElementReferenceException\u001b[0m: Message: stale element reference: element is not attached to the page document\n  (Session info: chrome=108.0.5359.95)\nStacktrace:\nBacktrace:\n\t(No symbol) [0x0109F243]\n\t(No symbol) [0x01027FD1]\n\t(No symbol) [0x00F1D04D]\n\t(No symbol) [0x00F1FD34]\n\t(No symbol) [0x00F1FBE5]\n\t(No symbol) [0x00F1FE80]\n\t(No symbol) [0x00F4DA74]\n\t(No symbol) [0x00F4311D]\n\t(No symbol) [0x00F6858C]\n\t(No symbol) [0x00F42BFF]\n\t(No symbol) [0x00F68804]\n\t(No symbol) [0x00F7C9EB]\n\t(No symbol) [0x00F68386]\n\t(No symbol) [0x00F4163C]\n\t(No symbol) [0x00F4269D]\n\tGetHandleVerifier [0x01339A22+2655074]\n\tGetHandleVerifier [0x0132CA24+2601828]\n\tGetHandleVerifier [0x01148C0A+619850]\n\tGetHandleVerifier [0x01147830+614768]\n\t(No symbol) [0x010305FC]\n\t(No symbol) [0x01035968]\n\t(No symbol) [0x01035A55]\n\t(No symbol) [0x0104051B]\n\tBaseThreadInitThunk [0x7762FEF9+25]\n\tRtlGetAppContainerNamedObjectPath [0x777F7BBE+286]\n\tRtlGetAppContainerNamedObjectPath [0x777F7B8E+238]\n"]}],"source":["driver.get(url=\"https://dart.fss.or.kr/main.do\")\n","\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[2]/div[1]/div[3]/div/div[2]/form/div[1]/div[2]/span[2]/div/input'\n","    ).send_keys('삼성전자')\n",")\n","\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[2]/div[1]/div[3]/div/div[2]/form/div[1]/div[3]/a'\n","    ).click()\n",")\n","\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[2]/div[2]/div[1]/ul/li[4]/div/span/ul/li[1]/label[1]/img'\n","    ).click()\n",")\n","\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[2]/div[2]/div[1]/ul/li[4]/div/span/ul/li[1]/label[1]/span[2]'\n","    ).click()\n",")\n","\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[2]/div[2]/div[3]/div[1]/ul/li[1]/span/input'\n","    ).click()\n",")\n","\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[2]/div[2]/div[1]/ul/li[4]/div/span/ul/li[1]/label[1]/img'\n","    ).click()\n",")\n","\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[2]/div[2]/div[1]/ul/li[4]/div/span/ul/li[1]/label[1]/img'\n","    ).click()\n",")\n","\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[2]/div[2]/div[2]/a[1]'\n","    ).click()\n",")\n","\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[2]/div[2]/div[3]/div[1]/ul/li[1]/span/input'\n","    ).click()\n",")\n","\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[2]/div[2]/div[2]/a[1]'\n","    ).click()\n",")\n","\n","(\n","    WebDriverWait(driver, 10)\n","    .until(\n","        EC.element_to_be_clickable((\n","            By.XPATH,\n","            '/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[3]/div'\n","            ))\n","        )\n",")\n","\n","(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[3]/div/div[1]/table/tbody/tr/td[3]/a'\n","    ).click()\n",")\n"]},{"cell_type":"code","execution_count":42,"metadata":{},"outputs":[],"source":["(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[4]/div[2]/div[1]/div[2]/div[2]/form[3]/div/div[1]/table/tbody/tr/td[3]/a'\n","    ).click()\n",")\n"]},{"cell_type":"code","execution_count":50,"metadata":{},"outputs":[{"data":{"text/plain":["[]"]},"execution_count":50,"metadata":{},"output_type":"execute_result"}],"source":["response = driver.page_source\n","bs = BeautifulSoup(response, 'lxml')\n","bs.select('div.leftPanelWrap')"]},{"cell_type":"code","execution_count":51,"metadata":{},"outputs":[],"source":["# 컨트롤 윈도우 변경\n","driver.switch_to.window(driver.window_handles[1])"]},{"cell_type":"code","execution_count":52,"metadata":{},"outputs":[{"data":{"text/plain":["[<div class=\"leftPanelWrap\">\n"," <div class=\"headerWrap\">문서목차</div>\n"," <div class=\"leftPanel\">\n"," <ul aria-activedescendant=\"2\" aria-busy=\"false\" class=\"jstree jstree-1 jstree-default\" id=\"listTree\" role=\"tree\" tabindex=\"0\"><ul class=\"jstree-container-ul jstree-children jstree-no-icons\" role=\"group\"><li aria-labelledby=\"1_anchor\" aria-level=\"1\" aria-selected=\"true\" class=\"jstree-node jstree-leaf\" id=\"1\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"1\" aria-selected=\"true\" class=\"jstree-anchor jstree-clicked\" href=\"#\" id=\"1_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>사 업 보 고 서</a></li><li aria-labelledby=\"2_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"2\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"2_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>【 대표이사 등의 확인 】</a></li><li aria-expanded=\"true\" aria-labelledby=\"3_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-open\" id=\"3\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-expanded=\"true\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"3_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>I. 회사의 개요</a><ul class=\"jstree-children\" role=\"group\"><li aria-labelledby=\"4_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"4\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"4_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>1. 회사의 개요</a></li><li aria-labelledby=\"5_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"5\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"5_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>2. 회사의 연혁</a></li><li aria-labelledby=\"6_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"6\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"6_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>3. 자본금 변동사항</a></li><li aria-labelledby=\"7_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"7\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"7_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>4. 주식의 총수 등</a></li><li aria-labelledby=\"8_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf jstree-last\" id=\"8\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"8_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>5. 정관에 관한 사항</a></li></ul></li><li aria-expanded=\"true\" aria-labelledby=\"9_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-open\" id=\"9\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-expanded=\"true\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"9_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>II. 사업의 내용</a><ul class=\"jstree-children\" role=\"group\"><li aria-labelledby=\"10_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"10\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"10_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>1. 사업의 개요</a></li><li aria-labelledby=\"11_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"11\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"11_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>2. 주요 제품 및 서비스</a></li><li aria-labelledby=\"12_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"12\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"12_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>3. 원재료 및 생산설비</a></li><li aria-labelledby=\"13_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"13\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"13_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>4. 매출 및 수주상황</a></li><li aria-labelledby=\"14_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"14\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"14_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>5. 위험관리 및 파생거래</a></li><li aria-labelledby=\"15_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"15\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"15_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>6. 주요계약 및 연구개발활동</a></li><li aria-labelledby=\"16_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf jstree-last\" id=\"16\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"16_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>7. 기타 참고사항</a></li></ul></li><li aria-expanded=\"true\" aria-labelledby=\"17_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-open\" id=\"17\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-expanded=\"true\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"17_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>III. 재무에 관한 사항</a><ul class=\"jstree-children\" role=\"group\"><li aria-labelledby=\"18_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"18\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"18_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>1. 요약재무정보</a></li><li aria-labelledby=\"19_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"19\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"19_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>2. 연결재무제표</a></li><li aria-labelledby=\"20_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"20\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"20_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>3. 연결재무제표 주석</a></li><li aria-labelledby=\"21_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"21\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"21_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>4. 재무제표</a></li><li aria-labelledby=\"22_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"22\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"22_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>5. 재무제표 주석</a></li><li aria-labelledby=\"23_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"23\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"23_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>6. 배당에 관한 사항</a></li><li aria-labelledby=\"24_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"24\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"24_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>7. 증권의 발행을 통한 자금조달에 관한 사항</a></li><li aria-labelledby=\"27_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf jstree-last\" id=\"27\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"27_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>8. 기타 재무에 관한 사항</a></li></ul></li><li aria-labelledby=\"28_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"28\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"28_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>IV. 이사의 경영진단 및 분석의견</a></li><li aria-expanded=\"true\" aria-labelledby=\"29_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-open\" id=\"29\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-expanded=\"true\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"29_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>V. 회계감사인의 감사의견 등</a><ul class=\"jstree-children\" role=\"group\"><li aria-labelledby=\"30_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"30\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"30_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>1. 외부감사에 관한 사항</a></li><li aria-labelledby=\"31_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf jstree-last\" id=\"31\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"31_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>2. 내부통제에 관한 사항</a></li></ul></li><li aria-expanded=\"true\" aria-labelledby=\"32_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-open\" id=\"32\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-expanded=\"true\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"32_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>VI. 이사회 등 회사의 기관에 관한 사항</a><ul class=\"jstree-children\" role=\"group\"><li aria-labelledby=\"33_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"33\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"33_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>1. 이사회에 관한 사항</a></li><li aria-labelledby=\"34_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"34\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"34_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>2. 감사제도에 관한 사항</a></li><li aria-labelledby=\"35_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf jstree-last\" id=\"35\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"35_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>3. 주주총회 등에 관한 사항</a></li></ul></li><li aria-labelledby=\"36_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"36\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"36_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>VII. 주주에 관한 사항</a></li><li aria-expanded=\"true\" aria-labelledby=\"37_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-open\" id=\"37\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-expanded=\"true\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"37_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>VIII. 임원 및 직원 등에 관한 사항</a><ul class=\"jstree-children\" role=\"group\"><li aria-labelledby=\"38_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"38\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"38_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>1. 임원 및 직원 등의 현황</a></li><li aria-labelledby=\"39_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf jstree-last\" id=\"39\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"39_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>2. 임원의 보수 등</a></li></ul></li><li aria-labelledby=\"40_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"40\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"40_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>IX. 계열회사 등에 관한 사항</a></li><li aria-labelledby=\"41_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"41\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"41_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>X. 대주주 등과의 거래내용</a></li><li aria-expanded=\"true\" aria-labelledby=\"42_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-open\" id=\"42\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-expanded=\"true\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"42_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>XI. 그 밖에 투자자 보호를 위하여 필요한 사항</a><ul class=\"jstree-children\" role=\"group\"><li aria-labelledby=\"43_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"43\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"43_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>1. 공시내용 진행 및 변경사항</a></li><li aria-labelledby=\"44_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"44\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"44_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>2. 우발부채 등에 관한 사항</a></li><li aria-labelledby=\"45_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"45\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"45_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>3. 제재 등과 관련된 사항</a></li><li aria-labelledby=\"46_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf jstree-last\" id=\"46\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"46_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>4. 작성기준일 이후 발생한 주요사항 등 기타사항</a></li></ul></li><li aria-expanded=\"true\" aria-labelledby=\"47_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-open\" id=\"47\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-expanded=\"true\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"47_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>XII. 상세표</a><ul class=\"jstree-children\" role=\"group\"><li aria-labelledby=\"48_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"48\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"48_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>1. 연결대상 종속회사 현황(상세)</a></li><li aria-labelledby=\"49_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"49\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"49_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>2. 계열회사 현황(상세)</a></li><li aria-labelledby=\"50_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"50\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"50_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>3. 타법인출자 현황(상세)</a></li><li aria-labelledby=\"51_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf jstree-last\" id=\"51\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"51_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>4. 연구개발실적(상세)</a></li></ul></li><li aria-expanded=\"true\" aria-labelledby=\"52_anchor\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-node jstree-open jstree-last\" id=\"52\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-expanded=\"true\" aria-level=\"1\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"52_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>【 전문가의 확인 】</a><ul class=\"jstree-children\" role=\"group\"><li aria-labelledby=\"53_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf\" id=\"53\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"53_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>1. 전문가의 확인</a></li><li aria-labelledby=\"54_anchor\" aria-level=\"2\" aria-selected=\"false\" class=\"jstree-node jstree-leaf jstree-last\" id=\"54\" role=\"presentation\"><i class=\"jstree-icon jstree-ocl\" role=\"presentation\"></i><a aria-level=\"2\" aria-selected=\"false\" class=\"jstree-anchor\" href=\"#\" id=\"54_anchor\" role=\"treeitem\" tabindex=\"-1\"><i class=\"jstree-icon jstree-themeicon\" role=\"presentation\"></i>2. 전문가와의 이해관계</a></li></ul></li></ul></ul>\n"," </div>\n"," </div>]"]},"execution_count":52,"metadata":{},"output_type":"execute_result"}],"source":["response = driver.page_source\n","bs = BeautifulSoup(response, 'lxml')\n","bs.select('div.leftPanelWrap')"]},{"cell_type":"code","execution_count":54,"metadata":{},"outputs":[],"source":["(\n","    driver.find_element(\n","        By.XPATH,\n","        '/html/body/div[3]/div/div[2]/div[1]/div[2]/ul/ul/li[5]/ul/li[2]/a'\n","    ).click()\n",")"]},{"cell_type":"code","execution_count":57,"metadata":{},"outputs":[{"data":{"text/plain":["[]"]},"execution_count":57,"metadata":{},"output_type":"execute_result"}],"source":["# selenium이 만능이 아님\n","response = driver.page_source\n","bs = BeautifulSoup(response, 'lxml')\n","bs.select('html table p')"]},{"cell_type":"code","execution_count":58,"metadata":{},"outputs":[],"source":["driver.close()"]},{"cell_type":"code","execution_count":59,"metadata":{},"outputs":[],"source":["# 컨트롤 윈도우 변경\n","driver.switch_to.window(driver.window_handles[0])"]},{"cell_type":"markdown","metadata":{"id":"ml_ZZBXcxLbY"},"source":["# 5.Data Collection with API"]},{"cell_type":"markdown","metadata":{"id":"Vz0_5IsP0Me2"},"source":["## 5.1 Youtube\n","\n","site: https://console.developers.google.com/ <br>\n","reference: https://developers.google.com/youtube/v3/docs?hl=ko <br>\n","\n","```python\n","from googleapiclient.discovery import build\n","from googleapiclient.errors import HttpError\n","from oauth2client.tools import argparser\n","\n","import pandas as pd\n","\n","\n","DEVELOPER_KEY=''\n","YOUTUBE_API_SERVICE_NAME='youtube'\n","YOUTUBE_API_VERSION='v3'\n","\n","youtube = build(\n","  YOUTUBE_API_SERVICE_NAME,\n","  YOUTUBE_API_VERSION,\n","  developerKey=DEVELOPER_KEY\n","  )\n","```"]},{"cell_type":"markdown","metadata":{"id":"f_e1_4EtoPPl"},"source":["### Videos\n","\n","채널 내 비디오 조회 기본 코드\n","```python\n","response = (\n","  youtube\n","  .playlistItems()\n","  .list(\n","    playlistId=*channel_id*,\n","    part='snippet',\n","    maxResults=50\n","    )\n","  .execute()\n","  )\n","```\n","\n","\n","1번의 query에 전체 데이터를 다 받을 수 없음 <br>\n","따라서 pageToken을 이용하여 다음 페이지의 정보를 받아야 함 <br>\n","```python\n","if 'nextPageToken' in response:\n","  response = (\n","    youtube\n","    .playlistItems()\n","    .list(\n","      playlistId=*channel_id*,\n","      pageToken=response.get('nextPageToken'),\n","      part='snippet',\n","      maxResults=50\n","      )\n","    .execute()\n","    )\n","```"]},{"cell_type":"markdown","metadata":{"id":"6UVVpXYMoR6Y"},"source":["### Comments\n","비디오 내 댓글 조회 기본 코드\n","```python\n","response = (\n","    youtube\n","    .commentThreads()\n","    .list(\n","      part='snippet,replies', \n","      videoId=*video_id*, \n","      maxResults=100\n","      )\n","    .execute()\n","    )\n","```\n","\n","1번의 query에 전체 데이터를 다 받을 수 없음 <br>\n","따라서 pageToken을 이용하여 다음 페이지의 정보를 받아야 함 <br>\n","```python\n","if 'nextPageToken' in response:\n","  response = (\n","    youtube\n","    .commentThreads()\n","    .list(\n","      part='snippet,replies',\n","      videoId=video_id,\n","      pageToken=response.get('nextPageToken'),\n","      maxResults=100\n","      )\n","    .execute()\n","    )\n","```"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# 설치\n","!pip install google-api-python-client\n","!pip install oauth2client"]},{"cell_type":"code","execution_count":60,"metadata":{},"outputs":[],"source":["from googleapiclient.discovery import build\n","from googleapiclient.errors import HttpError\n","from oauth2client.tools import argparser\n","\n","import pandas as pd\n","\n","DEVELOPER_KEY = 'AIzaSyCRYjXOc__v73vpnRHJbgy0JX0yzxYN_-0'\n","YOUTUBE_API_SERVICE_NAME = 'youtube'\n","YOUTUBE_API_VERSION = 'v3'\n","\n","youtube = build(\n","  YOUTUBE_API_SERVICE_NAME,\n","  YOUTUBE_API_VERSION,\n","  developerKey=DEVELOPER_KEY\n","  )"]},{"cell_type":"code","execution_count":61,"metadata":{},"outputs":[],"source":["# 댓글 수집\n","response = (\n","    youtube\n","    .commentThreads()\n","    .list(\n","      part='snippet,replies', \n","      videoId='riMHgOa8yf4', \n","      maxResults=100\n","      )\n","    .execute()\n","    )"]},{"cell_type":"code","execution_count":66,"metadata":{},"outputs":[{"data":{"text/plain":["100"]},"execution_count":66,"metadata":{},"output_type":"execute_result"}],"source":["# 개수 확인\n","len(response.get('items'))"]},{"cell_type":"code","execution_count":85,"metadata":{},"outputs":[{"data":{"text/plain":["'미션 파서블'"]},"execution_count":85,"metadata":{},"output_type":"execute_result"}],"source":["response.get('items')[0].get('snippet').get('topLevelComment').get('snippet').get('authorDisplayName')"]},{"cell_type":"code","execution_count":81,"metadata":{},"outputs":[{"data":{"text/plain":["'아니…. 지금 그게 아이라. 모든 군인과 임무가 다 중요하고. 예비역 모두가 중요한데. 우리가 귀찮아하고 피곤해하면 안된다고…….\\n\\n대한민국 군장병분들과 예비역 모든분들.\\n정말 자랑스럽고 존경합니다. \\n\\n늘 감사합니다.'"]},"execution_count":81,"metadata":{},"output_type":"execute_result"}],"source":["response.get('items')[0].get('snippet').get('topLevelComment').get('snippet').get('textOriginal')"]},{"cell_type":"code","execution_count":89,"metadata":{},"outputs":[{"data":{"text/plain":["'2022-06-19T04:43:02Z'"]},"execution_count":89,"metadata":{},"output_type":"execute_result"}],"source":["response.get('items')[0].get('snippet').get('topLevelComment').get('snippet').get('publishedAt')"]},{"cell_type":"code","execution_count":88,"metadata":{},"outputs":[{"data":{"text/plain":["'2022-06-19T04:43:02Z'"]},"execution_count":88,"metadata":{},"output_type":"execute_result"}],"source":["response.get('items')[0].get('snippet').get('topLevelComment').get('snippet').get('updatedAt')"]},{"cell_type":"code","execution_count":94,"metadata":{},"outputs":[],"source":["_comments = {\n","    'published_date': [response.get('items')[i].get('snippet').get('topLevelComment').get('snippet').get('publishedAt') for i in range(100)],\n","    'updated_date': [response.get('items')[i].get('snippet').get('topLevelComment').get('snippet').get('updatedAt') for i in range(100)],\n","    'commentor': [response.get('items')[i].get('snippet').get('topLevelComment').get('snippet').get('authorDisplayName') for i in range(100)],\n","    'comment': [response.get('items')[i].get('snippet').get('topLevelComment').get('snippet').get('textOriginal') for i in range(100)],\n","    'like': [response.get('items')[i].get('snippet').get('topLevelComment').get('snippet').get('likeCount') for i in range(100)]\n","}"]},{"cell_type":"code","execution_count":95,"metadata":{},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>published_date</th>\n","      <th>updated_date</th>\n","      <th>commentor</th>\n","      <th>comment</th>\n","      <th>like</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>2022-06-19T04:43:02Z</td>\n","      <td>2022-06-19T04:43:02Z</td>\n","      <td>미션 파서블</td>\n","      <td>아니…. 지금 그게 아이라. 모든 군인과 임무가 다 중요하고. 예비역 모두가 중요한...</td>\n","      <td>34823</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>2022-12-04T01:11:50Z</td>\n","      <td>2022-12-04T01:11:50Z</td>\n","      <td>BA 201</td>\n","      <td>간부님.... 그만.....</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2022-12-04T01:01:38Z</td>\n","      <td>2022-12-04T01:01:38Z</td>\n","      <td>user</td>\n","      <td>Thank you for your service</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>2022-12-04T00:08:47Z</td>\n","      <td>2022-12-04T00:08:47Z</td>\n","      <td>저스틴</td>\n","      <td>ㅋㅋㅋㅋ 존나웃기네</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>2022-12-03T18:39:45Z</td>\n","      <td>2022-12-03T18:39:45Z</td>\n","      <td>1421이상민</td>\n","      <td>사나이로 태어나서~</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>95</th>\n","      <td>2022-11-06T11:32:40Z</td>\n","      <td>2022-11-06T11:32:40Z</td>\n","      <td>살찐성인</td>\n","      <td>703 가리산 !!!!!!!! ㅎㅎㅎㅎ</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>96</th>\n","      <td>2022-11-06T11:08:48Z</td>\n","      <td>2022-11-06T11:08:48Z</td>\n","      <td>알fun한이야기</td>\n","      <td>찬호 형님 언제 UDT 다녀오셨데?</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>97</th>\n","      <td>2022-11-06T11:02:51Z</td>\n","      <td>2022-11-06T11:02:51Z</td>\n","      <td>SWKB</td>\n","      <td>어? 703? 굉장히 반갑네요...ㅎㅎㅎㅎㅎ\\n전 701이고 제 친구가 703이었는...</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>98</th>\n","      <td>2022-11-06T09:13:09Z</td>\n","      <td>2022-11-06T09:13:09Z</td>\n","      <td>돈천지</td>\n","      <td>니 어디 나왔는데? 이러는데 UDT 이러니깐 뭔가 임팩트는 없다 ..ㅎㅎ  특전사 ...</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>99</th>\n","      <td>2022-11-06T08:48:13Z</td>\n","      <td>2022-11-06T08:48:13Z</td>\n","      <td>칸예 당선</td>\n","      <td>아무리 좋아졌다 해도 ㅈ같긴함ㅋㅋ</td>\n","      <td>0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>100 rows × 5 columns</p>\n","</div>"],"text/plain":["          published_date          updated_date commentor  \\\n","0   2022-06-19T04:43:02Z  2022-06-19T04:43:02Z    미션 파서블   \n","1   2022-12-04T01:11:50Z  2022-12-04T01:11:50Z    BA 201   \n","2   2022-12-04T01:01:38Z  2022-12-04T01:01:38Z      user   \n","3   2022-12-04T00:08:47Z  2022-12-04T00:08:47Z       저스틴   \n","4   2022-12-03T18:39:45Z  2022-12-03T18:39:45Z   1421이상민   \n","..                   ...                   ...       ...   \n","95  2022-11-06T11:32:40Z  2022-11-06T11:32:40Z      살찐성인   \n","96  2022-11-06T11:08:48Z  2022-11-06T11:08:48Z  알fun한이야기   \n","97  2022-11-06T11:02:51Z  2022-11-06T11:02:51Z      SWKB   \n","98  2022-11-06T09:13:09Z  2022-11-06T09:13:09Z       돈천지   \n","99  2022-11-06T08:48:13Z  2022-11-06T08:48:13Z     칸예 당선   \n","\n","                                              comment   like  \n","0   아니…. 지금 그게 아이라. 모든 군인과 임무가 다 중요하고. 예비역 모두가 중요한...  34823  \n","1                                     간부님.... 그만.....      0  \n","2                          Thank you for your service      0  \n","3                                          ㅋㅋㅋㅋ 존나웃기네      0  \n","4                                          사나이로 태어나서~      0  \n","..                                                ...    ...  \n","95                              703 가리산 !!!!!!!! ㅎㅎㅎㅎ      0  \n","96                                찬호 형님 언제 UDT 다녀오셨데?      1  \n","97  어? 703? 굉장히 반갑네요...ㅎㅎㅎㅎㅎ\\n전 701이고 제 친구가 703이었는...      0  \n","98  니 어디 나왔는데? 이러는데 UDT 이러니깐 뭔가 임팩트는 없다 ..ㅎㅎ  특전사 ...      1  \n","99                                 아무리 좋아졌다 해도 ㅈ같긴함ㅋㅋ      0  \n","\n","[100 rows x 5 columns]"]},"execution_count":95,"metadata":{},"output_type":"execute_result"}],"source":["pd.DataFrame(_comments)"]},{"cell_type":"code","execution_count":100,"metadata":{},"outputs":[],"source":["def get_information_from_response(response):\n","    item_len = len(response.get('items'))\n","\n","    _comments = {\n","        'published_date': [response.get('items')[i].get('snippet').get('topLevelComment').get('snippet').get('publishedAt') for i in range(item_len)],\n","        'updated_date': [response.get('items')[i].get('snippet').get('topLevelComment').get('snippet').get('updatedAt') for i in range(item_len)],\n","        'commentor': [response.get('items')[i].get('snippet').get('topLevelComment').get('snippet').get('authorDisplayName') for i in range(item_len)],\n","        'comment': [response.get('items')[i].get('snippet').get('topLevelComment').get('snippet').get('textOriginal') for i in range(item_len)],\n","        'like': [response.get('items')[i].get('snippet').get('topLevelComment').get('snippet').get('likeCount') for i in range(item_len)]\n","    }\n","\n","    return pd.DataFrame(_comments)"]},{"cell_type":"code","execution_count":101,"metadata":{},"outputs":[],"source":["response = (\n","    youtube\n","    .commentThreads()\n","    .list(\n","      part='snippet,replies', \n","      videoId='riMHgOa8yf4', \n","      maxResults=100\n","      )\n","    .execute()\n","    )\n","\n","comments = get_information_from_response(response)\n","\n","while 'nextPageToken' in response:\n","    response = (\n","      youtube\n","      .commentThreads()\n","      .list(\n","        part='snippet,replies',\n","        videoId='riMHgOa8yf4',\n","        pageToken=response.get('nextPageToken'),\n","        maxResults=100\n","        )\n","    .execute()\n","    )\n","\n","    comments = comments.append(get_information_from_response(response))"]},{"cell_type":"code","execution_count":102,"metadata":{},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>published_date</th>\n","      <th>updated_date</th>\n","      <th>commentor</th>\n","      <th>comment</th>\n","      <th>like</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>2022-06-19T04:43:02Z</td>\n","      <td>2022-06-19T04:43:02Z</td>\n","      <td>미션 파서블</td>\n","      <td>아니…. 지금 그게 아이라. 모든 군인과 임무가 다 중요하고. 예비역 모두가 중요한...</td>\n","      <td>34824</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>2022-12-04T02:44:59Z</td>\n","      <td>2022-12-04T02:44:59Z</td>\n","      <td>망나뇽</td>\n","      <td>군대는 다 힘들지만\\n11군번 짬찌인 나보다 밑으로 갈수록 편하고 나보다 올라갈수록...</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2022-12-04T01:11:50Z</td>\n","      <td>2022-12-04T01:11:50Z</td>\n","      <td>BA 201</td>\n","      <td>간부님.... 그만.....</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>2022-12-04T01:01:38Z</td>\n","      <td>2022-12-04T01:01:38Z</td>\n","      <td>user</td>\n","      <td>Thank you for your service</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>2022-12-04T00:08:47Z</td>\n","      <td>2022-12-04T00:08:47Z</td>\n","      <td>저스틴</td>\n","      <td>ㅋㅋㅋㅋ 존나웃기네</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>78</th>\n","      <td>2022-06-19T04:24:21Z</td>\n","      <td>2022-06-19T04:24:21Z</td>\n","      <td>날[nal]</td>\n","      <td>군부심?</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>79</th>\n","      <td>2022-06-19T04:24:21Z</td>\n","      <td>2022-06-19T04:24:21Z</td>\n","      <td>갬갬</td>\n","      <td>1등 !</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>80</th>\n","      <td>2022-06-19T04:24:21Z</td>\n","      <td>2022-06-19T04:24:21Z</td>\n","      <td>Arthur_ J</td>\n","      <td>ㅘ앍</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>81</th>\n","      <td>2022-06-19T04:24:20Z</td>\n","      <td>2022-06-19T04:24:20Z</td>\n","      <td>Blue Cosmos</td>\n","      <td>♥️</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>82</th>\n","      <td>2022-06-19T04:24:20Z</td>\n","      <td>2022-06-19T04:24:20Z</td>\n","      <td>레전드</td>\n","      <td>ㅇ</td>\n","      <td>0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>4583 rows × 5 columns</p>\n","</div>"],"text/plain":["          published_date          updated_date    commentor  \\\n","0   2022-06-19T04:43:02Z  2022-06-19T04:43:02Z       미션 파서블   \n","1   2022-12-04T02:44:59Z  2022-12-04T02:44:59Z          망나뇽   \n","2   2022-12-04T01:11:50Z  2022-12-04T01:11:50Z       BA 201   \n","3   2022-12-04T01:01:38Z  2022-12-04T01:01:38Z         user   \n","4   2022-12-04T00:08:47Z  2022-12-04T00:08:47Z          저스틴   \n","..                   ...                   ...          ...   \n","78  2022-06-19T04:24:21Z  2022-06-19T04:24:21Z       날[nal]   \n","79  2022-06-19T04:24:21Z  2022-06-19T04:24:21Z           갬갬   \n","80  2022-06-19T04:24:21Z  2022-06-19T04:24:21Z    Arthur_ J   \n","81  2022-06-19T04:24:20Z  2022-06-19T04:24:20Z  Blue Cosmos   \n","82  2022-06-19T04:24:20Z  2022-06-19T04:24:20Z          레전드   \n","\n","                                              comment   like  \n","0   아니…. 지금 그게 아이라. 모든 군인과 임무가 다 중요하고. 예비역 모두가 중요한...  34824  \n","1   군대는 다 힘들지만\\n11군번 짬찌인 나보다 밑으로 갈수록 편하고 나보다 올라갈수록...      0  \n","2                                     간부님.... 그만.....      0  \n","3                          Thank you for your service      0  \n","4                                          ㅋㅋㅋㅋ 존나웃기네      0  \n","..                                                ...    ...  \n","78                                               군부심?      0  \n","79                                               1등 !      0  \n","80                                                 ㅘ앍      0  \n","81                                                 ♥️      0  \n","82                                                  ㅇ      0  \n","\n","[4583 rows x 5 columns]"]},"execution_count":102,"metadata":{},"output_type":"execute_result"}],"source":["comments"]},{"cell_type":"code","execution_count":126,"metadata":{},"outputs":[],"source":["# 대댓글\n","response = (\n","    youtube\n","    .commentThreads()\n","    .list(\n","      part='snippet,replies', \n","      videoId='riMHgOa8yf4', \n","      maxResults=100\n","      )\n","    .execute()\n","    )\n"]},{"cell_type":"code","execution_count":159,"metadata":{},"outputs":[],"source":["def get_replies_information_from_response(response):\n","    comments = pd.DataFrame()\n","\n","    for items in response.get('items'):\n","        if items.get('replies'):\n","            item_len = len(items.get('replies').get('comments'))\n","\n","            _comments = {\n","                'published_date': [items.get('replies').get('comments')[i].get('snippet').get('publishedAt') for i in range(item_len)],\n","                'updated_date': [items.get('replies').get('comments')[i].get('snippet').get('updatedAt') for i in range(item_len)],\n","                'commentor': [items.get('replies').get('comments')[i].get('snippet').get('authorDisplayName') for i in range(item_len)],\n","                'comment': [items.get('replies').get('comments')[i].get('snippet').get('textOriginal') for i in range(item_len)],\n","                'like': [items.get('replies').get('comments')[i].get('snippet').get('likeCount') for i in range(item_len)]\n","            }\n","\n","            comments = comments.append(pd.DataFrame(_comments))\n","\n","    return comments"]},{"cell_type":"code","execution_count":160,"metadata":{},"outputs":[],"source":["response = (\n","    youtube\n","    .commentThreads()\n","    .list(\n","      part='snippet,replies', \n","      videoId='riMHgOa8yf4', \n","      maxResults=100\n","      )\n","    .execute()\n","    )\n","\n","comments = get_information_from_response(response)\n","comments = comments.append(get_replies_information_from_response(response))\n","\n","while 'nextPageToken' in response:\n","    response = (\n","      youtube\n","      .commentThreads()\n","      .list(\n","        part='snippet,replies',\n","        videoId='riMHgOa8yf4',\n","        pageToken=response.get('nextPageToken'),\n","        maxResults=100\n","        )\n","    .execute()\n","    )\n","\n","    comments = comments.append(get_information_from_response(response))\n","    comments = comments.append(get_replies_information_from_response(response))"]},{"cell_type":"code","execution_count":161,"metadata":{},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>published_date</th>\n","      <th>updated_date</th>\n","      <th>commentor</th>\n","      <th>comment</th>\n","      <th>like</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>2022-06-19T04:43:02Z</td>\n","      <td>2022-06-19T04:43:02Z</td>\n","      <td>미션 파서블</td>\n","      <td>아니…. 지금 그게 아이라. 모든 군인과 임무가 다 중요하고. 예비역 모두가 중요한...</td>\n","      <td>34826</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>2022-12-04T02:44:59Z</td>\n","      <td>2022-12-04T02:44:59Z</td>\n","      <td>망나뇽</td>\n","      <td>군대는 다 힘들지만\\n11군번 짬찌인 나보다 밑으로 갈수록 편하고 나보다 올라갈수록...</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2022-12-04T01:11:50Z</td>\n","      <td>2022-12-04T01:11:50Z</td>\n","      <td>BA 201</td>\n","      <td>간부님.... 그만.....</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>2022-12-04T01:01:38Z</td>\n","      <td>2022-12-04T01:01:38Z</td>\n","      <td>user</td>\n","      <td>Thank you for your service</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>2022-12-04T00:08:47Z</td>\n","      <td>2022-12-04T00:08:47Z</td>\n","      <td>저스틴</td>\n","      <td>ㅋㅋㅋㅋ 존나웃기네</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>0</th>\n","      <td>2022-07-18T19:41:22Z</td>\n","      <td>2022-07-18T19:41:22Z</td>\n","      <td>김잉</td>\n","      <td>동현애미는 무슨죄를저질러서 저딴거낳고 미역국처먹었겠누 ㅜㅜ</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>2022-06-24T08:56:10Z</td>\n","      <td>2022-06-24T08:56:10Z</td>\n","      <td>동철이</td>\n","      <td>감사합니다 열심히 나라지키겠습니다 충성!</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>2022-06-21T05:08:22Z</td>\n","      <td>2022-06-21T05:08:28Z</td>\n","      <td>민1</td>\n","      <td>@ᄋᄋ 생캉스 생리하고 싶다 생리 개꿀이잖어 ㅋㅋㅋㅋㅋㅋ</td>\n","      <td>4</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>2022-06-19T13:02:13Z</td>\n","      <td>2022-06-19T13:02:13Z</td>\n","      <td>이예전골</td>\n","      <td>@동현 그러면 알바를 하지 말던갘ㅋㅋㅋㅋ 여자들은 그 동안 자기계발 같은 것들도 할...</td>\n","      <td>2</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>2022-06-19T12:59:38Z</td>\n","      <td>2022-06-19T12:59:38Z</td>\n","      <td>wd qs</td>\n","      <td>군쾅군쾅</td>\n","      <td>1</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>5009 rows × 5 columns</p>\n","</div>"],"text/plain":["          published_date          updated_date commentor  \\\n","0   2022-06-19T04:43:02Z  2022-06-19T04:43:02Z    미션 파서블   \n","1   2022-12-04T02:44:59Z  2022-12-04T02:44:59Z       망나뇽   \n","2   2022-12-04T01:11:50Z  2022-12-04T01:11:50Z    BA 201   \n","3   2022-12-04T01:01:38Z  2022-12-04T01:01:38Z      user   \n","4   2022-12-04T00:08:47Z  2022-12-04T00:08:47Z       저스틴   \n","..                   ...                   ...       ...   \n","0   2022-07-18T19:41:22Z  2022-07-18T19:41:22Z        김잉   \n","1   2022-06-24T08:56:10Z  2022-06-24T08:56:10Z       동철이   \n","2   2022-06-21T05:08:22Z  2022-06-21T05:08:28Z        민1   \n","3   2022-06-19T13:02:13Z  2022-06-19T13:02:13Z      이예전골   \n","4   2022-06-19T12:59:38Z  2022-06-19T12:59:38Z     wd qs   \n","\n","                                              comment   like  \n","0   아니…. 지금 그게 아이라. 모든 군인과 임무가 다 중요하고. 예비역 모두가 중요한...  34826  \n","1   군대는 다 힘들지만\\n11군번 짬찌인 나보다 밑으로 갈수록 편하고 나보다 올라갈수록...      0  \n","2                                     간부님.... 그만.....      0  \n","3                          Thank you for your service      0  \n","4                                          ㅋㅋㅋㅋ 존나웃기네      0  \n","..                                                ...    ...  \n","0                    동현애미는 무슨죄를저질러서 저딴거낳고 미역국처먹었겠누 ㅜㅜ      1  \n","1                              감사합니다 열심히 나라지키겠습니다 충성!      1  \n","2                     @ᄋᄋ 생캉스 생리하고 싶다 생리 개꿀이잖어 ㅋㅋㅋㅋㅋㅋ      4  \n","3   @동현 그러면 알바를 하지 말던갘ㅋㅋㅋㅋ 여자들은 그 동안 자기계발 같은 것들도 할...      2  \n","4                                                군쾅군쾅      1  \n","\n","[5009 rows x 5 columns]"]},"execution_count":161,"metadata":{},"output_type":"execute_result"}],"source":["comments"]}],"metadata":{"colab":{"authorship_tag":"ABX9TyMeRRyztVQSgdQa1bKIvPQO","provenance":[],"toc_visible":true},"kernelspec":{"display_name":"Python 3.9.12 ('base')","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.12"},"vscode":{"interpreter":{"hash":"ad2bdc8ecc057115af97d19610ffacc2b4e99fae6737bb82f5d7fb13d2f2c186"}}},"nbformat":4,"nbformat_minor":0}