{"cells":[{"cell_type":"markdown","metadata":{"id":"J7BmBhj_hd5n"},"source":["# Advanced web scraping\n","\n","## Before you begin\n","\n","Before you start webscraping make sure to consider what you're\n","doing. Does your scraping violate a terms of service? Will it inconvenience the site, other users? Per Uncle Ben: WGPCGR.\n","\n","Also, before you begin web scraping, look for a download data option\n","or existing solution. Probably someone has run up against the same\n","problem and worked it out. For example, we're going to scrape some\n","wikipedia tables, which there's a million other solutions for,\n","including a wikipedia\n","[api](https://www.mediawiki.org/wiki/API:Main_page).\n","\n","## Basic web scraping\n","\n","We covered this last chapter. However, let's do an example of static page parsing just to get\n","started. Consider scraping the table of top 10 heat waves from\n","[wikipedia](https://en.wikipedia.org/wiki/List_of_natural_disasters_by_death_toll). First,\n","we open the url, then parse it using BeautifulSoup, then load it into\n","a pandas dataframe.\n"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/","height":363},"id":"Elc1V_7fhd5s","executionInfo":{"status":"ok","timestamp":1713234049778,"user_tz":240,"elapsed":1350,"user":{"displayName":"Brian Caffo","userId":"07979705296072332292"}},"outputId":"b8b7e530-7c98-4f6b-c3bc-fc442463121a"},"source":["from urllib.request import urlopen\n","from bs4 import BeautifulSoup as bs\n","import pandas as pd\n","url = \"https://en.wikipedia.org/wiki/List_of_natural_disasters_by_death_toll\"\n","html = urlopen(url)\n","parsed = bs(html, 'html.parser').findAll(\"table\")\n","pd.read_html(str(parsed))[11]"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":[" Rank Death toll Event \\\n","0 1.0 1300 The Daulatpur–Saturia tornado \n","1 2.0 751 The Tri-State tornado outbreak \n","2 3.0 681 1973 Dhaka tornado \n","3 4.0 660 1969 East Pakistan tornado \n","4 5.0 600 The Valletta, Malta tornado \n","5 6.0 500 The 1851 Sicily tornadoes \n","6 6.0 500 Narail-Magura tornado \n","7 6.0 500 Madaripur-Shibchar tornado \n","8 9.0 400 The 1984 Soviet Union tornado outbreak \n","9 10.0 317 The Great Natchez Tornado \n","\n"," Location Date \n","0 Manikganj, Bangladesh 1989 \n","1 United States (Missouri–Illinois–Indiana) 1925 \n","2 Bangladesh 1973 \n","3 East Pakistan (now Bangladesh) 1969 \n","4 Malta 1551 or 1556 \n","5 Sicily, Two Sicilies (now Italy) 1851 \n","6 Jessore, East Pakistan, Pakistan (now Bangladesh) 1964 \n","7 Bangladesh 1977 \n","8 Soviet Union (Volga Federal District, Central ... 1984 \n","9 United States (Mississippi–Louisiana) 1840 "],"text/html":["\n","
\n","
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
RankDeath tollEventLocationDate
01.01300The Daulatpur–Saturia tornadoManikganj, Bangladesh1989
12.0751The Tri-State tornado outbreakUnited States (Missouri–Illinois–Indiana)1925
23.06811973 Dhaka tornadoBangladesh1973
34.06601969 East Pakistan tornadoEast Pakistan (now Bangladesh)1969
45.0600The Valletta, Malta tornadoMalta1551 or 1556
56.0500The 1851 Sicily tornadoesSicily, Two Sicilies (now Italy)1851
66.0500Narail-Magura tornadoJessore, East Pakistan, Pakistan (now Bangladesh)1964
76.0500Madaripur-Shibchar tornadoBangladesh1977
89.0400The 1984 Soviet Union tornado outbreakSoviet Union (Volga Federal District, Central ...1984
910.0317The Great Natchez TornadoUnited States (Mississippi–Louisiana)1840
\n","
\n","
\n","\n","
\n"," \n","\n"," \n","\n"," \n","
\n","\n","\n","
\n"," \n","\n","\n","\n"," \n","
\n","\n","
\n","
\n"],"application/vnd.google.colaboratory.intrinsic+json":{"type":"dataframe","summary":"{\n \"name\": \"pd\",\n \"rows\": 10,\n \"fields\": [\n {\n \"column\": \"Rank\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 2.859681411936962,\n \"min\": 1.0,\n \"max\": 10.0,\n \"num_unique_values\": 8,\n \"samples\": [\n 2.0,\n 6.0,\n 1.0\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Death toll\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 272,\n \"min\": 317,\n \"max\": 1300,\n \"num_unique_values\": 8,\n \"samples\": [\n 751,\n 500,\n 1300\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Event\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 10,\n \"samples\": [\n \"The 1984 Soviet Union tornado outbreak\",\n \"The Tri-State tornado outbreak\",\n \"The 1851 Sicily tornadoes\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Location\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 9,\n \"samples\": [\n \"Soviet Union (Volga Federal District, Central Federal District, and Northwestern Federal District in Russia)\",\n \"United States (Missouri\\u2013Illinois\\u2013Indiana)\",\n \"Sicily, Two Sicilies (now Italy)\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Date\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 10,\n \"samples\": [\n \"1984\",\n \"1925\",\n \"1851\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}"}},"metadata":{},"execution_count":17}]},{"cell_type":"markdown","metadata":{"id":"4zuwwymUhd55"},"source":["The workflow as as follows:\n","\n","+ We used the developer console on the webpage to inspect the page and its properties.\n","+ We opened the url with `urlopen`\n","+ We parsed the webpage with `BeautifulSoup` then used the method `findAll` on that to search for every table\n","+ Pandas has a utility that converts a html tables into a dataframe. In this case it creates a list of tables, where the 12th one is the heatwaves. Note it needs the data to be converted to a string before proceeding.\n","\n","This variation of web scraping couldn't be easier. However, what if the content we're interested in only exists after interacting with the page? Then we need a more sophisticated solution.\n","\n","## Form filling\n","Web scraping can require posting to forms, such as logins. This can be\n","done directly with python / R without elaborate programming, for\n","example using the `requests` library. However, make sure you aren't\n","violating a web site's TOS and also make sure you're not posting your\n","password to github as you commit scraping code. In general, don't\n","create a security hole for your account by web scraping it. Again,\n","also check to make sure that the site doesn't have an API with an\n","authentication solution already before writing the code to post\n","authentication. Many websites that want you to programmatically grab\n","the data build an API.\n","\n","## Programmatically web browsing\n","\n","Some web scraping requires us to interact with the webpage. This\n","requires a much more advanced solution where we programmatically use a\n","web browser to interact with the page. I'm using selenium and\n","chromedriver. To do this, I had to download\n","[chromedriver](https://chromedriver.chromium.org/downloads) and set it\n","so that it was in my unix `PATH`.\n","\n","Then, the following code opened up a browser then closed it.\n","\n","```\n","from selenium import webdriver\n","driver = webdriver.Chrome()\n","driver.quit()\n","```\n","\n"]},{"cell_type":"markdown","metadata":{"id":"SINJzeYahd56"},"source":["If all went well, a chrome window appeared then closed. That's the\n","browser we're going to program. If you look closely at the browser\n","before you close it, there's a banner up to that says \"Chrome is being\n","controlled by automated test software.\" Let's go through the example\n","on the selenium docs\n","[here](https://www.selenium.dev/documentation/webdriver/getting_started/first_script/). First\n","let's vist a few pages. We'll go to my totally awesome web page that I\n","meticulously maintain every day then duckduckgo. We'll wait a few\n","seconds in between. My site is created and hosted by google sites,\n","which seems reasonable that they would store a cookie so that I can\n","log in and edit my site (which I almost never do). Duckduckgo is a\n","privacy browser, so let's check to see if they create a cookie. (Hint,\n","I noticed that selenium doesn't like redirects, so use the actual page\n","url.)\n","\n","Let's go to my website and get the cookies that it saves.\n","```\n","driver.get(\"https://sites.google.com/view/bcaffo/home\")\n","print(driver.get_cookies())\n","```\n","You should see the browser go to my web page. Now let's enter in a web form.\n","First, let's go to duckduck go. (Note it appears duckduckgo changed it's page so that this no longer works.)\n","\n","```\n","## Let's get rid of all cookies before we visit duckduckgo\n","driver.delete_all_cookies()\n","driver.get(\"https://duckduckgo.com/\")\n","print(driver.get_cookies())\n","```\n","\n","Now let's find the page elements that we'd like to interact\n","with. There's a text box that we want to submit a search command into\n","and a button that we'll need to press. When I go to duckduckgo and press\n","CTRL-I, I find that the search box is:\n","\n","```\n","\n","```\n","\n","Notice, the `name=\"q\"` html name for the search form. When I dig around and find the submit button, it's code is:\n","\n","```\n","\n","```\n","\n","Notice its `id` is `search_button_homepage`. Let's find these elements.\n","\n","```\n","search_box = driver.find_element(by=By.NAME, value=\"q\")\n","search_button = driver.find_element(by=By.ID, value=\"search_button_homepage\")\n","```\n","\n","Now let's send the info and press submit.\n","\n","```\n","search_box.send_keys(\"Selenium\")\n","search_button.click()\n","driver.implicitly_wait(10)\n","driver.save_screenshot(\"assets/images/webscraping.png\")\n","page_source = driver.page_source\n","driver.close()\n","```\n","\n","Here, we saved the page_source as a variable that then can be parsed\n","with other html parses (like bs4). Play around with the methods\n","associated with `driver` and navigate the web. You'll see that\n","selenium is pretty incredible.\n","\n","Since I wrote this, duckduckgo changed their page, so the code no longer works.\n","\n","\n","\n","\n"]},{"cell_type":"markdown","source":["## Doing it in colab\n","\n","Getting this to work on Colab is a little harder, since we can't launch a browser. I found the instructures [here](https://nariyoo.com/python-how-to-run-selenium-in-google-colab/) useful (https://nariyoo.com/python-how-to-run-selenium-in-google-colab/)"],"metadata":{"id":"qbv98fDoplY8"}}],"metadata":{"kernelspec":{"name":"python3","language":"python","display_name":"Python 3"},"colab":{"provenance":[]}},"nbformat":4,"nbformat_minor":0}