Python Series track

Week 07 · Advanced Track

Week 7 — Web Scraping

Extract structured data from web pages ethically. Use requests for HTTP and BeautifulSoup for parsing HTML. Implement rate limiting, respect robots.txt, and cache results.

Advanced Track2 min readFree lesson

Level: Advanced | Prerequisite: Week 6

Learning Goals

  • HTTP requests with requests
  • Parse HTML with BeautifulSoup
  • Handling headers, sessions, and cookies
  • robots.txt and etiquette

1. requests

import requests

r = requests.get("https://example.com")
print(r.status_code)        # 200
print(r.text[:200])         #Some HTML
print(r.headers["Content-Type"])

# JSON API
r = requests.get("https://api.github.com/users/python")
data = r.json()
print(data["public_repos"])

2. POST Requests

r = requests.post("https://httpbin.org/post",
    data={"key": "value"},
    headers={"User-Agent": "MyBot 1.0"},
)

3. BeautifulSoup

pip install beautifulsoup4 lxml

from bs4 import BeautifulSoup

html = """<html><body>
    <h1>Title</h1>
    <ul>
        <li class="item">A</li>
        <li class="item">B</li>
    </ul>
</body></html>"""

soup = BeautifulSoup(html, "html.parser")

print(soup.h1.text)
print(soup.find("h1").text)
print([li.text for li in soup.find_all("li")])
print([li.text for li in soup.select("li.item")])   #CSS Selector

4. Sessions and Cookies

session = requests.Session()
session.headers.update({"User-Agent": "Mozilla/5.0"})

#Login
session.post("https://example.com/login", data={"user": "x", "pass": "y"})

#Request with the same session → Keep cookies
r = session.get("https://example.com/me")

5. Etiquette and Ethics

  • Check robots.txt: https://example.com/robots.txt
  • Request intervals: Recommended around time.sleep(1)
  • User agent: Display honestly
  • Copyright and terms compliance

6. Practical Patterns

import requests
from bs4 import BeautifulSoup
import time

URL = "https://news.ycombinator.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (study)"}

def fetch_titles():
    r = requests.get(URL, headers=HEADERS, timeout=10)
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "html.parser")
    titles = [a.get_text() for a in soup.select(".titleline > a")]
    return titles[:10]

for i, title in enumerate(fetch_titles(), 1):
    print(f"{i}. {title}")

Key Examples (examples/)

File Content
01_requests_basic.py GET, JSON
02_bs4_parse.py HTML Parsing
03_session.py Session + Cookies
04_scrape_demo.py Actual Scraping (httpbin)

⚠️ Common Mistakes

  1. Missing timeout — Risk of infinite wait. Always around timeout=10.
  2. Not checking r.raise_for_status() — Can't detect 404.
  3. HTML parser not specified — Explicitly "html.parser" or "lxml".
  4. Selectors too fragile — id/class changes often. Use robust selectors.
  5. Getting blocked — Requests are too fast. Interval of time.sleep.

❓ FAQ

Q1. Pages rendered by JavaScript? A. Cannot be done with requests. Use Selenium or Playwright.

Q2. What is a dynamic site? A. Usually calls the internal API directly (found in Developer Tools → Network tab).

Q3. How to fetch a page after logging in? A. Use a session. You may also need to handle the CSRF token.

📝 Exercises (exercises/)

  • exercise1.md — Top 10 news headlines
  • exercise2.md — GitHub user repository information
  • exercise3.md — Function to check robots.txt

Next Week

Week 8. NumPy / Pandas

Example files

The runnable files used in this lesson. Copy them into your own project and experiment.

examples/01_requests_basic.pyPython
"""requests basic — requires pip install requests"""
import requests

#JSON API Call
r = requests.get("https://httpbin.org/json", timeout=10)
print(r.status_code)
print(r.json())

# GET with params
r = requests.get("https://httpbin.org/get",
    params={"q": "python", "lang": "ko"},
    timeout=10)
print(r.url)
print(r.json()["args"])

# POST
r = requests.post("https://httpbin.org/post",
    data={"key": "value"},
    timeout=10)
print(r.json()["form"])
examples/02_bs4_parse.pyPython
"""BeautifulSoup Basics — pip install beautifulsoup4"""
from bs4 import BeautifulSoup

html = """<html><body>
    <h1>Title</h1>
    <ul>
        <li class="item">Apple</li>
        <li class="item special">Banana</li>
        <li class="item">Grape</li>
    </ul>
    <a href="/about">About</a>
</body></html>"""

soup = BeautifulSoup(html, "html.parser")

#Tag
print(soup.h1.text)
print(soup.find("a")["href"])

# find_all
items = soup.find_all("li")
for li in items:
    print("-", li.text, li.get("class"))

#CSS Selector (powerful)
print([li.text for li in soup.select("li.item")])
print([li.text for li in soup.select("li.special")])
examples/03_session.pyPython
"""Use Session — Maintain same headers/cookies"""
import requests

session = requests.Session()
session.headers.update({"User-Agent": "Mozilla/5.0 (study)"})

#First request
r = session.get("https://httpbin.org/cookies/set?session_id=abc123")
print("Received cookie:", session.cookies.get_dict())

#The next request in the same session automatically sends cookies
r = session.get("https://httpbin.org/cookies")
print("Cookie seen by the server:", r.json())
examples/04_scrape_demo.pyPython
"""Actual Scraping Demo — Using httpbin's HTML Page"""
import requests
from bs4 import BeautifulSoup

URL = "https://httpbin.org/html"
HEADERS = {"User-Agent": "Mozilla/5.0 (study)"}

r = requests.get(URL, headers=HEADERS, timeout=10)
r.raise_for_status()
soup = BeautifulSoup(r.text, "html.parser")

print("=== Title ===")
print(soup.h1.text)

print("=== Part of the Text ===")
p = soup.find("p")
if p:
    print(p.text[:200])

Assignment

Goal

Extract 10 headlines from any news site (or Hacker News).

Requirements

  • requests + BeautifulSoup
  • Set timeout, raise_for_status, and User-Agent.
  • Output the results as CSV or on the screen.

Recommended Sites

Assignment 2. GitHub User Repository Information

Goal

Use the GitHub API to print a list of a specific user's public repositories and star counts.

Endpoint

https://api.github.com/users/{username}/repos

Output Example

User: python
Number of repositories: 30
1. cpython - 50000 stars
2. python-docs-translations - 200 stars
...

Challenge

  • Sort (by star count in descending order)
  • pagination (per_page=100)

Assignment 3. robots.txt Check Function

Goal

A function to check via robots.txt whether a specific URL is allowed to be crawled.

Function Signature

def can_crawl(url: str, user_agent: str = "*") -> bool: ...

Hint

  • Use urllib.robotparser
  • Or parse robots.txt directly

Example of use

print(can_crawl("https://example.com/page"))    # True
print(can_crawl("https://example.com/admin"))   #False (If it is a disallowed path)
Show the reference solution

Try the assignment yourself first — comparing your attempt with a working solution is where most of the learning happens.

solution1.pyPython
"""Headline Scraping — For Operation Check (HN)"""
import requests
from bs4 import BeautifulSoup

URL = "https://news.ycombinator.com"
HEADERS = {"User-Agent": "Mozilla/5.0 (study)"}

r = requests.get(URL, headers=HEADERS, timeout=10)
r.raise_for_status()
soup = BeautifulSoup(r.text, "html.parser")

titles = [a.get_text() for a in soup.select(".titleline > a")][:10]
for i, t in enumerate(titles, 1):
    print(f"{i}. {t}")
solution2.pyPython
import requests

USER = "python"
r = requests.get(f"https://api.github.com/users/{USER}/repos",
    params={"per_page": 100},
    headers={"User-Agent": "study"},
    timeout=10)
r.raise_for_status()
repos = r.json()

print(f"User: {USER}")
print(f"Number of repositories: {len(repos)}")
ranked = sorted(repos, key=lambda x: -x["stargazers_count"])
for i, repo in enumerate(ranked[:10], 1):
    print(f"{i}. {repo['name']} - {repo['stargazers_count']} stars")
solution3.pyPython
from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse

def can_crawl(url: str, user_agent: str = "*") -> bool:
    parsed = urlparse(url)
    robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
    rp = RobotFileParser()
    rp.set_url(robots_url)
    try:
        rp.read()
    except Exception:
        return True   #If robots.txt cannot be read, handle it as allowed for now
    return rp.can_fetch(user_agent, url)

print(can_crawl("https://www.python.org/about/"))
print(can_crawl("https://www.google.com/search?q=python"))

Learn this lesson with a mentor

Work through the Python Series track on your own, or join a guided cohort at our Pitampura campus and online.

Enroll nowTrack overview
Call NowEnroll Now