How to Download a Webpage in Python

The following python program demonstrates the use of urllib module to download a webpage to a local folder. Please note that this program downloads the webpage html content only, it doesn’t download the linked images or other resources. The following program also demonstrates use of exception handling.

# python 3 only
import urllib.request
from urllib.error import URLError

target_file_path = "/Users/user/downloaded.html" # downloaded page saved here

try:
    response = urllib.request.urlopen('http://www.google.com/')
    html_content = response.read()

    with open(target_file_path,"wb") as fp:
        fp.write(html_content)

except URLError as e:
    print("Unable to download page: "+str(e.reason))

If you are looking for a way to scrap entire web pages including resources, you should look at the scrapy library. If you are more interested in the parsing of the web content, you should look at Beautiful Soup.

Do you have a programming problem that you are unable to solve? Please send us your problem and we will publish the solution! Please email us.

Leave a Reply