This python project is a very straightforward comparison between socket and urllib for sending HTTP requests to servers.
Socket is a low-level networking interface, while urllib.request,urlopen() will do all the hard work of making Get requests to the URL provided and do the encode as well. What is returned is an HTTPResponse object.
HTTPResponse object is in bytes. We can use .read() to read the content, or .headers to get all the headers information. Also, with the help of an Open Source library called beautiful soup, the bytes content can be parsed into a BeautifulSoup Object, which is a nested data structure for further analysis of the content and scraping.
For more complicated use of urllib library, here is the python doc
For the simplest example, here is the code :
''' Compare socket with urllib for sending http requests '''
import socket
import urllib.request
# client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# client_socket.connect(('www.yahoo.com', 80))
# request = "GET / HTTP/1.1\r\nHost:https://www.yahoo.com\r\n\r\n"
# client_socket.send(request.encode())
# response = client_socket.recv(4096)
# print(f'************* {len(response)}')
# print(response.decode())
# client_socket.close()
'''urllib library is a lot easier than socket module
for getting http requests and return responses'''
# httpResponse_data = urllib.request.urlopen('https://www.yahoo.com')
# print(f'Headers: {httpResponse_data.headers}')
# use with statement instead:
with urllib.request.urlopen('https://www.yahoo.com') as response_file:
data = response_file.read()
print(data[:1000].decode())However, when we talk about web scraping, we shouldn't skip Requests library. Requests is an open-source python library that makes HTTP requests more human-friendly and simple to use, it is powered by urllib3 and used more nowadays because of its readability and POST/GET freedom, etc.
When dealing with scraping a website, requests and beautiful soup usually join force for processing and finding information on a web page.
Related reading:
1. Find most frequent words in an article online
2.Scrape Ted.com videos
