Web scraping and Covid-19
Presenter(s): Dr Diarmuid McDonnell
This online resource is intended to provide you with an introduction to web scraping techniques for collecting data of interest to social scientists. The value, logic and process of capturing data stored on the web are described in detail, and practical examples and exercises are demonstrated using the Python programming language
Web scraping and Covid-19: Introduction
In this video, Dr Diarmuid McDonnell introduces the logic, practice and value of web scraping, as well as demonstrating an example from the instructor’s research area.
Web scraping and Covid-19: html
In the second video in this series, Dr Diarmuid McDonnell covers how to interpret the structure of web pages, an essential piece of knowledge for scraping their contents
Web scraping and Covid-19: Example 1
In this third video of the series, Dr Diarmuid McDonnell demonstrates a simple web scraping example: extracting text from a web page.
Web scraping and Covid-19: Example 2
In the fourth video on web scraping, Dr Diarmuid McDonnell demonstrates a simple web scraping example: downloading files stored on a web page.
Web scraping and Covid-19: Covid 1
This video shows how web scraping can be used to collect up-to-date summary statistics on the COVID-19 pandemic.
Web scraping and Covid-19: Covid 2
This video shows how web scraping can be used to collect country-level data on the COVID-19 pandemic.
Web scraping and Covid-19: Ethics-and-conclusion
In this final video in the series, Diarmuid reflects on some of the legal and ethical implications of web scraping, and summarises what we have covered. Two supplementary videos, on how to use Jupyter Notebooks to execute the Python code underpinning the web scraping activities can be viewed under related datasets and other links.
About the author
Diarmuid McDonnell is a Lecturer in Social Sciences at the University of the West of Scotland, having previously worked at the universities of Edinburgh and Birmingham. His research applies a range of quantitative methods to the study of nonprofit organisations, in particular how they are regulated, governed and funded. Diarmuid has methodological interests in the use of administrative data for social science research, data science, and policy evaluation methods
- Published on: 21 September 2021
- Event hosted by: University of the West of Scotland
- Keywords: Digital Social Research | Big data | Web scraping | Python |
- To cite this resource:
Dr Diarmuid McDonnell. (2021). Web scraping and Covid-19. National Centre for Research Methods online learning resource. Available at https://www.ncrm.ac.uk/resources/online/all/?main&id=20782 [accessed: 24 April 2024]
⌃BACK TO TOP