Web scraping and Covid-19

Presenter(s): Dr Diarmuid McDonnell

This online resource is intended to provide you with an introduction to web scraping techniques for collecting data of interest to social scientists. The value, logic and process of capturing data stored on the web are described in detail, and practical examples and exercises are demonstrated using the Python programming language

Web scraping and Covid-19: Introduction

In this video, Dr Diarmuid McDonnell introduces the logic, practice and value of web scraping, as well as demonstrating an example from the instructor’s research area.


  Download transcript |   [ 2 Views ]


Web scraping and Covid-19: html

In the second video in this series, Dr Diarmuid McDonnell covers how to interpret the structure of web pages, an essential piece of knowledge for scraping their contents


   Download transcript    |   [ 46 Views ]


Web scraping and Covid-19: Example 1

In this third video of the series, Dr Diarmuid McDonnell demonstrates a simple web scraping example: extracting text from a web page.


   Download transcript    |   [ 63 Views ]


Web scraping and Covid-19: Example 2

In the fourth video on web scraping, Dr Diarmuid McDonnell demonstrates a simple web scraping example: downloading files stored on a web page.


   Download transcript    |   [ 32 Views ]


Web scraping and Covid-19: Covid 1

This video shows how web scraping can be used to collect up-to-date summary statistics on the COVID-19 pandemic.


   Download transcript    |   [ 166 Views ]


Web scraping and Covid-19: Covid 2

This video shows how web scraping can be used to collect country-level data on the COVID-19 pandemic.


   Download transcript    |   [ 122 Views ]


Web scraping and Covid-19: Ethics-and-conclusion

In this final video in the series, Diarmuid reflects on some of the legal and ethical implications of web scraping, and summarises what we have covered. Two supplementary videos, on how to use Jupyter Notebooks to execute the Python code underpinning the web scraping activities can be viewed under related datasets and other links.


   Download transcript    |   [ 81 Views ]


About the author

Diarmuid McDonnell is a Lecturer in Social Sciences at the University of the West of Scotland, having previously worked at the universities of Edinburgh and Birmingham. His research applies a range of quantitative methods to the study of nonprofit organisations, in particular how they are regulated, governed and funded. Diarmuid has methodological interests in the use of administrative data for social science research, data science, and policy evaluation methods

Primary author profile page