Web scraping to build social research data (online) (join a waiting list)

Course Code


Organised by

The University of Edinburgh


Dr Diarmuid McDonnell, Research Fellow, Third Sector Research Centre, University of Birmingham




Online delivery by The University of Edinburgh


View in Google Maps  (EH8 9LN)


Laura Marshall (Monday - Wednesday)
Currently working from home following government guidance


There is an unprecedented amount of information on the internet that could usefully be harvested in order to build social science research datasets.

This one-day online course will showcase suitable techniques for web scraping.

The value, logic and process of capturing data stored on websites will be described in detail, and practical examples and exercises will be demonstrated using the Python programming language.

It is most suited to empirical social science researchers.

Course Contents

  • The structure of a web page and how it can be navigated to collect data
  • Scraping, formatting and saving data scraped from the web
  • Ethical and legal considerations when scraping data
  • Introduction to the Python programming language

Learning Outcomes

Participants will gain

  • A clearer understanding of the concept of web scraping
  • An understanding of the fundamental skills required for scraping a web page
  • An understanding of how to marshal scrapped data and construct a dataset suitable for social research
  • An awareness of the scope and limitation of web scraping approaches
  • An awareness of the theoretical, practical and ethical issues relating to scraping data
  • Experience and examples of using the Python programming language for scraping data from a web page

The course will be delivered online and will be held from 10:00 - 16:00.

Course Programme

10:00 – 10:10          Welcome

10:10 – 10:45          The value, logic and practice of web scraping

10:45 – 11:00          Tea Break

11:00 – 12:00          Web scraping using Python: Example 1

12:00 – 13:00          Lunch

13:00 – 13:45          Web scraping using Python: Example 2

13:45 – 14:00          Tea Break

14:00 – 14:50          Practical exercise

14:50 – 15:00          Break

15:00 – 15:30          Ethical and legal considerations of web scraping

15:30 – 16:00          Concluding remarks


Entry (no or almost no prior knowledge)


The fee per teaching day is: • £30 per day for UK/EU registered students • £60 per day for staff at UK/EU academic institutions, UK/EU Research Councils researchers, UK/EU public sector staff and staff at UK/EU registered charity organisations and recognised UK/EU research institutions. • £100 per day for all other participants. Cancellation Policy: Please be reminded that you will be charged the full registration fee if you cancel your place within 4 weeks before the training delivery date or you fail to attend. If you are able to fill the place on the course you are cancelling then the cancellation charge will not apply

Website and registration




Digital Social Research, Big data, Web scraping, Python, Digital Social Research, research ethics, online data collection

Related publications and presentations

Digital Social Research
Big data

Back to archive...