Data_source_cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data acquisition and cleaning\n",
    "\n",
    "The data used in this study were collected by UK police forces and made available to the public, excluding the sensitive variables, through the open data UK government site (https://data.gov.uk/).\n",
    "\n",
    "The database on UK accidents holds records on road accidents with the current set of definitions and detail of information since 1979. UK police forces use a standard form STAT19 (https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/230590/stats19.pdf) to report information on road accidents.  \n",
    "These datasets are made available to the public under the Open Government Licence (http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/).\n",
    "\n",
    "The web address for the download is the following: https://data.gov.uk/dataset/cb7ae6f0-4be6-4935-9277-47e5ce24a11f/road-safety-data\n",
    "\n",
    "The databases provide detailed data about the circumstances of road accidents involving injuries or deaths. In this latter case the data refer to people killed immediately or who died within 30 days since the accident.\n",
    "\n",
    "For this study, only the most recent years were taken into account. In particular, only the 2017 and 2018 'accident' databases were considered and merged into a single database (the details of 'casualties' and 'vehicles' databases were not taken into account). The 2016 database will be used for model testing.\n",
    "\n",
    "The main strengths of these databases are the following:\n",
    "- their data are rigorously validated before distribution, for example by the relevant police forces or local road authorities;\n",
    "- they are collected with the same methodology across the whole UK;\n",
    "- they include a lot of details regarding the accidents.\n",
    "\n",
    "The main weaknesses of these data are due to the fact that they record only accidents involving at least one vehicle in which at least one person was injured, and which were reported to the police. The first point determines the fact that these data do not contain details of damage-only accidents, i.e. with no human casualties. The second point has the consequence that accidents on private roads or car parks are not reported, as well as a considerable proportion of non-fatal injury accidents.  \n",
    "This may determine an under-sampling of non-fatal injury accidents.\n",
    "\n",
    "All the data variables are coded instead of displaying textual strings; for some columns '-1' means an unknown or undefined value. Each accident is identified by a unique accident_index; 31 additional columns, with meaningful names, describe the accident details.\n",
    "\n",
    "The accident severity column does not contain null values. Some of the other columns contains null (or -1) values.  \n",
    "For the analysis, the following columns were dropped:\n",
    "- 'Accident_Index', since no connection to the casualties or vehicle databases were taken into account;\n",
    "- 'Location_Easting_OSGR', 'Location_Northing_OSGR', 'Longitude', 'Latitude', 'Police_Force', 'Local_Authority_(District)', 'Local_Authority_(Highway)', '1st_Road_Number', '2nd_Road_Number', 'LSOA_of_Accident_Location', since the geographical position or the department in which the accident happened was not taken into account;\n",
    "- 'Junction_Control', '2nd_Road_Class', since roughly 40% of the values for these columns were null;\n",
    "- 'Pedestrian_Crossing-Human_Control', 'Pedestrian_Crossing-Physical_Facilities', because a distinction between accidents involving pedestrians and other accidents was not considered in this study;\n",
    "- 'Did_Police_Officer_Attend_Scene_of_Accident', 'Number_of_Vehicles', 'Number_of_Casualties', because the aim of the model is to predict the severity of an accident given the particular weather, time and road conditions, not the possible number of vehicles involved and casualties, or the attendance of a police officer.\n",
    "\n",
    "From the 'Date' column only the month was kept; from the 'Time' column only the hour was kept.  \n",
    "For columns 'Special_Conditions_at_Site' and 'Carriageway_Hazards' null values were substituted with the mode, corresponding to no special conditions (value '0') and no carriageway hazards (value '0'), respectively; in case of particular conditions it is highly probable that it would have been reported by the those attending to the accident. A similar reasoning can be done for the missing values of 'Junction_Detail', considering 'Not at junction or within 20 metres' (i.e. '0' value) as the most probable case for those accidents (0.55% of the total accidents).  \n",
    "All other rows containing null values in at least one of the variables taken into account were dropped. The original combined database contained 252617 rows. After this cleaning operation, the database contained 249441 rows; less than 1.5% of the initial data were lost.\n",
    "\n",
    "In addition to the accident severity column ('Accident_Severity'), which will be used as target column, 13 other columns were kept:\n",
    "- 'Day_of_Week', 'Hour', 'Month', to identify the time during the year, the week and the day when the accident happened\n",
    "- '1st_Road_Class', 'Road_Type', 'Speed_limit', 'Junction_Detail', 'Urban_or_Rural_Area', to identify the type of road involved in the accident, and its characteristics\n",
    "- 'Light_Conditions', 'Weather_Conditions' to identify the visibility during the accident\n",
    "- 'Road_Surface_Conditions', 'Special_Conditions_at_Site', 'Carriageway_Hazards', to identify the temporary characteristics of the road when the accident happened."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}