{ "cells": [ { "cell_type": "markdown", "id": "73f22d29", "metadata": {}, "source": [ "# `kl_divergence_profile` function walkthrough\n", "This function calculates the Kullback-Leiber (KL) divergence of different areas and the total population, in order to create a divergence profile across the whole area. This provides a multi-scalar measurement of the segregation, understanding how each area of a population interacts with those areas in close and distant proximity to it.\n", "\n", "For theoretical background to this methodology, please see the [Olteanu et al. (2019)](https://doi.org/10.1073/pnas.1900192116) article that inspired the creation of this function within PySAL's library.\n", "\n", "Many thanks to the work of [Cécile de Bézenac](https://github.com/ceciledebezenac/segregation_index), who's primary iteration of a wrapped code to compute this metric was incredibly helpful in directing this function's interpretation." ] }, { "cell_type": "markdown", "id": "73e3c286", "metadata": {}, "source": [ "## Contents\n", "1. **[The function](#The-function)** - the full function from the PySAL library\n", "\n", "\n", "2. **[Cincinnati example dataset](#Cincinnati-(cincin)-example-dataset)** - an overview of the example dataset used in the walkthrough, and creating the different forms of input acceptable for the data\n", "\n", "\n", "2. **[Making the function work](#Making-the-function-work)** - providing the functions with the correct inputs to ensure that this will run successfully\n", "\n", "\n", "3. **[Using the outputs](#Using-the-outputs)** - how the outputs can be deconstructed to create useful visualisations\n", "\n", "\n", "4. **[How the function works](#How-the-function-works)** - taking a step-by-step work through of the inner workings of the function to understand exactly what it is doing" ] }, { "cell_type": "markdown", "id": "666242aa", "metadata": {}, "source": [ "## The function" ] }, { "cell_type": "code", "execution_count": 1, "id": "318b6aae", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import geopandas as gpd\n", "import pandas as pd\n", "\n", "from scipy.spatial.distance import pdist, squareform\n", "from scipy.special import rel_entr as relative_entropy\n", "\n", "\n", "def kl_divergence_profile(populations, coordinates = None, metric = 'euclidean'):\n", " \"\"\"\n", " A segregation metric, using Kullback-Leiber (KL) divergence to quantify the\n", " difference in the population characteristics between (1) an area and (2) the total population.\n", "\n", " This function utilises the methodology proposed in\n", " Olteanu et al. (2019): 'Segregation through the multiscalar lens'. Which can be\n", " found here: https://doi.org/10.1073/pnas.1900192116\n", "\n", " Arguments\n", " ----------\n", " populations : GeoPandas GeoDataFrame object\n", " Pandas DataFrame object\n", " NumPy Array object\n", " Population information of raw group numbers (not percentages) to be\n", " included in the analysis.\n", " coordinates : GeoPandas GeoSeries object\n", " NumPy Array object\n", " Spatial information relating to the areas to be included in the analysis.\n", " metric : Acceptable inputs to `scipy.spatial.distance.pdist` - including:\n", " ‘braycurtis’, ‘canberra’, ‘chebyshev’, ‘cityblock’, ‘correlation’,\n", " ‘cosine’, ‘dice’, ‘euclidean’, ‘hamming’, ‘jaccard’, ‘jensenshannon’,\n", " ‘kulsinski’, ‘mahalanobis’, ‘matching’, ‘minkowski’, ‘rogerstanimoto’,\n", " ‘russellrao’, ‘seuclidean’, ‘sokalmichener’, ‘sokalsneath’, ‘sqeuclidean’, ‘yule’.\n", " Distance metric for calculating pairwise distances,\n", " using `scipy.spatial.distance.pdist` - 'euclidean' by default.\n", "\n", " Returns\n", " ----------\n", " observation : an identifier of the area that forms the centre of the aggregation\n", " of population, from which the divergence is calculated.\n", " distance : how far the most recently aggregated area is from the 'observation'\n", " area, starting at zero for each observation to represent that only the\n", " 'observation' area being aggregated.\n", " divergence : the KL divergence measure, between the aggregated population and the\n", " total population, will converge to zero for the final row of each\n", " observation to represent that the total population is covered.\n", " population_covered : the population count within the aggregated population.\n", " Returns a concatenated object of Pandas dataframes. Each dataframe contains a\n", " set of divergence levels between an area and the total population. These areas\n", " become consecutively larger, starting from a single location and aggregating\n", " outward from this location, until the area represents the total population.\n", " Thus, together the divergence levels within a dataframe represent a profile\n", " of divergence from an area. The concatenated object is the collection of these\n", " divergence profiles for every areas within the total population.\n", "\n", " Example\n", " ----------\n", " from libpysal.examples import get_path\n", " from libpysal.examples import load_example\n", " cincin = load_example('Cincinnati')\n", " cincin.get_file_list()\n", " cincin_df = gpd.read_file(cincin.get_path('cincinnati.shp'))\n", " cincin_ethnicity = cincin_df[[\"WHITE\", \"BLACK\", \"AMINDIAN\", \"ASIAN\", \"HAWAIIAN\", \"OTHER_RACE\", \"geometry\"]]\n", " cincin_ethnicity.head()\n", " kl_divergence_profile(cincin_ethnicity)\n", " \"\"\"\n", " # Store the observation index to return with the results\n", " if hasattr(populations, 'index'):\n", " indices = populations.index\n", " else:\n", " indices = np.arange(len(populations))\n", "\n", " # Check for geometry present in populations argument\n", " if hasattr(populations, 'geometry'):\n", " if coordinates is None:\n", " coordinates = populations.geometry\n", " populations = populations.drop(populations.geometry.name, axis = 1).values\n", " populations = np.asarray(populations)\n", "\n", " # Creating consistent coordinates - GeoSeries input\n", " if hasattr(coordinates,'geometry'):\n", " centroids = coordinates.geometry.centroid\n", " coordinates = np.column_stack((centroids.x, centroids.y))\n", " # Creating consistent coordinates - Array input\n", " else:\n", " assert len(coordinates) == len(populations), \"Length of coordinates input needs to be of the same length as populations input\"\n", "\n", " # Creating distance matrix using defined metric (default euclidean distance)\n", " dist_matrix = squareform(pdist(coordinates, metric = metric))\n", "\n", " # Preparing list for results\n", " results = []\n", "\n", " # Loop to calculate KL divergence profile\n", " for (i, distances) in enumerate(dist_matrix):\n", "\n", " # Creating the q and r objects\n", " sorted_indices = np.argsort(distances)\n", " cumul_pop_by_group = np.cumsum(populations[sorted_indices], axis = 0)\n", " obs_cumul_pop = np.sum(cumul_pop_by_group, axis = 1)[:, np.newaxis]\n", " q_cumul_proportions = cumul_pop_by_group / obs_cumul_pop\n", " total_pop_by_group = np.sum(populations, axis = 0, keepdims = True)\n", " total_pop = np.sum(populations)\n", " r_total_proportions = total_pop_by_group / total_pop\n", "\n", " # Input q and r objects into relative entropy (KL divergence) function\n", " kl_divergence = relative_entropy(q_cumul_proportions,\n", " r_total_proportions).sum(axis = 1)\n", "\n", " # Creating an output dataframe\n", " output = pd.DataFrame().from_dict(dict(\n", " observation = indices[i],\n", " distance = distances[sorted_indices],\n", " divergence = kl_divergence,\n", " population_covered = obs_cumul_pop.sum(axis=1)\n", " ))\n", "\n", " # Append (bring together) all outputs into results list\n", " results.append(output)\n", "\n", " return(pd.concat(results))\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "0c64ff19", "metadata": {}, "source": [ "## Cincinnati (cincin) example dataset" ] }, { "cell_type": "markdown", "id": "9fa2a609", "metadata": {}, "source": [ "For this example, we will be using an example dataset that is available via the [`examples`](https://pysal.org/notebooks/lib/libpysal/Example_Datasets.html) package of **`libpysal`** - it is a remote dataset, originating from the GeoDa program, an output of the [Center for Spatial Data Science at the Unversity of Chicago](https://spatial.uchicago.edu).\n", "\n", "This dataset is entitled '[**2008 Cincinnati Crime + Socio-Demographics**](https://geodacenter.github.io/data-and-lab/walnut_hills/)', and contains \"*Crime and socio-demographic data for the Clifton, Walnut Hills, Evanston, and Avondale neighborhoods in Cincinnati, OH for the last 6 months of 2008*\".\n", "\n", "Throughout this workbook, we will only be inputting the socio-demographic information it possesses. The total population of Cincinnati can be broken down within the data by six different ethnic groups: *AMINDIAN* (American-Indian), *ASIAN*, *BLACK*, *HAWAIIAN*, *WHITE*, and *OTHER_RACE*.\n", "\n", "This data will be referred to it as ***cincin*** throughout." ] }, { "cell_type": "markdown", "id": "834ae96c", "metadata": {}, "source": [ "You can read in this example dataset, and then repeat all the code in this notebook, by doing the following:" ] }, { "cell_type": "code", "execution_count": 2, "id": "2d43890c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Example not available: Cincinnati\n", "Example not downloaded: Chicago parcels\n", "Example not downloaded: Chile Migration\n", "Example not downloaded: Spirals\n" ] } ], "source": [ "import geopandas as gpd\n", "from libpysal.examples import get_path\n", "from libpysal.examples import load_example\n", "\n", "cincin = load_example('Cincinnati')\n", "cincin = gpd.read_file(cincin.get_path('cincinnati.shp'))\n", "cincin = cincin.set_index('ID')" ] }, { "cell_type": "markdown", "id": "ad24f596", "metadata": {}, "source": [ "Having read in this shapefile - as ***cincin*** - the data required for this analysis can be collected using either a *geodataframe* (gdf) or a combination of a *dataframe* (df) and *geoseries* data." ] }, { "cell_type": "code", "execution_count": 3, "id": "2295eac4", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
| \n", " | WHITE | \n", "BLACK | \n", "AMINDIAN | \n", "ASIAN | \n", "HAWAIIAN | \n", "OTHER_RACE | \n", "geometry | \n", "
|---|---|---|---|---|---|---|---|
| ID | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| 726907.0 | \n", "433.0 | \n", "32.0 | \n", "0.0 | \n", "5.0 | \n", "0.0 | \n", "2.0 | \n", "POLYGON ((1407302.966 415693.734, 1407473.141 ... | \n", "
| 695744.0 | \n", "16.0 | \n", "66.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "POLYGON ((1398841.243 416718.444, 1399605.382 ... | \n", "
| 695762.0 | \n", "15.0 | \n", "12.0 | \n", "0.0 | \n", "1.0 | \n", "0.0 | \n", "1.0 | \n", "POLYGON ((1398733.468 416975.853, 1398794.240 ... | \n", "
| 695780.0 | \n", "12.0 | \n", "103.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "2.0 | \n", "POLYGON ((1399564.078 416046.633, 1399605.382 ... | \n", "
| 695798.0 | \n", "52.0 | \n", "39.0 | \n", "0.0 | \n", "1.0 | \n", "0.0 | \n", "0.0 | \n", "POLYGON ((1398841.243 416718.444, 1398733.468 ... | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 703629.0 | \n", "5.0 | \n", "102.0 | \n", "0.0 | \n", "0.0 | \n", "1.0 | \n", "2.0 | \n", "POLYGON ((1401900.256 422880.130, 1401964.217 ... | \n", "
| 703648.0 | \n", "0.0 | \n", "85.0 | \n", "0.0 | \n", "1.0 | \n", "0.0 | \n", "0.0 | \n", "POLYGON ((1402856.266 422494.690, 1403591.085 ... | \n", "
| 703666.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "POLYGON ((1402861.897 422749.576, 1402925.846 ... | \n", "
| 703686.0 | \n", "0.0 | \n", "270.0 | \n", "1.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "POLYGON ((1402709.472 423554.407, 1403020.559 ... | \n", "
| 703708.0 | \n", "10.0 | \n", "43.0 | \n", "1.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "POLYGON ((1402421.734 424653.671, 1402817.865 ... | \n", "
457 rows × 7 columns
\n", "| \n", " | WHITE | \n", "BLACK | \n", "AMINDIAN | \n", "ASIAN | \n", "HAWAIIAN | \n", "OTHER_RACE | \n", "
|---|---|---|---|---|---|---|
| ID | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| 726907.0 | \n", "433.0 | \n", "32.0 | \n", "0.0 | \n", "5.0 | \n", "0.0 | \n", "2.0 | \n", "
| 695744.0 | \n", "16.0 | \n", "66.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "
| 695762.0 | \n", "15.0 | \n", "12.0 | \n", "0.0 | \n", "1.0 | \n", "0.0 | \n", "1.0 | \n", "
| 695780.0 | \n", "12.0 | \n", "103.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "2.0 | \n", "
| 695798.0 | \n", "52.0 | \n", "39.0 | \n", "0.0 | \n", "1.0 | \n", "0.0 | \n", "0.0 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 703629.0 | \n", "5.0 | \n", "102.0 | \n", "0.0 | \n", "0.0 | \n", "1.0 | \n", "2.0 | \n", "
| 703648.0 | \n", "0.0 | \n", "85.0 | \n", "0.0 | \n", "1.0 | \n", "0.0 | \n", "0.0 | \n", "
| 703666.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "
| 703686.0 | \n", "0.0 | \n", "270.0 | \n", "1.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "
| 703708.0 | \n", "10.0 | \n", "43.0 | \n", "1.0 | \n", "0.0 | \n", "0.0 | \n", "0.0 | \n", "
457 rows × 6 columns
\n", "| \n", " | geometry | \n", "
|---|---|
| ID | \n", "\n", " |
| 726907.0 | \n", "POLYGON ((1407302.966 415693.734, 1407473.141 ... | \n", "
| 695744.0 | \n", "POLYGON ((1398841.243 416718.444, 1399605.382 ... | \n", "
| 695762.0 | \n", "POLYGON ((1398733.468 416975.853, 1398794.240 ... | \n", "
| 695780.0 | \n", "POLYGON ((1399564.078 416046.633, 1399605.382 ... | \n", "
| 695798.0 | \n", "POLYGON ((1398841.243 416718.444, 1398733.468 ... | \n", "
| ... | \n", "... | \n", "
| 703629.0 | \n", "POLYGON ((1401900.256 422880.130, 1401964.217 ... | \n", "
| 703648.0 | \n", "POLYGON ((1402856.266 422494.690, 1403591.085 ... | \n", "
| 703666.0 | \n", "POLYGON ((1402861.897 422749.576, 1402925.846 ... | \n", "
| 703686.0 | \n", "POLYGON ((1402709.472 423554.407, 1403020.559 ... | \n", "
| 703708.0 | \n", "POLYGON ((1402421.734 424653.671, 1402817.865 ... | \n", "
457 rows × 1 columns
\n", "| \n", " | observation | \n", "distance | \n", "divergence | \n", "population_covered | \n", "
|---|---|---|---|---|
| 0 | \n", "726907.0 | \n", "0.000000 | \n", "8.964293e-01 | \n", "472.0 | \n", "
| 1 | \n", "726907.0 | \n", "674.319177 | \n", "3.970494e-01 | \n", "654.0 | \n", "
| 2 | \n", "726907.0 | \n", "755.315782 | \n", "3.966244e-01 | \n", "728.0 | \n", "
| 3 | \n", "726907.0 | \n", "830.744890 | \n", "3.082997e-01 | \n", "872.0 | \n", "
| 4 | \n", "726907.0 | \n", "1067.478163 | \n", "3.110239e-01 | \n", "981.0 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 452 | \n", "703708.0 | \n", "13063.729732 | \n", "1.881053e-06 | \n", "37805.0 | \n", "
| 453 | \n", "703708.0 | \n", "13099.457827 | \n", "1.377660e-06 | \n", "37874.0 | \n", "
| 454 | \n", "703708.0 | \n", "13163.435867 | \n", "1.492472e-06 | \n", "37919.0 | \n", "
| 455 | \n", "703708.0 | \n", "13170.741000 | \n", "1.595123e-07 | \n", "37963.0 | \n", "
| 456 | \n", "703708.0 | \n", "13223.508518 | \n", "0.000000e+00 | \n", "38012.0 | \n", "
208849 rows × 4 columns
\n", "| \n", " | observation | \n", "distance | \n", "divergence | \n", "population_covered | \n", "
|---|---|---|---|---|
| 0 | \n", "726907.0 | \n", "0.000000 | \n", "8.964293e-01 | \n", "472.0 | \n", "
| 1 | \n", "726907.0 | \n", "674.319177 | \n", "3.970494e-01 | \n", "654.0 | \n", "
| 2 | \n", "726907.0 | \n", "755.315782 | \n", "3.966244e-01 | \n", "728.0 | \n", "
| 3 | \n", "726907.0 | \n", "830.744890 | \n", "3.082997e-01 | \n", "872.0 | \n", "
| 4 | \n", "726907.0 | \n", "1067.478163 | \n", "3.110239e-01 | \n", "981.0 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 452 | \n", "703708.0 | \n", "13063.729732 | \n", "1.881053e-06 | \n", "37805.0 | \n", "
| 453 | \n", "703708.0 | \n", "13099.457827 | \n", "1.377660e-06 | \n", "37874.0 | \n", "
| 454 | \n", "703708.0 | \n", "13163.435867 | \n", "1.492472e-06 | \n", "37919.0 | \n", "
| 455 | \n", "703708.0 | \n", "13170.741000 | \n", "1.595123e-07 | \n", "37963.0 | \n", "
| 456 | \n", "703708.0 | \n", "13223.508518 | \n", "0.000000e+00 | \n", "38012.0 | \n", "
208849 rows × 4 columns
\n", "| \n", " | observation | \n", "distance | \n", "divergence | \n", "population_covered | \n", "
|---|---|---|---|---|
| 0 | \n", "726907.0 | \n", "0.000000 | \n", "8.964293e-01 | \n", "472.0 | \n", "
| 1 | \n", "726907.0 | \n", "674.319177 | \n", "3.970494e-01 | \n", "654.0 | \n", "
| 2 | \n", "726907.0 | \n", "755.315782 | \n", "3.966244e-01 | \n", "728.0 | \n", "
| 3 | \n", "726907.0 | \n", "830.744890 | \n", "3.082997e-01 | \n", "872.0 | \n", "
| 4 | \n", "726907.0 | \n", "1067.478163 | \n", "3.110239e-01 | \n", "981.0 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 452 | \n", "703708.0 | \n", "13063.729732 | \n", "1.881053e-06 | \n", "37805.0 | \n", "
| 453 | \n", "703708.0 | \n", "13099.457827 | \n", "1.377660e-06 | \n", "37874.0 | \n", "
| 454 | \n", "703708.0 | \n", "13163.435867 | \n", "1.492472e-06 | \n", "37919.0 | \n", "
| 455 | \n", "703708.0 | \n", "13170.741000 | \n", "1.595123e-07 | \n", "37963.0 | \n", "
| 456 | \n", "703708.0 | \n", "13223.508518 | \n", "0.000000e+00 | \n", "38012.0 | \n", "
208849 rows × 4 columns
\n", "| \n", " | observation | \n", "distance | \n", "divergence | \n", "population_covered | \n", "geometry | \n", "
|---|---|---|---|---|---|
| 0 | \n", "654795.0 | \n", "0.0 | \n", "0.142210 | \n", "69.0 | \n", "POLYGON ((1398269.926 411448.786, 1398333.955 ... | \n", "
| 1 | \n", "654816.0 | \n", "0.0 | \n", "0.060205 | \n", "49.0 | \n", "POLYGON ((1397564.804 411646.644, 1397600.475 ... | \n", "
| 2 | \n", "654853.0 | \n", "0.0 | \n", "0.210239 | \n", "0.0 | \n", "POLYGON ((1397634.522 412227.972, 1397580.224 ... | \n", "
| 3 | \n", "654869.0 | \n", "0.0 | \n", "0.189339 | \n", "44.0 | \n", "POLYGON ((1397788.495 411495.938, 1397826.597 ... | \n", "
| 4 | \n", "654889.0 | \n", "0.0 | \n", "0.210239 | \n", "142.0 | \n", "POLYGON ((1397634.522 412227.972, 1397695.312 ... | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 452 | \n", "739686.0 | \n", "0.0 | \n", "1.244836 | \n", "45.0 | \n", "POLYGON ((1410127.237 417672.072, 1410268.460 ... | \n", "
| 453 | \n", "741823.0 | \n", "0.0 | \n", "0.934698 | \n", "29.0 | \n", "POLYGON ((1410140.124 416441.192, 1410495.720 ... | \n", "
| 454 | \n", "741843.0 | \n", "0.0 | \n", "0.945546 | \n", "256.0 | \n", "POLYGON ((1408692.225 416938.374, 1408734.116 ... | \n", "
| 455 | \n", "741889.0 | \n", "0.0 | \n", "1.244836 | \n", "10.0 | \n", "POLYGON ((1411000.065 417361.617, 1411163.889 ... | \n", "
| 456 | \n", "741903.0 | \n", "0.0 | \n", "0.276574 | \n", "357.0 | \n", "POLYGON ((1410268.460 418944.037, 1410337.877 ... | \n", "
457 rows × 5 columns
\n", "