Analyzing hurricane tracks - Part 2/3¶

This is the second part to a three part set of notebooks that process and analyze historic hurricane tracks. In the previous notebook we saw

1. downloading historic hurricane datasets using Python
2. cleaning and merging hurricane observations using DASK
3. aggregating point observations into hurricane tracks using ArcGIS GeoAnalytics server

In this notebook you will analyze the aggregated tracks to answer important questions about prevalance of hurricanes, their seasonality, their density, places where they make landfall and investigate the communities that are most affected.

Import the libraries necessary for this notebook.

In [1]:

# import ArcGIS Libraries
from arcgis.gis import GIS
from arcgis.geometry import filters
from arcgis.geocoding import geocode
from arcgis.features.manage_data import overlay_layers
from arcgis.geoenrichment import enrich

# import Pandas for data exploration
import pandas as pd
import numpy as np
from scipy import stats

# import plotting libraries
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

# import display tools
from pprint import pprint
from IPython.display import display

# import system libs
from sys import getsizeof

# import ArcGIS Libraries from arcgis.gis import GIS from arcgis.geometry import filters from arcgis.geocoding import geocode from arcgis.features.manage_data import overlay_layers from arcgis.geoenrichment import enrich # import Pandas for data exploration import pandas as pd import numpy as np from scipy import stats # import plotting libraries import matplotlib.pyplot as plt import seaborn as sns %matplotlib inline # import display tools from pprint import pprint from IPython.display import display # import system libs from sys import getsizeof

In [3]:

gis = GIS('home')

gis = GIS('home')

In [2]:

import warnings
warnings.filterwarnings('ignore')

import warnings warnings.filterwarnings('ignore')

Access aggregated hurricane data¶

Below, we access the tracks aggregated using GeoAnalytics in the previous notebook.

In [4]:

hurricane_tracks_item = gis.content.search('title:hurricane_tracks_aggregated_ga')[0]
hurricane_fl = hurricane_tracks_item.layers[0]

hurricane_tracks_item = gis.content.search('title:hurricane_tracks_aggregated_ga')[0] hurricane_fl = hurricane_tracks_item.layers[0]

The GeoAnalytics step calculated summary statistics of all numeric fields. However only a few of the columns are of interest to us.

In [5]:

pprint([f['name'] for f in hurricane_fl.properties.fields], compact=True, width=80)

pprint([f['name'] for f in hurricane_fl.properties.fields], compact=True, width=80)

['objectid', 'serial_num', 'count', 'count_col_1', 'sum_col_1', 'min_col_1',
 'max_col_1', 'mean_col_1', 'range_col_1', 'sd_col_1', 'var_col_1',
 'count_season', 'sum_season', 'min_season', 'max_season', 'mean_season',
 'range_season', 'sd_season', 'var_season', 'count_num', 'sum_num', 'min_num',
 'max_num', 'mean_num', 'range_num', 'sd_num', 'var_num', 'count_basin',
 'any_basin', 'count_sub_basin', 'any_sub_basin', 'count_name', 'any_name',
 'count_iso_time', 'any_iso_time', 'count_nature', 'any_nature', 'count_center',
 'any_center', 'count_track_type', 'any_track_type', 'count_current_basin',
 'any_current_basin', 'count_latitude_merged', 'sum_latitude_merged',
 'min_latitude_merged', 'max_latitude_merged', 'mean_latitude_merged',
 'range_latitude_merged', 'sd_latitude_merged', 'var_latitude_merged',
 'count_longitude_merged', 'sum_longitude_merged', 'min_longitude_merged',
 'max_longitude_merged', 'mean_longitude_merged', 'range_longitude_merged',
 'sd_longitude_merged', 'var_longitude_merged', 'count_wind_merged',
 'sum_wind_merged', 'min_wind_merged', 'max_wind_merged', 'mean_wind_merged',
 'range_wind_merged', 'sd_wind_merged', 'var_wind_merged',
 'count_pressure_merged', 'sum_pressure_merged', 'min_pressure_merged',
 'max_pressure_merged', 'mean_pressure_merged', 'range_pressure_merged',
 'sd_pressure_merged', 'var_pressure_merged', 'count_grade_merged',
 'sum_grade_merged', 'min_grade_merged', 'max_grade_merged',
 'mean_grade_merged', 'range_grade_merged', 'sd_grade_merged',
 'var_grade_merged', 'count_eye_dia_merged', 'sum_eye_dia_merged',
 'min_eye_dia_merged', 'max_eye_dia_merged', 'mean_eye_dia_merged',
 'range_eye_dia_merged', 'sd_eye_dia_merged', 'var_eye_dia_merged',
 'track_duration', 'end_datetime', 'start_datetime']

Below we select the following fields for the rest of this analysis.

In [9]:

fields_to_query = ['objectid', 'count', 'min_season', 'any_basin', 'any_sub_basin',
                   'any_name', 'mean_latitude_merged', 'mean_longitude_merged',
                   'max_wind_merged', 'range_wind_merged', 'min_pressure_merged',
                   'range_pressure_merged', 'max_eye_dia_merged', 'track_duration',
                   'end_datetime', 'start_datetime']

fields_to_query = ['objectid', 'count', 'min_season', 'any_basin', 'any_sub_basin', 'any_name', 'mean_latitude_merged', 'mean_longitude_merged', 'max_wind_merged', 'range_wind_merged', 'min_pressure_merged', 'range_pressure_merged', 'max_eye_dia_merged', 'track_duration', 'end_datetime', 'start_datetime']

Query hurricane tracks into a Spatially enabled DataFrame¶

In [10]:

%%time
all_hurricanes_df = hurricane_fl.query(out_fields=','.join(fields_to_query), as_df=True)

%%time all_hurricanes_df = hurricane_fl.query(out_fields=','.join(fields_to_query), as_df=True)

CPU times: user 1.12 s, sys: 318 ms, total: 1.43 s
Wall time: 4.5 s

In [11]:

all_hurricanes_df.shape

all_hurricanes_df.shape

Out[11]:

(12362, 17)

There are 12,362 hurricanes identified by GeoAnalytics aggregate tracks tool. To get an idea about this aggregated dataset, call the head() method.

In [12]:

all_hurricanes_df.head()

all_hurricanes_df.head()

Out[12]:

	SHAPE	any_basin	any_name	any_sub_basin	count	end_datetime	max_eye_dia_merged	max_wind_merged	mean_latitude_merged	mean_longitude_merged	min_pressure_merged	min_season	objectid	range_pressure_merged	range_wind_merged	start_datetime	track_duration
0	{"paths": [[[59.60000000000002, -17.6000000000...	SI	NOT NAMED	MM	7.0	1854-02-10 18:00:00	NaN	NaN	-19.318571	60.639286	NaN	1854.0	1	NaN	NaN	1854-02-08 06:00:00	1.296000e+08
1	{"paths": [[[-23.5, 12.5], [-24.19999999999999...	NA	NOT NAMED	NA	9.0	1859-08-26 12:00:00	NaN	45.0	14.000000	-26.222222	NaN	1859.0	2	NaN	10.0	1859-08-24 12:00:00	1.728000e+08
2	{"paths": [[[-23.19999999999999, 12.1000000000...	NA	UNNAMED	NA	50.0	1853-09-12 18:00:00	NaN	130.0	26.982000	-51.776000	924.0	1853.0	3	53.0	90.0	1853-08-30 00:00:00	1.058400e+09
3	{"paths": [[[59.80000000000001, -15.5], [59.49...	SI	XXXX856017	MM	13.0	1856-04-05 18:00:00	NaN	NaN	-20.185385	59.573077	NaN	1856.0	4	NaN	NaN	1856-04-02 18:00:00	2.592000e+08
4	{"paths": [[[99.60000000000002, -11.5], [98.30...	SI	NOT NAMED	WA	13.0	1861-03-15 18:00:00	NaN	NaN	-12.940769	94.183846	NaN	1861.0	5	NaN	NaN	1861-03-12 18:00:00	2.592000e+08

To better analyze this data set, the date columns need to be changed to a format that Pandas understands better. This is accomplished by calling to_datetime() method and passing the appropriate time columns.

In [14]:

all_hurricanes_df['start_datetime'] = pd.to_datetime(all_hurricanes_df['start_datetime'])
all_hurricanes_df['end_datetime'] = pd.to_datetime(all_hurricanes_df['end_datetime'])
all_hurricanes_df.index = all_hurricanes_df['start_datetime']
all_hurricanes_df.head()

all_hurricanes_df['start_datetime'] = pd.to_datetime(all_hurricanes_df['start_datetime']) all_hurricanes_df['end_datetime'] = pd.to_datetime(all_hurricanes_df['end_datetime']) all_hurricanes_df.index = all_hurricanes_df['start_datetime'] all_hurricanes_df.head()

Out[14]:

	SHAPE	any_basin	any_name	any_sub_basin	count	end_datetime	max_eye_dia_merged	max_wind_merged	mean_latitude_merged	mean_longitude_merged	min_pressure_merged	min_season	objectid	range_pressure_merged	range_wind_merged	start_datetime	track_duration
start_datetime
1854-02-08 06:00:00	{"paths": [[[59.60000000000002, -17.6000000000...	SI	NOT NAMED	MM	7.0	1854-02-10 18:00:00	NaN	NaN	-19.318571	60.639286	NaN	1854.0	1	NaN	NaN	1854-02-08 06:00:00	1.296000e+08
1859-08-24 12:00:00	{"paths": [[[-23.5, 12.5], [-24.19999999999999...	NA	NOT NAMED	NA	9.0	1859-08-26 12:00:00	NaN	45.0	14.000000	-26.222222	NaN	1859.0	2	NaN	10.0	1859-08-24 12:00:00	1.728000e+08
1853-08-30 00:00:00	{"paths": [[[-23.19999999999999, 12.1000000000...	NA	UNNAMED	NA	50.0	1853-09-12 18:00:00	NaN	130.0	26.982000	-51.776000	924.0	1853.0	3	53.0	90.0	1853-08-30 00:00:00	1.058400e+09
1856-04-02 18:00:00	{"paths": [[[59.80000000000001, -15.5], [59.49...	SI	XXXX856017	MM	13.0	1856-04-05 18:00:00	NaN	NaN	-20.185385	59.573077	NaN	1856.0	4	NaN	NaN	1856-04-02 18:00:00	2.592000e+08
1861-03-12 18:00:00	{"paths": [[[99.60000000000002, -11.5], [98.30...	SI	NOT NAMED	WA	13.0	1861-03-15 18:00:00	NaN	NaN	-12.940769	94.183846	NaN	1861.0	5	NaN	NaN	1861-03-12 18:00:00	2.592000e+08

The track duration and length columns need to be projected to units (days, hours, miles) that are meaningful for analysis.

In [15]:

all_hurricanes_df['track_duration_hrs'] = all_hurricanes_df['track_duration'] / 3600000
all_hurricanes_df['track_duration_days'] = all_hurricanes_df['track_duration'] / (3600000*24)

all_hurricanes_df['track_duration_hrs'] = all_hurricanes_df['track_duration'] / 3600000 all_hurricanes_df['track_duration_days'] = all_hurricanes_df['track_duration'] / (3600000*24)

Exploratory data analysis¶

In this section we perform exploratory analysis of the dataset and answer some interesting questions.

In [21]:

map1 = gis.map('USA')
map1

map1 = gis.map('USA') map1

Out[21]:

In [17]:

all_hurricanes_df.sample(n=500, random_state=2).spatial.plot(map1, 
                                                             renderer_type='u',
                                                             col='any_basin',
                                                            cmap='prism')

all_hurricanes_df.sample(n=500, random_state=2).spatial.plot(map1, renderer_type='u', col='any_basin', cmap='prism')

Out[17]:

True

The map above draws a set of 500 hurricanes chosen at random. You can visualize the Spatially Enabled DataFrame object with different types of renderers. In the example above a unique value renderer is applied on the basin column. You can switch the map to 3D mode and view the same on a globe.

In [22]:

map2 = gis.map()
map2.mode= '3D'
map2

map2 = gis.map() map2.mode= '3D' map2

Out[22]: