vector_geoanalytics

Creating hurricane tracks using Geoanalytics

The sample code below uses big data analytics (GeoAnalytics) to reconstruct hurricane tracks using data registered on a big data file share in the GIS. Note that this functionality is currently available on ArcGIS Enterprise 10.5 and not yet with ArcGIS Online.

Reconstruct tracks

Reconstruct tracks is a type of data aggregation tool available in the arcgis.geoanalytics module. This tool works with a layer of point features or polygon features that are time enabled. It first determines which points belong to a track using an identification number or identification string. Using the time at each location, the tracks are ordered sequentially and transformed into a line representing the path of movement.

Data used

For this sample, hurricane data from over a period of 50 years, totalling about 150,000 points split into 5 shape files was used. The National Hurricane Center provides similar datasets that can be used for exploratory purposes.

To illustrate the nature of the data a subset was published as a feature service and can be visualized as below:

In [1]:
from arcgis.gis import GIS

# Create an anonymous connection to ArcGIS Online
arcgis_online = GIS()
hurricane_pts = arcgis_online.content.search("Hurricane_tracks_points AND owner:atma.mani", "Feature Layer")[0]
hurricane_pts
Out[1]:
Hurricane_tracks_points
Years 1932 - 1942Feature Layer Collection by atma.mani
Last Modified: September 15, 2016
0 comments, 0 views
In [2]:
subset_map = arcgis_online.map("USA")
subset_map
In [3]:
subset_map.add_layer(hurricane_pts)

Inspect the data attributes

Let us query the first layer in hurricane_pts and view its attribute table as a Pandas dataframe.

In [4]:
hurricane_pts.layers[0].query().df.head()
Out[4]:
ATC_eye ATC_grade ATC_poci ATC_pres ATC_rmw ATC_roci ATC_w34_r1 ATC_w34_r2 ATC_w34_r3 ATC_w34_r4 day hour min_ month wmo_pres wmo_pres__ wmo_wind wmo_wind__ year SHAPE
0 -999 -999. -999 -999 -999 -999 -999 -999 -999 -999 1 0 0 1 -999 -999.0 -999 -999.0 1932 {‘spatialReference’: {‘wkid’: 4326, ‘latestWki…
1 -999 -999. -999 -999 -999 -999 -999 -999 -999 -999 1 6 0 1 0 -100.0 0 -100.0 1932 {‘spatialReference’: {‘wkid’: 4326, ‘latestWki…
2 -999 -999. -999 -999 -999 -999 -999 -999 -999 -999 1 12 0 1 -999 -999.0 -999 -999.0 1932 {‘spatialReference’: {‘wkid’: 4326, ‘latestWki…
3 -999 -999. -999 -999 -999 -999 -999 -999 -999 -999 1 18 0 1 -999 -999.0 -999 -999.0 1932 {‘spatialReference’: {‘wkid’: 4326, ‘latestWki…
4 -999 -999. -999 -999 -999 -999 -999 -999 -999 -999 2 0 0 1 -999 -999.0 -999 -999.0 1932 {‘spatialReference’: {‘wkid’: 4326, ‘latestWki…

5 rows × 148 columns

Create a data store

For the GeoAnalytics server to process your big data, it needs the data to be registered as a data store. In our case, the data is in multiple shape files and we will register the folder containing the files as a data store of type bigDataFileShare.

Let us connect to an ArcGIS Enterprise

In [5]:
from getpass import getpass
password = getpass()
gis = GIS("https://esriwebgis.webgistesting.net/portal", "admin",password)
········

Get the geoanalytics datastores and search it for the registered datasets:

In [6]:
# Query the data stores available
from arcgis import geoanalytics
datastores = geoanalytics.get_datastores()
bigdata_fileshares = datastores.search()
bigdata_fileshares
Out[6]:
[<Datastore title:"/bigDataFileShares/Hurricane_tracks" type:"bigDataFileShare">,
 <Datastore title:"/bigDataFileShares/hurricanes" type:"bigDataFileShare">,
 <Datastore title:"/bigDataFileShares/test" type:"bigDataFileShare">]
In [9]:
data_store_item = bigdata_fileshares[1]
In [10]:
len(data_store_item.datasets)
Out[10]:
2
In [11]:
data_store_item.datasets[1]
Out[11]:
{'format': {'extension': 'shp', 'type': 'shapefile'},
 'geometry': {'geometryType': 'esriGeometryPoint',
  'spatialReference': {'wkid': 4326}},
 'name': 'full_dataset',
 'schema': {'fields': [{'name': 'serial_num', 'type': 'esriFieldTypeString'},
   {'name': 'season', 'type': 'esriFieldTypeBigInteger'},
   {'name': 'num', 'type': 'esriFieldTypeBigInteger'},
   {'name': 'basin', 'type': 'esriFieldTypeString'},
   {'name': 'sub_basin', 'type': 'esriFieldTypeString'},
   {'name': 'name', 'type': 'esriFieldTypeString'},
   {'name': 'iso_time', 'type': 'esriFieldTypeString'},
   {'name': 'nature', 'type': 'esriFieldTypeString'},
   {'name': 'latitude', 'type': 'esriFieldTypeDouble'},
   {'name': 'longitude', 'type': 'esriFieldTypeDouble'},
   {'name': 'wind_wmo_', 'type': 'esriFieldTypeDouble'},
   {'name': 'pres_wmo_', 'type': 'esriFieldTypeBigInteger'},
   {'name': 'center', 'type': 'esriFieldTypeString'},
   {'name': 'wind_wmo1', 'type': 'esriFieldTypeDouble'},
   {'name': 'pres_wmo1', 'type': 'esriFieldTypeDouble'},
   {'name': 'track_type', 'type': 'esriFieldTypeString'},
   {'name': 'size', 'type': 'esriFieldTypeString'},
   {'name': 'Wind', 'type': 'esriFieldTypeBigInteger'}]},
 'time': {'fields': [{'formats': ['yyyy-MM-dd HH:mm:ss'], 'name': 'iso_time'}],
  'timeReference': {'timeZone': 'UTC'},
  'timeType': 'instant'}}

If there is no big data file share for hurricane track data registered on the server, we can register one that points to the shared folder containing the shape files.

In [ ]:
data_item = datastores.add_bigdata("Hurricane_tracks", r"\\path_to_hurricane_data")
In [12]:
search_result = gis.content.search('', item_type = 'Big Data File Share')
search_result
Out[12]:
[<Item title:"bigDataFileShares_hurricanes" type:Big Data File Share owner:atma.mani>]
In [13]:
hurricanes_data_item = search_result[0]
hurricanes_data_item
Out[13]:
bigDataFileShares_hurricanes
Big Data File Share by atma.mani
Last Modified: March 06, 2017
0 comments, 0 views

Perform data aggregation using reconstruct tracks tool

When you add a big data file share, a corresponding item gets created in your GIS. You can search for it like a regular item and query its layers.

In [14]:
hurricanes_data_item.layers
Out[14]:
[<Layer url:"https://ESRIgeoanalytics.webgistesting.net/server/rest/services/DataStoreCatalogs/bigDataFileShares_hurricanes/BigDataCatalogServer/split_by_year">,
 <Layer url:"https://ESRIgeoanalytics.webgistesting.net/server/rest/services/DataStoreCatalogs/bigDataFileShares_hurricanes/BigDataCatalogServer/full_dataset">]
In [15]:
years_50 = hurricanes_data_item.layers[1]
years_50
Out[15]:
<Layer url:"https://ESRIgeoanalytics.webgistesting.net/server/rest/services/DataStoreCatalogs/bigDataFileShares_hurricanes/BigDataCatalogServer/full_dataset">

Reconstruct tracks tool

The reconstruct_tracks() function is available in the arcgis.geoanalytics.summarize_data module. In this example, we are using this tool to aggregate the numerous points into line segments showing the tracks followed by the hurricanes. The tool creates a feature layer item as an output which can be accessed once the processing is complete.

In [16]:
agg_result = geoanalytics.summarize_data.reconstruct_tracks(input_layer=years_50,
                                                            track_fields='Serial_Num',
                                                            method='GEODESIC')
Submitted.
Executing...
Executing (ReconstructTracks): ReconstructTracks "Feature Set" Serial_Num Geodesic # # # # {"itemProperties":{"itemId":"b8f8b055c70c44878b578e7c29079c79"},"serviceProperties":{"serviceUrl":"http://ESRIwebgis.webgistesting.net/server/rest/services/Hosted/Reconstructed_Tracks_WXXBKC/FeatureServer","name":"Reconstructed_Tracks_WXXBKC"}} #
Start Time: Mon Mar 06 23:33:20 2017
Using URL based GPRecordSet param: https://ESRIgeoanalytics.webgistesting.net/server/rest/services/DataStoreCatalogs/bigDataFileShares_hurricanes/BigDataCatalogServer/full_dataset?token=ucfxgdYPHkgWdsiVXtNV2ioEr1XKPnKqVtCmxgMwj7w0NY5p69bX_ZNp_LAad5ce6aMuhfBs1BkaNazsY8I7VJ3hKXDPGwzho28PkmZWzAOgWSqcvgP9JZZxp9P9oQdmjbxboZfbZQZG5dkzrN-wYg..
{"messageCode":"BD_101028","message":"Starting new distributed job with 8 tasks.","params":{"totalTasks":"8"}}
{"messageCode":"BD_101029","message":"1/8 distributed tasks completed.","params":{"completedTasks":"1","totalTasks":"8"}}
{"messageCode":"BD_101029","message":"7/8 distributed tasks completed.","params":{"completedTasks":"7","totalTasks":"8"}}
{"messageCode":"BD_101051","message":"Possible issues were found while reading 'inputLayer'.","params":{"paramName":"inputLayer"}}
{"messageCode":"BD_101052","message":"Some records have either missing or invalid time values."}
  extent = Some(Envelope: [-103.0, -39.8, 80.0, 63.0])
  interval = Some(Interval(MutableInstant(1848-01-11 06:00:00.000),MutableInstant(1899-12-26 06:00:00.000)))
  count = 568
{"messageCode":"BD_0","message":"Feature service layer created: http://ESRIwebgis.webgistesting.net/server/rest/services/Hosted/Reconstructed_Tracks_WXXBKC/FeatureServer/0","params":{"serviceUrl":"http://ESRIwebgis.webgistesting.net/server/rest/services/Hosted/Reconstructed_Tracks_WXXBKC/FeatureServer/0"}}
{"messageCode":"BD_101051","message":"Possible issues were found while reading 'inputLayer'.","params":{"paramName":"inputLayer"}}
{"messageCode":"BD_101052","message":"Some records have either missing or invalid time values."}
      > GenericFeature(attributes=[1900032S20050,1900,7,SI,MM,XXXX900274,2/1/1900 6:00,NR,-19.9,50.2,0.0,0,reunion,-100.0,-100.0,split,0,0],geometry={"x":50.2,"y":-19.9},time=null)
      > GenericFeature(attributes=[1900032S20050,1900,7,SI,MM,XXXX900274,2/2/1900 6:00,NR,-20.8,51.1,0.0,0,reunion,-100.0,-100.0,split,0,0],geometry={"x":51.1,"y":-20.8},time=null)
      > GenericFeature(attributes=[1900032S20050,1900,7,SI,MM,XXXX900274,2/3/1900 6:00,NR,-23.2,53.3,0.0,0,reunion,-100.0,-100.0,split,0,0],geometry={"x":53.3,"y":-23.2},time=null)
Succeeded at Mon Mar 06 23:33:36 2017 (Elapsed Time: 15.32 seconds)

Inspect the results

Let us create a map and load the processed result which is a feature service

In [ ]:
processed_map = gis.map("USA")
processed_map
In [ ]:
processed_map.add_layer(agg_result)

Thus we transformed a bunch of ponints into tracks that represents paths taken by the hurricanes over a period of 50 years. We can pull up another map and inspect the results a bit more closely

Our input data and the map widget is time enabled. Thus we can filter the data to represent the tracks from only the years 1860 to 1870

In [ ]:
processed_map.set_time_extent('1860', '1870')

What can geoanalytics do for you?

With this sample we just scratched the surface of what big data analysis can do for you. ArcGIS Enterprise at 10.5 packs a powerful set of tools that let you derive a lot of value from your data. You can do so by asking the right questions, for instance, a weather dataset such as this could be used to answer a few interesting questions such as

  • did the number of hurricanes per season increase over the years?
  • give me the hurricanes that travelled longest distance
  • give me the ones that stayed for longest time. Do we see a trend?
  • how are wind speed and distance travelled correlated?
  • my assets are located in a tornado corridor. How many times in the past century, was there a hurricane within 50 miles from my assets?
  • my industry is dependent on tourism, which is heavily impacted by the vagaries of weather. From historical weather data, can I correlate my profits with major weather events? How well is my business insulated from freak weather events?
  • over the years do we see any shifts in major weather events - do we notice a shift in when the hurricane season starts?

The ArcGIS API for Python gives you a gateway to easily access the big data tools from your ArcGIS Enterprise. By combining it with other powerful libraries from the pandas and scipy stack and the rich visualization capabilities of the Jupyter notebook, you can extract a lot of value from your data, big or small.