Reading multidimensional data using open geo tools

Reading multidimensional science data in NetCDF using open geo toos

This notebook reads aerosol index and Tropospheric $NO_{2}$ Concentration from Sentinel-5P TROPOMI data using Opengeo Tools

In [1]:
import numpy as np
# from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as plt
import sys
from netCDF4 import Dataset
from pprint import pprint, pp
import pandas as pd
In [2]:
ls TROPOMI_PythonCodesAndData/
S5P_OFFL_L2__AER_AI_20180816T183016_20180816T201146_04361_01_010100_20180822T174822.nc*
S5P_OFFL_L2__CO_____20180816T183016_20180816T201146_04361_01_010100_20180822T174815.nc*
S5P_RPRO_L2__CH4____20180816T182917_20180816T201245_04361_01_010202_20190101T194705.nc*
fileList.txt*
read_and_map_tropomi_no2_ai.py*
read_tropomi_and_list_sds.py*
read_tropomi_no2_ai_and_dump_ascii.py*
read_tropomi_no2_ai_at_a_location.py*

Explore NetCDF file for its contents

The NetCDF file is like a folder with multiple sub-folders and files within it. Folders are called as groups and files within it are called as variables. NASA supplies a cross-platform app called Panoply which gives you a UI to query and visualize NetCDF files. Below is a screenshot of Panoply reading the Aerosol Index file.

Screenshot of Panoply software with Aerosol Index NetCDF file opened.

The first step is to read this file as a netCDF4.Dataset class.

In [3]:
file_path = 'TROPOMI_PythonCodesAndData/S5P_OFFL_L2__AER_AI_20180816T183016_20180816T201146_04361_01_010100_20180822T174822.nc'
ds = Dataset(file_path, mode='r')
type(ds)
Out[3]:
netCDF4._netCDF4.Dataset
In [4]:
ds.groups.keys()
Out[4]:
dict_keys(['PRODUCT', 'METADATA'])

Explore the different variables as a DataFrame

In [5]:
v = {'variables':[], 'long_name':[], 'units':[]}
for var in list(ds.groups['PRODUCT'].variables.keys()):
    v['variables'].append(ds.groups['PRODUCT'].variables[var].name)
    v['long_name'].append(ds.groups['PRODUCT'].variables[var].long_name)
    try:
        v['units'].append(ds.groups['PRODUCT'].variables[var].units)
    except:
        v['units'].append(None)

vars_df = pd.DataFrame.from_dict(v)
vars_df
Out[5]:
variables long_name units
0 scanline along-track dimension index 1
1 ground_pixel across-track dimension index 1
2 time reference time for the measurements seconds since 2010-01-01 00:00:00
3 corner pixel corner index 1
4 latitude pixel center latitude degrees_north
5 longitude pixel center longitude degrees_east
6 delta_time offset from reference start time of measurement milliseconds
7 time_utc Time of observation as ISO 8601 date-time string None
8 qa_value data quality value 1
9 aerosol_index_354_388 Aerosol index from 388 and 354 nm 1
10 aerosol_index_340_380 Aerosol index from 380 and 340 nm 1
11 aerosol_index_354_388_precision Precision of aerosol index from 388 and 354 nm 1
12 aerosol_index_340_380_precision Precision of aerosol index from 380 and 340 nm 1

Read Aerosol Index 354 - 388 nm into memory

In [6]:
# preview contents of the variable
ds.groups['PRODUCT'].variables['aerosol_index_354_388']
Out[6]:
<class 'netCDF4._netCDF4.Variable'>
float32 aerosol_index_354_388(time, scanline, ground_pixel)
    units: 1
    proposed_standard_name: ultraviolet_aerosol_index
    comment: Aerosol index from 388 and 354 nm
    long_name: Aerosol index from 388 and 354 nm
    radiation_wavelength: [354. 388.]
    coordinates: longitude latitude
    ancillary_variables: aerosol_index_354_388_precision
    _FillValue: 9.96921e+36
path = /PRODUCT
unlimited dimensions: 
current shape = (1, 3245, 450)
filling on
In [7]:
ai_data = ds.groups['PRODUCT'].variables['aerosol_index_354_388'][:]
type(ai_data)
Out[7]:
numpy.ma.core.MaskedArray
In [8]:
ai_data.shape
Out[8]:
(1, 3245, 450)
In [9]:
plt.imshow(ai_data[0]);
In [28]:
import xarray
In [31]:
xr_data = xarray.open_dataset(file_path, group='PRODUCT', 
                              engine='netcdf4', decode_coords=True)
type(xr_data)
Out[31]:
xarray.core.dataset.Dataset
In [32]:
xr_data
Out[32]:
<xarray.Dataset>
Dimensions:                          (corner: 4, ground_pixel: 450, scanline: 3245, time: 1)
Coordinates:
  * scanline                         (scanline) float64 0.0 1.0 ... 3.244e+03
  * ground_pixel                     (ground_pixel) float64 0.0 1.0 ... 449.0
  * time                             (time) datetime64[ns] 2018-08-16
  * corner                           (corner) float64 0.0 1.0 2.0 3.0
    latitude                         (time, scanline, ground_pixel) float32 ...
    longitude                        (time, scanline, ground_pixel) float32 ...
Data variables:
    delta_time                       (time, scanline) timedelta64[ns] 18:51:5...
    time_utc                         (time, scanline) object '2018-08-16T18:5...
    qa_value                         (time, scanline, ground_pixel) float32 ...
    aerosol_index_354_388            (time, scanline, ground_pixel) float32 ...
    aerosol_index_340_380            (time, scanline, ground_pixel) float32 ...
    aerosol_index_354_388_precision  (time, scanline, ground_pixel) float32 ...
    aerosol_index_340_380_precision  (time, scanline, ground_pixel) float32 ...
In [33]:
xr_data_ai = xr_data['aerosol_index_340_380']
print(type(xr_data_ai))
print(xr_data_ai.shape)
<class 'xarray.core.dataarray.DataArray'>
(1, 3245, 450)
In [34]:
xr_data_ai[0].plot();
In [37]:
(xr_data.latitude.attrs, xr_data.longitude.attrs)
Out[37]:
({'long_name': 'pixel center latitude',
  'units': 'degrees_north',
  'standard_name': 'latitude',
  'valid_min': -90.0,
  'valid_max': 90.0,
  'bounds': '/PRODUCT/SUPPORT_DATA/GEOLOCATIONS/latitude_bounds'},
 {'long_name': 'pixel center longitude',
  'units': 'degrees_east',
  'standard_name': 'longitude',
  'valid_min': -180.0,
  'valid_max': 180.0,
  'bounds': '/PRODUCT/SUPPORT_DATA/GEOLOCATIONS/longitude_bounds'})
Plot using matplotlib
In [42]:
plt.figure(figsize=(14,8))
ax = plt.axes()

xr_data.aerosol_index_340_380[0].plot.pcolormesh(ax=ax, x='longitude', 
                                                 y='latitude',
                                                 add_colorbar=True, cmap='jet');
Plotting using cartopy
In [44]:
import cartopy.crs as ccrs
plt.figure(figsize=(14,6))
ax = plt.axes(projection = ccrs.PlateCarree())

xr_data.aerosol_index_340_380[0].plot.pcolormesh(ax=ax, x='longitude', 
                                                 y='latitude',
                                                 add_colorbar=True, cmap='jet')

ax.set_global()
ax.coastlines();
In [47]:
plt.figure(figsize=(10,10))
ax = plt.axes(projection = ccrs.Orthographic(-88,40))

xr_data.aerosol_index_340_380[0].plot.pcolormesh(ax=ax, x='longitude', 
                                                 y='latitude',
                                                 add_colorbar=True, cmap='jet',
                                                transform=ccrs.PlateCarree())

ax.set_global()
ax.coastlines();
In [51]:
xr_data_rio = xr_data_ai.rio
type(xr_data_rio)
Out[51]:
rioxarray.rioxarray.RasterArray
In [55]:
xr_data.aerosol_index_340_380.rio.to_raster('xr_test.tif')

Reading using rioxarray

In [60]:
import rioxarray
import warnings; warnings.simplefilter('ignore')
In [85]:
rds = rioxarray.open_rasterio(filename = file_path, parse_coordinates=True,
                             )
type(rds)
Out[85]:
list
In [87]:
rds[0]
Out[87]:
<xarray.Dataset>
Dimensions:                                   (band: 1, time: 1, x: 450, y: 3245)
Coordinates:
  * y                                         (y) float64 3.244e+03 ... 0.0
  * x                                         (x) float64 0.0 1.0 ... 449.0
  * time                                      (time) int64 272073600
    spatial_ref                               int64 0
  * band                                      (band) int64 1
Data variables:
    latitude                                  (time, y, x) float32 ...
    solar_zenith_angle                        (band, y, x) float32 ...
    solar_azimuth_angle                       (band, y, x) float32 ...
    viewing_zenith_angle                      (band, y, x) float32 ...
    viewing_azimuth_angle                     (band, y, x) float32 ...
    longitude                                 (time, y, x) float32 ...
    geolocation_flags                         (band, y, x) uint8 ...
    processing_quality_flags                  (band, y, x) uint32 ...
    number_of_spectral_points_in_retrieval    (band, y, x) uint16 ...
    scene_albedo_388                          (band, y, x) float32 ...
    scene_albedo_388_precision                (band, y, x) float32 ...
    reflectance_measured_354                  (band, y, x) float32 ...
    reflectance_measured_354_precision        (band, y, x) float32 ...
    reflectance_measured_388                  (band, y, x) float32 ...
    reflectance_measured_388_precision        (band, y, x) float32 ...
    reflectance_calculated_354                (band, y, x) float32 ...
    reflectance_calculated_354_precision      (band, y, x) float32 ...
    scene_albedo_380                          (band, y, x) float32 ...
    scene_albedo_380_precision                (band, y, x) float32 ...
    reflectance_measured_340                  (band, y, x) float32 ...
    reflectance_measured_340_precision        (band, y, x) float32 ...
    reflectance_measured_380                  (band, y, x) float32 ...
    reflectance_measured_380_precision        (band, y, x) float32 ...
    reflectance_calculated_340                (band, y, x) float32 ...
    reflectance_calculated_340_precision      (band, y, x) float32 ...
    wavelength_calibration_offset             (band, y, x) float32 ...
    wavelength_calibration_offset_precision   (band, y, x) float32 ...
    wavelength_calibration_stretch            (band, y, x) float32 ...
    wavelength_calibration_stretch_precision  (band, y, x) float32 ...
    wavelength_calibration_chi_square         (band, y, x) float32 ...
    surface_altitude                          (band, y, x) float32 ...
    surface_altitude_precision                (band, y, x) float32 ...
    surface_classification                    (band, y, x) uint8 ...
    qa_value                                  (time, y, x) uint8 ...
    scaled_small_pixel_variance               (band, y, x) float32 ...
    ozone_total_column                        (band, y, x) float32 ...
    surface_pressure                          (band, y, x) float32 ...
    aerosol_index_354_388                     (time, y, x) float32 ...
    aerosol_index_340_380                     (time, y, x) float32 ...
    aerosol_index_354_388_precision           (time, y, x) float32 ...
    aerosol_index_340_380_precision           (time, y, x) float32 ...
Attributes:
    algo.algorithm_variant:                                            1
    algo.n_pair:                                                       2
    algo.pair_1.delta_wavelength:                                      2.0
    algo.pair_1.id:                                                    TOMS_pair
    algo.pair_1.min_wavelength:                                        1
    algo.pair_1.number_spectral_pixels:                                7
    algo.pair_1.wavelength_1:                                          340
    algo.pair_1.wavelength_2:                                          380
    algo.pair_2.delta_wavelength:                                      2.0
    algo.pair_2.id:                                                    OMI_pair
    algo.pair_2.min_wavelength:                                        1
    algo.pair_2.number_spectral_pixels:                                7
    algo.pair_2.wavelength_1:                                          354
    algo.pair_2.wavelength_2:                                          388
    configuration.version.algorithm:                                   1.1.0
    configuration.version.framework:                                   1.1.0
    input.1.band:                                                      3
    input.1.irrType:                                                   L1B_IR...
    input.1.type:                                                      L1B_RA...
    input.count:                                                       1
    output.1.band:                                                     3
    output.1.config:                                                   cfg/pr...
    output.1.type:                                                     L2__AE...
    output.compressionLevel:                                           3
    output.count:                                                      1
    output.histogram.aerosol_index_340_380.end:                        14
    output.histogram.aerosol_index_340_380.start:                      -6
    output.histogram.aerosol_index_354_388.end:                        14
    output.histogram.aerosol_index_354_388.start:                      -6
    output.useCompression:                                             true
    output.useFletcher32:                                              true
    output.useShuffleFilter:                                           true
    processing.algorithm:                                              AER_AI
    processing.correct_surface_pressure_for_altitude:                  true
    processing.exclude_flags:                                          429496...
    processing.groupDem:                                               DEM_RA...
    processing.ignore_pixel_flags:                                     False
    processing.radiancePixelsMinError:                                 2
    processing.radiancePixelsMinWarning:                               7
    processing.signal_to_noise.test:                                   yes
    processing.signal_to_noise.threshold:                              12
    processing.signal_to_noise.window.range:                           350.0,...
    processing.szaMax:                                                 88.0
    processing.szaMin:                                                 0.0
    processing.vzaMax:                                                 78.0
    processing.vzaMin:                                                 0.0
    qa_value.AAI_warning:                                              100.0
    qa_value.altitude_consistency_warning:                             100.0
    qa_value.cloud_warning:                                            100.0
    qa_value.data_range_warning:                                       100.0
    qa_value.deconvolution_warning:                                    100.0
    qa_value.extrapolation_warning:                                    100.0
    qa_value.input_spectrum_warning:                                   70.0
    qa_value.interpolation_warning:                                    100.0
    qa_value.low_cloud_fraction_warning:                               100.0
    qa_value.pixel_level_input_data_missing:                           80.0
    qa_value.signal_to_noise_ratio_warning:                            100.0
    qa_value.snow_ice_warning:                                         100.0
    qa_value.so2_volcanic_origin_certain_warning:                      100.0
    qa_value.so2_volcanic_origin_likely_warning:                       100.0
    qa_value.south_atlantic_anomaly_warning:                           100.0
    qa_value.sun_glint_correction:                                     100.0
    qa_value.sun_glint_warning:                                        70.0
    qa_value.wavelength_calibration_warning:                           90.0
    wavelength_calibration.convergence_threshold:                      1.0
    wavelength_calibration.include_stretch:                            no
    wavelength_calibration.initial_guess.a0:                           1.0
    wavelength_calibration.initial_guess.a1:                           0.1
    wavelength_calibration.initial_guess.a2:                           0.01
    wavelength_calibration.initial_guess.ring:                         0.06
    wavelength_calibration.initial_guess.shift:                        0.0
    wavelength_calibration.initial_guess.stretch:                      0.0
    wavelength_calibration.irr.include_ring:                           no
    wavelength_calibration.irr.max_iterations:                         20
    wavelength_calibration.irr.polynomial_order:                       2
    wavelength_calibration.max_iterations:                             12
    wavelength_calibration.perform_wavelength_fit:                     yes
    wavelength_calibration.rad.include_ring:                           yes
    wavelength_calibration.rad.polynomial_order:                       3
    wavelength_calibration.sigma.a0:                                   1.0
    wavelength_calibration.sigma.a1:                                   0.1
    wavelength_calibration.sigma.ring:                                 0.06
    wavelength_calibration.sigma.shift:                                0.07
    wavelength_calibration.sigma.stretch:                              0.07
    wavelength_calibration.window:                                     338.0,...
    /METADATA/EOP_METADATA/eop:                                        metaDa...
    gml:                                                               id=S5P...
    objectType:                                                        gmi:MI...
    /METADATA/EOP_METADATA/om:                                         proced...
    File_Class:                                                        OFFL
    File_Description:                                                  Aeroso...
    File_Name:                                                         S5P_OF...
    File_Type:                                                         L2__AE...
    File_Version:                                                      1
    Mission:                                                           S5P
    Notes:                                                             
    Creation_Date:                                                     UTC=20...
    Creator:                                                           TROPNL...
    Creator_Version:                                                   1.1.0
    System:                                                            PDGS-OP
    Validity_Start:                                                    UTC=20...
    Validity_Stop:                                                     UTC=20...
    /METADATA/ESA_METADATA/earth_explorer_header/variable_header/gmd:  lineag...
    GranuleEnd:                                                        2018-0...
    GranuleStart:                                                      2018-0...
    InstrumentName:                                                    TROPOMI
    LongitudeOfDaysideNadirEquatorCrossing:                            -86.44...
    MissionName:                                                       Sentin...
    MissionShortName:                                                  S5P
    ProcessingCenter:                                                  PDGS-OP
    ProcessingMode:                                                    Offline
    ProcessingNode:                                                    s5p-of...
    ProcessLevel:                                                      2
    ProcessorVersion:                                                  1.1.0
    ProductFormatVersion:                                              10000
    ProductShortName:                                                  L2__AE...
    /METADATA/ISO_METADATA/gmd:                                        langua...
    /METADATA/ISO_METADATA/gmi:                                        acquis...
    gmd:                                                               metada...
    global_processing_warnings:                                        --- TE...
    number_of_aai_filter_occurrences:                                  0
    number_of_aai_scene_albedo_filter_occurrences:                     0
    number_of_AAI_warning_occurrences:                                 0
    number_of_abort_error_occurrences:                                 0
    number_of_aerosol_boundary_error_occurrences:                      0
    number_of_airmass_factor_error_occurrences:                        0
    number_of_altitude_consistency_filter_occurrences:                 0
    number_of_altitude_consistency_warning_occurrences:                0
    number_of_altitude_roughness_filter_occurrences:                   0
    number_of_aot_lower_boundary_convergence_error_occurrences:        0
    number_of_assertion_error_occurrences:                             31
    number_of_boundary_hit_error_occurrences:                          0
    number_of_cf_viirs_nir_ifov_filter_occurrences:                    0
    number_of_cf_viirs_nir_ofova_filter_occurrences:                   0
    number_of_cf_viirs_nir_ofovb_filter_occurrences:                   0
    number_of_cf_viirs_nir_ofovc_filter_occurrences:                   0
    number_of_cf_viirs_swir_ifov_filter_occurrences:                   0
    number_of_cf_viirs_swir_ofova_filter_occurrences:                  0
    number_of_cf_viirs_swir_ofovb_filter_occurrences:                  0
    number_of_cf_viirs_swir_ofovc_filter_occurrences:                  0
    number_of_ch4_noscat_ratio_filter_occurrences:                     0
    number_of_ch4_noscat_ratio_std_filter_occurrences:                 0
    number_of_ch4_noscat_zero_error_occurrences:                       0
    number_of_chi2_error_occurrences:                                  0
    number_of_cirrus_reflectance_viirs_filter_occurrences:             0
    number_of_cloud_error_occurrences:                                 0
    number_of_cloud_filter_convergence_error_occurrences:              0
    number_of_cloud_filter_occurrences:                                0
    number_of_cloud_fraction_fresco_filter_occurrences:                0
    number_of_cloud_fraction_viirs_filter_occurrences:                 0
    number_of_cloud_pressure_spread_too_low_error_occurrences:         0
    number_of_cloud_too_low_level_error_occurrences:                   0
    number_of_cloud_warning_occurrences:                               0
    number_of_configuration_error_occurrences:                         0
    number_of_convergence_error_occurrences:                           0
    number_of_coregistration_error_occurrences:                        0
    number_of_data_range_warning_occurrences:                          0
    number_of_deconvolution_warning_occurrences:                       0
    number_of_dfs_error_occurrences:                                   0
    number_of_diff_psurf_fresco_ecmwf_filter_occurrences:              0
    number_of_diff_refl_cirrus_viirs_filter_occurrences:               0
    number_of_extrapolation_warning_occurrences:                       0
    number_of_failed_retrievals:                                       222068
    number_of_generic_exception_occurrences:                           0
    number_of_generic_range_error_occurrences:                         0
    number_of_geographic_region_filter_occurrences:                    0
    number_of_geolocation_error_occurrences:                           0
    number_of_groundpixels:                                            1460250
    number_of_ground_pixels_with_warnings:                             84516
    number_of_h2o_noscat_ratio_filter_occurrences:                     0
    number_of_h2o_noscat_ratio_std_filter_occurrences:                 0
    number_of_h2o_noscat_zero_error_occurrences:                       0
    number_of_initialization_error_occurrences:                        0
    number_of_input_spectrum_alignment_error_occurrences:              0
    number_of_input_spectrum_missing_occurrences:                      0
    number_of_input_spectrum_warning_occurrences:                      0
    number_of_interpolation_warning_occurrences:                       0
    number_of_io_error_occurrences:                                    0
    number_of_irradiance_missing_occurrences:                          0
    number_of_ISRF_error_occurrences:                                  0
    number_of_key_error_occurrences:                                   0
    number_of_ler_range_error_occurrences:                             0
    number_of_low_cloud_fraction_warning_occurrences:                  0
    number_of_lut_error_occurrences:                                   0
    number_of_lut_range_error_occurrences:                             0
    number_of_max_iteration_convergence_error_occurrences:             0
    number_of_max_optical_thickness_error_occurrences:                 0
    number_of_memory_error_occurrences:                                0
    number_of_mixed_surface_type_filter_occurrences:                   0
    number_of_model_error_occurrences:                                 0
    number_of_number_of_input_data_points_too_low_error_occurrences:   0
    number_of_numerical_error_occurrences:                             0
    number_of_ocean_filter_occurrences:                                0
    number_of_optimal_estimation_error_occurrences:                    0
    number_of_other_boundary_convergence_error_occurrences:            0
    number_of_ozone_range_error_occurrences:                           0
    number_of_pixel_level_input_data_missing_occurrences:              0
    number_of_pixel_or_scanline_index_filter_occurrences:              0
    number_of_processed_pixels:                                        1460250
    number_of_profile_error_occurrences:                               0
    number_of_psurf_fresco_stdv_filter_occurrences:                    0
    number_of_radiance_missing_occurrences:                            0
    number_of_radiative_transfer_error_occurrences:                    0
    number_of_reflectance_range_error_occurrences:                     0
    number_of_refl_cirrus_viirs_nir_filter_occurrences:                0
    number_of_refl_cirrus_viirs_swir_filter_occurrences:               0
    number_of_rejected_pixels_not_enough_spectrum:                     0
    number_of_saturation_error_occurrences:                            0
    number_of_saturation_warning_occurrences:                          0
    number_of_signal_to_noise_ratio_error_occurrences:                 0
    number_of_signal_to_noise_ratio_warning_occurrences:               0
    number_of_slant_column_density_error_occurrences:                  0
    number_of_small_pixel_radiance_std_filter_occurrences:             0
    number_of_snow_ice_filter_occurrences:                             0
    number_of_snow_ice_warning_occurrences:                            0
    number_of_snr_range_error_occurrences:                             0
    number_of_so2_volcanic_origin_certain_warning_occurrences:         0
    number_of_so2_volcanic_origin_likely_warning_occurrences:          0
    number_of_solar_eclipse_filter_occurrences:                        0
    number_of_south_atlantic_anomaly_warning_occurrences:              0
    number_of_successfully_processed_pixels:                           1238182
    number_of_sun_glint_correction_occurrences:                        0
    number_of_sun_glint_filter_occurrences:                            0
    number_of_sun_glint_warning_occurrences:                           84516
    number_of_svd_error_occurrences:                                   0
    number_of_sza_range_error_occurrences:                             222037
    number_of_time_range_filter_occurrences:                           0
    number_of_vertical_column_density_error_occurrences:               0
    number_of_vza_range_error_occurrences:                             0
    number_of_wavelength_calibration_error_occurrences:                0
    number_of_wavelength_calibration_warning_occurrences:              0
    number_of_wavelength_offset_error_occurrences:                     0
    number_of_wrong_input_type_error_occurrences:                      0
    time_for_algorithm_initialization:                                 23.927286
    time_for_processing:                                               212.96926
    time_per_pixel:                                                    0.0015...
    time_standard_deviation_per_pixel:                                 2.5657...
    algorithm_version:                                                 1.1.0
    build_date:                                                        2018-0...
    cdm_data_type:                                                     Swath
    Conventions:                                                       CF-1.7
    cpp_compiler_flags:                                                -g -O2...
    cpp_compiler_version:                                              g++ (G...
    creator_email:                                                     EOSupp...
    creator_name:                                                      The Se...
    creator_url:                                                       http:/...
    date_created:                                                      2018-0...
    f90_compiler_flags:                                                -gdwar...
    f90_compiler_version:                                              GNU Fo...
    geolocation_grid_from_band:                                        3
    geospatial_lat_max:                                                89.972939
    geospatial_lat_min:                                                -86.81926
    geospatial_lon_max:                                                -179.9...
    geospatial_lon_min:                                                179.99924
    history:                                                           2018-0...
    id:                                                                S5P_OF...
    identifier_product_doi:                                            10.527...
    identifier_product_doi_authority:                                  http:/...
    institution:                                                       KNMI
    keywords:                                                          0300 A...
    keywords_vocabulary:                                               AGU in...
    license:                                                           No con...
    naming_authority:                                                  nl.knmi
    orbit:                                                             4361
    platform:                                                          S5P
    processing_status:                                                 Nominal
    processor_version:                                                 1.1.0
    product_version:                                                   1.0.0
    project:                                                           Sentin...
    references:                                                        http:/...
    revision_control_identifier:                                       df649d...
    sensor:                                                            TROPOMI
    source:                                                            Sentin...
    spatial_resolution:                                                7x3.5km2
    standard_name_vocabulary:                                          NetCDF...
    Status_MET_2D:                                                     Nominal
    summary:                                                           TROPOM...
    time_coverage_duration:                                            PT3503...
    time_coverage_end:                                                 2018-0...
    time_coverage_resolution:                                          PT1.080S
    time_coverage_start:                                               2018-0...
    time_reference:                                                    2018-0...
    time_reference_days_since_1950:                                    25064
    time_reference_julian_day:                                         2458346.5
    time_reference_seconds_since_1970:                                 153437...
    title:                                                             TROPOM...
    tracking_id:                                                       0b05f3...
In [97]:
r1 = rds[0]
type(r1)
Out[97]:
xarray.core.dataset.Dataset
In [99]:
r1.geolocation_flags
Out[99]:
<xarray.DataArray 'geolocation_flags' (band: 1, y: 3245, x: 450)>
array([[[12, 12, ..., 12, 12],
        [12, 12, ..., 12, 12],
        ...,
        [ 8,  8, ...,  8,  8],
        [ 8,  8, ...,  8,  8]]], dtype=uint8)
Coordinates:
  * y            (y) float64 3.244e+03 3.243e+03 3.242e+03 ... 2.0 1.0 0.0
  * x            (x) float64 0.0 1.0 2.0 3.0 4.0 ... 446.0 447.0 448.0 449.0
    spatial_ref  int64 0
  * band         (band) int64 1
Attributes:
    coordinates:      /PRODUCT/longitude /PRODUCT/latitude
    flag_masks:       [  0.   1.   2.   4.   8.  16. 128.]
    flag_meanings:    no_error solar_eclipse sun_glint_possible descending ni...
    flag_values:      [  0.   1.   2.   4.   8.  16. 128.]
    long_name:        ground pixel quality flag
    max_val:          254
    min_val:          0
    NETCDF_DIM_time:  1
    units:            1
    _FillValue:       255.0
    scale_factor:     1.0
    add_offset:       0.0
    grid_mapping:     spatial_ref
In [101]:
r1.spatial_ref
Out[101]:
<xarray.DataArray 'spatial_ref' ()>
array(0)
Coordinates:
    spatial_ref  int64 0
Attributes:
    GeoTransform:  -0.5 1.0 0.0 3244.5 0.0 -1.0
In [102]:
r1.spatial_resolution
Out[102]:
'7x3.5km2'
In [103]:
(r1.geospatial_lat_max, r1.geospatial_lat_min,
 r1.geospatial_lon_max, r1.geospatial_lon_min)
Out[103]:
(89.972939, -86.81926, -179.99773, 179.99924)
In [112]:
r1_ai = r1['aerosol_index_340_380']
r1_ai.plot()
Out[112]:
<matplotlib.collections.QuadMesh at 0x1965f1520>
In [113]:
r1_ai.spatial_ref
Out[113]:
<xarray.DataArray 'spatial_ref' ()>
array(0)
Coordinates:
    spatial_ref  int64 0
Attributes:
    GeoTransform:  -0.5 1.0 0.0 3244.5 0.0 -1.0
In [202]:
#  rds[0].rio.set_spatial_dims(x_dim='/PRODUCT/longitude', y_dim='/PRODUCT/latitude')
rds_crs_set = rds[0].rio.set_crs(4326)
rds_crs_set
Out[202]:
<xarray.Dataset>
Dimensions:                                   (band: 1, time: 1, x: 450, y: 3245)
Coordinates:
  * y                                         (y) float64 3.244e+03 ... 0.0
  * x                                         (x) float64 0.0 1.0 ... 449.0
  * time                                      (time) int64 272073600
    spatial_ref                               int64 0
  * band                                      (band) int64 1
Data variables:
    latitude                                  (time, y, x) float32 53.289627 ...
    solar_zenith_angle                        (band, y, x) float32 97.89976 ....
    solar_azimuth_angle                       (band, y, x) float32 54.418926 ...
    viewing_zenith_angle                      (band, y, x) float32 66.529594 ...
    viewing_azimuth_angle                     (band, y, x) float32 -60.276833...
    longitude                                 (time, y, x) float32 119.45596 ...
    geolocation_flags                         (band, y, x) uint8 12 12 ... 8 8
    processing_quality_flags                  (band, y, x) uint32 7 7 7 ... 7 7
    number_of_spectral_points_in_retrieval    (band, y, x) uint16 65535 ... 6...
    scene_albedo_388                          (band, y, x) float32 9.96921e+3...
    scene_albedo_388_precision                (band, y, x) float32 9.96921e+3...
    reflectance_measured_354                  (band, y, x) float32 9.96921e+3...
    reflectance_measured_354_precision        (band, y, x) float32 9.96921e+3...
    reflectance_measured_388                  (band, y, x) float32 9.96921e+3...
    reflectance_measured_388_precision        (band, y, x) float32 9.96921e+3...
    reflectance_calculated_354                (band, y, x) float32 9.96921e+3...
    reflectance_calculated_354_precision      (band, y, x) float32 9.96921e+3...
    scene_albedo_380                          (band, y, x) float32 9.96921e+3...
    scene_albedo_380_precision                (band, y, x) float32 9.96921e+3...
    reflectance_measured_340                  (band, y, x) float32 9.96921e+3...
    reflectance_measured_340_precision        (band, y, x) float32 9.96921e+3...
    reflectance_measured_380                  (band, y, x) float32 9.96921e+3...
    reflectance_measured_380_precision        (band, y, x) float32 9.96921e+3...
    reflectance_calculated_340                (band, y, x) float32 9.96921e+3...
    reflectance_calculated_340_precision      (band, y, x) float32 9.96921e+3...
    wavelength_calibration_offset             (band, y, x) float32 9.96921e+3...
    wavelength_calibration_offset_precision   (band, y, x) float32 9.96921e+3...
    wavelength_calibration_stretch            (band, y, x) float32 9.96921e+3...
    wavelength_calibration_stretch_precision  (band, y, x) float32 9.96921e+3...
    wavelength_calibration_chi_square         (band, y, x) float32 9.96921e+3...
    surface_altitude                          (band, y, x) float32 9.96921e+3...
    surface_altitude_precision                (band, y, x) float32 9.96921e+3...
    surface_classification                    (band, y, x) uint8 255 255 ... 255
    qa_value                                  (time, y, x) uint8 0 0 0 ... 0 0 0
    scaled_small_pixel_variance               (band, y, x) float32 9.96921e+3...
    ozone_total_column                        (band, y, x) float32 9.96921e+3...
    surface_pressure                          (band, y, x) float32 9.96921e+3...
    aerosol_index_354_388                     (time, y, x) float32 9.96921e+3...
    aerosol_index_340_380                     (time, y, x) float32 9.96921e+3...
    aerosol_index_354_388_precision           (time, y, x) float32 9.96921e+3...
    aerosol_index_340_380_precision           (time, y, x) float32 9.96921e+3...
Attributes:
    algo.algorithm_variant:                                            1
    algo.n_pair:                                                       2
    algo.pair_1.delta_wavelength:                                      2.0
    algo.pair_1.id:                                                    TOMS_pair
    algo.pair_1.min_wavelength:                                        1
    algo.pair_1.number_spectral_pixels:                                7
    algo.pair_1.wavelength_1:                                          340
    algo.pair_1.wavelength_2:                                          380
    algo.pair_2.delta_wavelength:                                      2.0
    algo.pair_2.id:                                                    OMI_pair
    algo.pair_2.min_wavelength:                                        1
    algo.pair_2.number_spectral_pixels:                                7
    algo.pair_2.wavelength_1:                                          354
    algo.pair_2.wavelength_2:                                          388
    configuration.version.algorithm:                                   1.1.0
    configuration.version.framework:                                   1.1.0
    input.1.band:                                                      3
    input.1.irrType:                                                   L1B_IR...
    input.1.type:                                                      L1B_RA...
    input.count:                                                       1
    output.1.band:                                                     3
    output.1.config:                                                   cfg/pr...
    output.1.type:                                                     L2__AE...
    output.compressionLevel:                                           3
    output.count:                                                      1
    output.histogram.aerosol_index_340_380.end:                        14
    output.histogram.aerosol_index_340_380.start:                      -6
    output.histogram.aerosol_index_354_388.end:                        14
    output.histogram.aerosol_index_354_388.start:                      -6
    output.useCompression:                                             true
    output.useFletcher32:                                              true
    output.useShuffleFilter:                                           true
    processing.algorithm:                                              AER_AI
    processing.correct_surface_pressure_for_altitude:                  true
    processing.exclude_flags:                                          429496...
    processing.groupDem:                                               DEM_RA...
    processing.ignore_pixel_flags:                                     False
    processing.radiancePixelsMinError:                                 2
    processing.radiancePixelsMinWarning:                               7
    processing.signal_to_noise.test:                                   yes
    processing.signal_to_noise.threshold:                              12
    processing.signal_to_noise.window.range:                           350.0,...
    processing.szaMax:                                                 88.0
    processing.szaMin:                                                 0.0
    processing.vzaMax:                                                 78.0
    processing.vzaMin:                                                 0.0
    qa_value.AAI_warning:                                              100.0
    qa_value.altitude_consistency_warning:                             100.0
    qa_value.cloud_warning:                                            100.0
    qa_value.data_range_warning:                                       100.0
    qa_value.deconvolution_warning:                                    100.0
    qa_value.extrapolation_warning:                                    100.0
    qa_value.input_spectrum_warning:                                   70.0
    qa_value.interpolation_warning:                                    100.0
    qa_value.low_cloud_fraction_warning:                               100.0
    qa_value.pixel_level_input_data_missing:                           80.0
    qa_value.signal_to_noise_ratio_warning:                            100.0
    qa_value.snow_ice_warning:                                         100.0
    qa_value.so2_volcanic_origin_certain_warning:                      100.0
    qa_value.so2_volcanic_origin_likely_warning:                       100.0
    qa_value.south_atlantic_anomaly_warning:                           100.0
    qa_value.sun_glint_correction:                                     100.0
    qa_value.sun_glint_warning:                                        70.0
    qa_value.wavelength_calibration_warning:                           90.0
    wavelength_calibration.convergence_threshold:                      1.0
    wavelength_calibration.include_stretch:                            no
    wavelength_calibration.initial_guess.a0:                           1.0
    wavelength_calibration.initial_guess.a1:                           0.1
    wavelength_calibration.initial_guess.a2:                           0.01
    wavelength_calibration.initial_guess.ring:                         0.06
    wavelength_calibration.initial_guess.shift:                        0.0
    wavelength_calibration.initial_guess.stretch:                      0.0
    wavelength_calibration.irr.include_ring:                           no
    wavelength_calibration.irr.max_iterations:                         20
    wavelength_calibration.irr.polynomial_order:                       2
    wavelength_calibration.max_iterations:                             12
    wavelength_calibration.perform_wavelength_fit:                     yes
    wavelength_calibration.rad.include_ring:                           yes
    wavelength_calibration.rad.polynomial_order:                       3
    wavelength_calibration.sigma.a0:                                   1.0
    wavelength_calibration.sigma.a1:                                   0.1
    wavelength_calibration.sigma.ring:                                 0.06
    wavelength_calibration.sigma.shift:                                0.07
    wavelength_calibration.sigma.stretch:                              0.07
    wavelength_calibration.window:                                     338.0,...
    /METADATA/EOP_METADATA/eop:                                        metaDa...
    gml:                                                               id=S5P...
    objectType:                                                        gmi:MI...
    /METADATA/EOP_METADATA/om:                                         proced...
    File_Class:                                                        OFFL
    File_Description:                                                  Aeroso...
    File_Name:                                                         S5P_OF...
    File_Type:                                                         L2__AE...
    File_Version:                                                      1
    Mission:                                                           S5P
    Notes:                                                             
    Creation_Date:                                                     UTC=20...
    Creator:                                                           TROPNL...
    Creator_Version:                                                   1.1.0
    System:                                                            PDGS-OP
    Validity_Start:                                                    UTC=20...
    Validity_Stop:                                                     UTC=20...
    /METADATA/ESA_METADATA/earth_explorer_header/variable_header/gmd:  lineag...
    GranuleEnd:                                                        2018-0...
    GranuleStart:                                                      2018-0...
    InstrumentName:                                                    TROPOMI
    LongitudeOfDaysideNadirEquatorCrossing:                            -86.44...
    MissionName:                                                       Sentin...
    MissionShortName:                                                  S5P
    ProcessingCenter:                                                  PDGS-OP
    ProcessingMode:                                                    Offline
    ProcessingNode:                                                    s5p-of...
    ProcessLevel:                                                      2
    ProcessorVersion:                                                  1.1.0
    ProductFormatVersion:                                              10000
    ProductShortName:                                                  L2__AE...
    /METADATA/ISO_METADATA/gmd:                                        langua...
    /METADATA/ISO_METADATA/gmi:                                        acquis...
    gmd:                                                               metada...
    global_processing_warnings:                                        --- TE...
    number_of_aai_filter_occurrences:                                  0
    number_of_aai_scene_albedo_filter_occurrences:                     0
    number_of_AAI_warning_occurrences:                                 0
    number_of_abort_error_occurrences:                                 0
    number_of_aerosol_boundary_error_occurrences:                      0
    number_of_airmass_factor_error_occurrences:                        0
    number_of_altitude_consistency_filter_occurrences:                 0
    number_of_altitude_consistency_warning_occurrences:                0
    number_of_altitude_roughness_filter_occurrences:                   0
    number_of_aot_lower_boundary_convergence_error_occurrences:        0
    number_of_assertion_error_occurrences:                             31
    number_of_boundary_hit_error_occurrences:                          0
    number_of_cf_viirs_nir_ifov_filter_occurrences:                    0
    number_of_cf_viirs_nir_ofova_filter_occurrences:                   0
    number_of_cf_viirs_nir_ofovb_filter_occurrences:                   0
    number_of_cf_viirs_nir_ofovc_filter_occurrences:                   0
    number_of_cf_viirs_swir_ifov_filter_occurrences:                   0
    number_of_cf_viirs_swir_ofova_filter_occurrences:                  0
    number_of_cf_viirs_swir_ofovb_filter_occurrences:                  0
    number_of_cf_viirs_swir_ofovc_filter_occurrences:                  0
    number_of_ch4_noscat_ratio_filter_occurrences:                     0
    number_of_ch4_noscat_ratio_std_filter_occurrences:                 0
    number_of_ch4_noscat_zero_error_occurrences:                       0
    number_of_chi2_error_occurrences:                                  0
    number_of_cirrus_reflectance_viirs_filter_occurrences:             0
    number_of_cloud_error_occurrences:                                 0
    number_of_cloud_filter_convergence_error_occurrences:              0
    number_of_cloud_filter_occurrences:                                0
    number_of_cloud_fraction_fresco_filter_occurrences:                0
    number_of_cloud_fraction_viirs_filter_occurrences:                 0
    number_of_cloud_pressure_spread_too_low_error_occurrences:         0
    number_of_cloud_too_low_level_error_occurrences:                   0
    number_of_cloud_warning_occurrences:                               0
    number_of_configuration_error_occurrences:                         0
    number_of_convergence_error_occurrences:                           0
    number_of_coregistration_error_occurrences:                        0
    number_of_data_range_warning_occurrences:                          0
    number_of_deconvolution_warning_occurrences:                       0
    number_of_dfs_error_occurrences:                                   0
    number_of_diff_psurf_fresco_ecmwf_filter_occurrences:              0
    number_of_diff_refl_cirrus_viirs_filter_occurrences:               0
    number_of_extrapolation_warning_occurrences:                       0
    number_of_failed_retrievals:                                       222068
    number_of_generic_exception_occurrences:                           0
    number_of_generic_range_error_occurrences:                         0
    number_of_geographic_region_filter_occurrences:                    0
    number_of_geolocation_error_occurrences:                           0
    number_of_groundpixels:                                            1460250
    number_of_ground_pixels_with_warnings:                             84516
    number_of_h2o_noscat_ratio_filter_occurrences:                     0
    number_of_h2o_noscat_ratio_std_filter_occurrences:                 0
    number_of_h2o_noscat_zero_error_occurrences:                       0
    number_of_initialization_error_occurrences:                        0
    number_of_input_spectrum_alignment_error_occurrences:              0
    number_of_input_spectrum_missing_occurrences:                      0
    number_of_input_spectrum_warning_occurrences:                      0
    number_of_interpolation_warning_occurrences:                       0
    number_of_io_error_occurrences:                                    0
    number_of_irradiance_missing_occurrences:                          0
    number_of_ISRF_error_occurrences:                                  0
    number_of_key_error_occurrences:                                   0
    number_of_ler_range_error_occurrences:                             0
    number_of_low_cloud_fraction_warning_occurrences:                  0
    number_of_lut_error_occurrences:                                   0
    number_of_lut_range_error_occurrences:                             0
    number_of_max_iteration_convergence_error_occurrences:             0
    number_of_max_optical_thickness_error_occurrences:                 0
    number_of_memory_error_occurrences:                                0
    number_of_mixed_surface_type_filter_occurrences:                   0
    number_of_model_error_occurrences:                                 0
    number_of_number_of_input_data_points_too_low_error_occurrences:   0
    number_of_numerical_error_occurrences:                             0
    number_of_ocean_filter_occurrences:                                0
    number_of_optimal_estimation_error_occurrences:                    0
    number_of_other_boundary_convergence_error_occurrences:            0
    number_of_ozone_range_error_occurrences:                           0
    number_of_pixel_level_input_data_missing_occurrences:              0
    number_of_pixel_or_scanline_index_filter_occurrences:              0
    number_of_processed_pixels:                                        1460250
    number_of_profile_error_occurrences:                               0
    number_of_psurf_fresco_stdv_filter_occurrences:                    0
    number_of_radiance_missing_occurrences:                            0
    number_of_radiative_transfer_error_occurrences:                    0
    number_of_reflectance_range_error_occurrences:                     0
    number_of_refl_cirrus_viirs_nir_filter_occurrences:                0
    number_of_refl_cirrus_viirs_swir_filter_occurrences:               0
    number_of_rejected_pixels_not_enough_spectrum:                     0
    number_of_saturation_error_occurrences:                            0
    number_of_saturation_warning_occurrences:                          0
    number_of_signal_to_noise_ratio_error_occurrences:                 0
    number_of_signal_to_noise_ratio_warning_occurrences:               0
    number_of_slant_column_density_error_occurrences:                  0
    number_of_small_pixel_radiance_std_filter_occurrences:             0
    number_of_snow_ice_filter_occurrences:                             0
    number_of_snow_ice_warning_occurrences:                            0
    number_of_snr_range_error_occurrences:                             0
    number_of_so2_volcanic_origin_certain_warning_occurrences:         0
    number_of_so2_volcanic_origin_likely_warning_occurrences:          0
    number_of_solar_eclipse_filter_occurrences:                        0
    number_of_south_atlantic_anomaly_warning_occurrences:              0
    number_of_successfully_processed_pixels:                           1238182
    number_of_sun_glint_correction_occurrences:                        0
    number_of_sun_glint_filter_occurrences:                            0
    number_of_sun_glint_warning_occurrences:                           84516
    number_of_svd_error_occurrences:                                   0
    number_of_sza_range_error_occurrences:                             222037
    number_of_time_range_filter_occurrences:                           0
    number_of_vertical_column_density_error_occurrences:               0
    number_of_vza_range_error_occurrences:                             0
    number_of_wavelength_calibration_error_occurrences:                0
    number_of_wavelength_calibration_warning_occurrences:              0
    number_of_wavelength_offset_error_occurrences:                     0
    number_of_wrong_input_type_error_occurrences:                      0
    time_for_algorithm_initialization:                                 23.927286
    time_for_processing:                                               212.96926
    time_per_pixel:                                                    0.0015...
    time_standard_deviation_per_pixel:                                 2.5657...
    algorithm_version:                                                 1.1.0
    build_date:                                                        2018-0...
    cdm_data_type:                                                     Swath
    Conventions:                                                       CF-1.7
    cpp_compiler_flags:                                                -g -O2...
    cpp_compiler_version:                                              g++ (G...
    creator_email:                                                     EOSupp...
    creator_name:                                                      The Se...
    creator_url:                                                       http:/...
    date_created:                                                      2018-0...
    f90_compiler_flags:                                                -gdwar...
    f90_compiler_version:                                              GNU Fo...
    geolocation_grid_from_band:                                        3
    geospatial_lat_max:                                                89.972939
    geospatial_lat_min:                                                -86.81926
    geospatial_lon_max:                                                -179.9...
    geospatial_lon_min:                                                179.99924
    history:                                                           2018-0...
    id:                                                                S5P_OF...
    identifier_product_doi:                                            10.527...
    identifier_product_doi_authority:                                  http:/...
    institution:                                                       KNMI
    keywords:                                                          0300 A...
    keywords_vocabulary:                                               AGU in...
    license:                                                           No con...
    naming_authority:                                                  nl.knmi
    orbit:                                                             4361
    platform:                                                          S5P
    processing_status:                                                 Nominal
    processor_version:                                                 1.1.0
    product_version:                                                   1.0.0
    project:                                                           Sentin...
    references:                                                        http:/...
    revision_control_identifier:                                       df649d...
    sensor:                                                            TROPOMI
    source:                                                            Sentin...
    spatial_resolution:                                                7x3.5km2
    standard_name_vocabulary:                                          NetCDF...
    Status_MET_2D:                                                     Nominal
    summary:                                                           TROPOM...
    time_coverage_duration:                                            PT3503...
    time_coverage_end:                                                 2018-0...
    time_coverage_resolution:                                          PT1.080S
    time_coverage_start:                                               2018-0...
    time_reference:                                                    2018-0...
    time_reference_days_since_1950:                                    25064
    time_reference_julian_day:                                         2458346.5
    time_reference_seconds_since_1970:                                 153437...
    title:                                                             TROPOM...
    tracking_id:                                                       0b05f3...