hdpws and cdasws Example Jupyter Notebook¶

SPASE inside This Jupyter notebook demonstrates using the hdpws Python package to access Space Physics Archive Search and Extract (SPASE) metadata documents from the Heliophysics Data Portal (HDP). Additionally, it utilizes cdasws to retrieve the data described by the SPASE metadata. It assumes some familarity with the SPASE data model. This notebook contains the following sections:

  1. Prerequisites
  2. Setup
  3. Discover the Software used to access data
  4. Get MeasurementTypes
  5. Get ObservedRegions
  6. Get ObservatoryIDs
  7. Get NumericalData
  8. Additional Documentation

Prerequisites¶

Install the prerequisite software from Python Package Index (PyPI) software repository.

  1. pip install hdpws
  2. pip install xarray
  3. pip install cdflib
  4. pip install cdasws

Setup¶

Execute some preliminary code that is necessary before the code that follows.

In [20]:
from datetime import datetime

from hdpws.hdpws import HdpWs
from hdpws import NAMESPACES as NS
from hdpws.resourcetype import ResourceType as rt
from hdpws.spase import AccessURL

from cdasws import CdasWs
from cdasws.datarepresentation import DataRepresentation as dr

from IPython.core.display import HTML


hdp = HdpWs()
cdas = CdasWs()
# limit display of long lists
DISP_LIMIT = 5

today = datetime.now().strftime('%Y-%B-%d')

def display_link(
    url: str,
    text: str) -> None:
    display(HTML(f'<a href="{url}" target="_blank">{text}</a>'))

Discover the Software used to access data¶

The following code demonstrates discovering the software we will use below to get the data we want.

In [19]:
query = {
    'CodeLanguage': 'Python',
    'Description': 'CDAS'
}
result = hdp.get_spase_software(query)
if result['HttpStatus'] == 200:
    print(f'Software search results:')
    for software in result['Result'].findall('.//Software', namespaces=NS):
        resource_id = software.findall('.//ResourceID', namespaces=NS)[0].text
        resource_name = software.findall('.//ResourceName', namespaces=NS)[0].text
        display_link(f'{hdp.endpoint}Spase?ResourceID={resource_id}', resource_name)
else:
    print(result['HttpStatus'])

Software search results:
Coordinated Data Analysis System Web Services Client Library

Get MeasurementTypes¶

The following code demonstrates how to get the list of available /Spase/MeasurementType values.

In [3]:
result = hdp.get_measurement_types()
print('HDP MeasurementTypes:')
for i, value in enumerate(result['MeasurementType']):
    print(f'{i}. {value}')

HDP MeasurementTypes:
0. ActivityIndex
1. Dopplergram
2. Dust
3. ElectricField
4. EnergeticParticles
5. Ephemeris
6. ImageIntensity
7. InstrumentStatus
8. IonComposition
9. Irradiance
10. MagneticField
11. Magnetogram
12. NeutralAtomImages
13. NeutralGas
14. Profile
15. Radiance
16. Spectrum
17. SPICE
18. ThermalPlasma
19. Waves
20. Waves.Active
21. Waves.Passive

Get ObservedRegions¶

The following code demonstrates how to get the list of available /Spase/ObservedRegion values.

In [4]:
result = hdp.get_observed_regions()
observed_regions = result['ObservedRegion']
print(f'HDP Observed Regions:')
for i, value in enumerate(observed_regions):
    print(f'{i}. {value}')
    if i == DISP_LIMIT:
        print('...')
        print(f'{len(observed_regions) - 1}. {observed_regions[-1]}')
        break

HDP Observed Regions:
0. Asteroid
1. Comet
2. Comet.1PHalley
3. Comet.26PGriggSkjellerup
4. Comet.67PChuryumovGerasimenko
5. Earth
...
124. Venus.Magnetosphere.RingCurrent

Get ObservatoryIDs¶

The following code demonstrates how to get the list of available /Spase/Observatory/ResourceID values.

In [5]:
result = hdp.get_observatory_ids()
observatory_ids = result['ObservatoryID']
print(f'HDP ObservatoryIDs:')
for i, value in enumerate(observatory_ids):
    print(f'{i}. {value}')
    if i == DISP_LIMIT:
        print('...')
        print(f'{len(observatory_ids) - 1}. {observatory_ids[-1]}')
        break

HDP ObservatoryIDs:
0. spase://SMWG/Observatory/AE-D
1. spase://SMWG/Observatory/Helios1
2. spase://SMWG/Observatory/DynamicsExplorer1
3. spase://SMWG/Observatory/SolarOrbiter
4. spase://SMWG/Observatory/IMP8
5. spase://SMWG/Observatory/MarsExpress
...
4043. spase://SMWG/Observatory/UNH

Get NumericalData¶

The following code demonstrates how to find SPASE NumericalData documents matching the specified search criteria. Additionally, it utilizes cdasws to retrieve the data described by the SPASE metadata when the data is available from CDAWeb.

In [22]:
query = {
    'InstrumentID': 'spase://SMWG/Instrument/ACE/MAG',
    'Cadence': '=PT4M',
    'ObservedRegion': 'Heliosphere.NearEarth',
    'MeasurementType': 'MagneticField',
    'AccessRights': 'Open',
    'Style': 'WebService'
}       
types = [rt.NUMERICAL_DATA]
time_range = ['2022-01-01', '2022-01-02']
result = hdp.get_spase_data(types, query, time_range)
if result['HttpStatus'] == 200:
    for spase in result['Result'].findall('.//Spase', namespaces=NS):
        id = spase.find('.//ResourceID', namespaces=NS)
        name = spase.find('.//Name', namespaces=NS)
        description = spase.find('.//Description', namespaces=NS)
        #print('Name: ', name.text)
        print(f'ResourceID: {id.text}')
        print(f'Description: {description.text[:60]}...')
        display_link(hdp.get_spase_url(id.text), 'HTML representation of SPASE')
        ws_access_url_element = \
            spase.find('.//AccessURL[Style="WebService"]', namespaces=NS)
        if ws_access_url_element is not None:
            ws_access_url = AccessURL(ws_access_url_element)
            ws_name = ws_access_url.name
            ws_product_key = ws_access_url.product_key[0]
            if "CDAWeb" in ws_name:
                display_link(ws_access_url.url, 'Example data access code')
                description = cdas.get_datasets(id = ws_product_key)
                doi = description[0]['Doi']
                display(HTML(f'<b>Data Citation</b>: {cdas.get_citation(doi)}.  Accessed on {today}.'))
                print('Retrieving data...')
                example_interval = cdas.get_example_time_interval(ws_product_key)
                var_names = cdas.get_variable_names(ws_product_key)
                data = cdas.get_data(ws_product_key, var_names,
                                     example_interval,
                                     dataRepresentation = dr.XARRAY)[1]
                print(data)

ResourceID: spase://NASA/NumericalData/ACE/MAG/L2/PT4M
Description: This Data Product contains Measurements from the ACE Magneti...
HTML representation of SPASE
Example data access code
Data Citation: Smith, C. W., & Ness, N. F. (2022). ACE Magnetic Field (MAG) Geocentric Solar Ecliptic, GSE, and Geocentric Solar Magnetospheric, GSM, Coordinates, Level 2 (H1), 4 min Data [Data set]. NASA Space Physics Data Facility. https://doi.org/10.48322/BRF1-G493. Accessed on 2024-November-15. 
Retrieving data...
<xarray.Dataset> Size: 2kB
Dimensions:     (Epoch: 31, cartesian: 3, dim0: 3)
Coordinates:
  * Epoch       (Epoch) datetime64[ns] 248B 2024-06-12T21:56:00 ... 2024-06-1...
  * cartesian   (cartesian) <U11 132B 'x_component' 'y_component' 'z_component'
    metavar0    (cartesian) <U6 72B 'Bx GSE' 'By GSE' 'Bz GSE'
    metavar2    (cartesian) <U9 108B 'ACE X-GSE' 'ACE Y-GSE' 'ACE Z-GSE'
    metavar3    (cartesian) <U9 108B 'ACE X-GSM' 'ACE Y-GSM' 'ACE Z-GSM'
Dimensions without coordinates: dim0
Data variables:
    Magnitude   (Epoch) float32 124B 2.381 2.335 2.55 ... 3.996 3.977 3.921
    BGSEc       (Epoch, cartesian) float32 372B 0.476 0.737 ... 2.145 0.352
    BGSM        (Epoch, dim0) float32 372B 0.476 1.196 -1.9 ... 3.26 2.018 0.808
    SC_pos_GSE  (Epoch, cartesian) float32 372B 1.48e+06 2.496e+05 ... 1.04e+05
    SC_pos_GSM  (Epoch, cartesian) float32 372B 1.48e+06 2.197e+05 ... 1.556e+05
    metavar1    (dim0) <U8 96B 'Bx (GSM)' 'By (GSM)' 'Bz (GSM)'
Attributes: (12/30)
    TITLE:                       ['ACE> Magnetometer Parameters']
    Project:                     ['ISTP>International Solar-Terrestrial Physi...
    Discipline:                  ['Space Physics>Interplanetary Studies']
    Source_name:                 ['AC>Advanced Composition Explorer']
    Data_type:                   ['H1>4-Min Level 2 Data']
    Descriptor:                  ['MAG>ACE Magnetic Field Instrument']
    ...                          ...
    Acknowledgement:             ['Please acknowledge the Principal', 'Invest...
    Rules_of_use:                ['See the rules of use available from the AC...
    Alt_logical_source:          ['ACE_MAG_Solar-Wind-Magnetic-Field-Level2-H...
    spase_DatasetResourceID:     ['spase://NASA/NumericalData/ACE/MAG/L2/PT4M']
    DOI:                         ['https://doi.org/10.48322/brf1-g493']
    CDAWEB_PARENTS:              ['ac_h1_mfi_00000000_v01', 'ac_h1_mfi_202406...

Additional Documentation¶

View the hdpws and cdasws API documentation for a description of additional features.

In [ ]: