hdpws Example Jupyter Notebook¶

SPASE inside This Jupyter notebook demonstrates using the hdpws Python package to access Space Physics Archive Search and Extract (SPASE) metadata documents from the Heliophysics Data Portal (HDP). It assumes some familarity with the SPASE data model. This notebook contains the following sections:

  1. Prerequisites
  2. Setup
  3. Get ApplicationInterfaces
  4. Get MeasurementTypes
  5. Get SpectralRanges
  6. Get PhenomenonTypes
  7. Get ObservedRegions
  8. Get ObservatoryIDs
  9. Get Keywords
  10. Get Numerical/Display Data
  11. Get Catalog
  12. Get Collection
  13. Get Document
  14. Get Software
  15. Get a SPASE document by ResourceID
  16. Conditionally get a SPASE document
  17. Error Handling
  18. Additional Documentation

Prerequisites¶

Install the prerequisite software from Python Package Index (PyPI) software repository.

  1. pip install hdpws

Setup¶

Execute some preliminary code that is necessary before the code that follows.

In [34]:
from hdpws.hdpws import HdpWs
from hdpws import NAMESPACES as NS
from hdpws.resourcetype import ResourceType as rt
from hdpws.spase import AccessURL, HapiAccessURL

from IPython.core.display import HTML

hdp = HdpWs()
# limit display of long lists
DISP_LIMIT = 5

def display_link(
    url: str,
    text: str) -> None:
    display(HTML(f'&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<a href="{url}" target="_blank">{text}</a>'))

Get ApplicationInterfaces¶

The following code demonstrates how to get the list of available /Spase/Software/ApplicationInterface values.

In [35]:
result = hdp.get_application_interfaces()
print('HDP ApplicationInterfaces:')
for value in result['ApplicationInterface']:
    print(f'    {value}')

HDP ApplicationInterfaces:
    CLI
    GUI
    API

Get MeasurementTypes¶

The following code demonstrates how to get the list of available /Spase/MeasurementType values.

In [36]:
result = hdp.get_measurement_types()
print(f'HDP MeasurementTypes:')
for i, value in enumerate(result['MeasurementType']):
    print(f'{i}. {value}')

HDP MeasurementTypes:
0. ActivityIndex
1. Dopplergram
2. Dust
3. ElectricField
4. EnergeticParticles
5. Ephemeris
6. ImageIntensity
7. InstrumentStatus
8. IonComposition
9. Irradiance
10. MagneticField
11. Magnetogram
12. NeutralAtomImages
13. NeutralGas
14. Profile
15. Radiance
16. Spectrum
17. SPICE
18. ThermalPlasma
19. Waves
20. Waves.Active
21. Waves.Passive

Get SpectralRanges¶

The following code demonstrates how to get the list of available /Spase/SpectralRange values.

In [37]:
result = hdp.get_spectral_ranges()
print(f'HDP SpectralRanges:')
for i, value in enumerate(result['SpectralRange']):
    print(f'{i}. {value}')

HDP SpectralRanges:
0. CaK
1. ExtremeUltraviolet
2. FarUltraviolet
3. GammaRays
4. Halpha
5. HardXRays
6. He10830
7. He304
8. Infrared
9. K7699
10. LBHBand
11. Microwave
12. NaD
13. Ni6768
14. Optical
15. RadioFrequency
16. SoftXRays
17. Ultraviolet
18. WhiteLight
19. XRays

Get PhenomenonTypes¶

The following code demonstrates how to get the list of available /Spase/PhenomenonType values.

In [38]:
result = hdp.get_phenomenon_types()
print(f'HDP PhenomenonTypes:')
for i, value in enumerate(result['PhenomenonType']):
    print(f'{i}. {value}')

HDP PhenomenonTypes:
0. ActiveRegion
1. Aurora
2. BowShockCrossing
3. CoronalHole
4. CoronalMassEjection
5. EITWave
6. EnergeticSolarParticleEvent
7. ForbushDecrease
8. GeomagneticStorm
9. InterplanetaryCoronalMassEjection
10. InterplanetaryShock
11. MagneticCloud
12. MagnetopauseCrossing
13. RadioBurst
14. SectorBoundaryCrossing
15. SolarFlare
16. SolarWindExtreme
17. StreamInteractionRegion
18. Substorm

Get ObservedRegions¶

The following code demonstrates how to get the list of available /Spase/ObservedRegion values.

In [39]:
result = hdp.get_observed_regions()
observed_regions = result['ObservedRegion']
print(f'HDP Observed Regions:')
for i, value in enumerate(observed_regions):
    print(f'{i}. {value}')
    if i == DISP_LIMIT:
        print('...')
        print(f'{len(observed_regions) - 1}. {observed_regions[-1]}')
        break

HDP Observed Regions:
0. Asteroid
1. Comet
2. Comet.1PHalley
3. Comet.26PGriggSkjellerup
4. Comet.67PChuryumovGerasimenko
5. Earth
...
124. Venus.Magnetosphere.RingCurrent

Get ObservatoryIDs¶

The following code demonstrates how to get the list of available /Spase/Observatory/ResourceID values.

In [40]:
result = hdp.get_observatory_ids()
observatory_ids = result['ObservatoryID']
print(f'HDP ObservatoryIDs:')
for i, value in enumerate(observatory_ids):
    print(f'{i}. {value}')
    if i == DISP_LIMIT:
        print('...')
        print(f'{len(observatory_ids) - 1}. {observatory_ids[-1]}')
        break

HDP ObservatoryIDs:
0. spase://SMWG/Observatory/AE-D
1. spase://SMWG/Observatory/Helios1
2. spase://SMWG/Observatory/DynamicsExplorer1
3. spase://SMWG/Observatory/SolarOrbiter
4. spase://SMWG/Observatory/IMP8
5. spase://SMWG/Observatory/MarsExpress
...
4043. spase://SMWG/Observatory/UNH

Get Keywords¶

The following code demonstrates how to get a list of available //Keyword values.

In [41]:
result = hdp.get_keywords()
keywords = result['Keyword']
print(f'HDP Keywords:')
for i, value in enumerate(keywords):
    print(f'{i}. {value}')
    if i == DISP_LIMIT:
        print('...')
        print(f'{len(keywords) - 1}. {keywords[-1]}')
        break

HDP Keywords:
0. 60 sec
1. ground magnetometer data
2. electroject ionospheric currents
3. auroral electrojet index
4. 10 sec
5. spherical elementary current amplitudes
...
1518. WDC Station Code: KZN

Get Numerical/Display Data¶

The following code demonstrates how to get SPASE Numerical/Display data documents matching the specified search criteria.

In [42]:
query = {
    'InstrumentID': 'spase://SMWG/Instrument/ACE/CRIS',
    'Cadence': '=PT1H',
    'ObservedRegion': 'Heliosphere.NearEarth',
    'MeasurementType': 'EnergeticParticles',
    'AccessRights': 'Open',
    'Format': 'CDF'
}       
types = [rt.NUMERICAL_DATA, rt.DISPLAY_DATA]
time_range = ['2022-01-01', '2022-01-02']
result = hdp.get_spase_data(types, query, time_range)
if result['HttpStatus'] == 200:
    for i, spase in enumerate(result['Result'].findall('.//Spase', namespaces=NS)):
        id = spase.find('.//ResourceID', namespaces=NS)
        name = spase.find('.//Name', namespaces=NS)
        description = spase.find('.//Description', namespaces=NS)
        #print(name.text)
        print(f'{i}. ResourceID: {id.text}')
        print('--------------------------------------------------')
        print(f'Description: {description.text[:60]} ...')
        display_link(hdp.get_spase_url(id.text), 'HTML representation of SPASE')
        ws_access_url_element = \
            spase.find('.//AccessURL[Style="WebService"]', namespaces=NS)
        if ws_access_url_element is not None:
            ws_access_url = AccessURL(ws_access_url_element)
            print('Web Service access:')

            display_link(ws_access_url.url, 'Example data access code')
            #print('    URL: ', ws_access_url.url) 
        hapi_access_url_element = \
            spase.find('.//AccessURL[Style="HAPI"]', namespaces=NS)
        if hapi_access_url_element is not None:
            hapi_access_url = HapiAccessURL(hapi_access_url_element)
            print('HAPI access:')
            print('    URL: ', hapi_access_url.url)
            print('    Dataset IDs: ', hapi_access_url.product_key)
            hapi_display_url = hapi_access_url.get_html_url()
            display_link(hapi_access_url.get_html_url(), 'HAPI access information')
        print()

0. ResourceID: spase://NASA/NumericalData/ACE/CRIS/L2/PT1H
--------------------------------------------------
Description: ACE Cosmic Ray Isotope Spectrometer (CRIS) intensities and c ...
     HTML representation of SPASE 
Web Service access:
     Example data access code 
HAPI access:
    URL:  https://cdaweb.gsfc.nasa.gov/hapi
    Dataset IDs:  ['AC_H2_CRIS']
     HAPI access information 
1. ResourceID: spase://NASA/NumericalData/ParkerSolarProbe/SWEAP/SPAN-I/Level2/AlphaDifferentialEnergyFlux/VariableCadence
--------------------------------------------------
Description: Parker Solar Probe SPAN-I Alpha particle differential energy ...
     HTML representation of SPASE 
Web Service access:
     Example data access code 
HAPI access:
    URL:  https://cdaweb.gsfc.nasa.gov/hapi
    Dataset IDs:  ['PSP_SWP_SPI_SF01_L2_8DX32EX8A']
     HAPI access information 
























Get Catalog¶
The following code demonstrates how to get a SPASE Catalog document matching the specified criteria.


















In [43]:






query = { 
#    'InstrumentID': 'spase://SMWG/Instrument/ACE/MAG',
#    'PhenomenonType': 'MagneticCloud',
#    'Description': 'ICME'
    'Keyword': 'current sheet'
}   
#time_range = ['1999-01-01', '1999-01-02']
time_range = ['2005-01-01', '2014-01-01']
result = hdp.get_spase_catalog(query, time_range)
if result['HttpStatus'] == 200:
    print('Result Catalogs:')
    for catalog in result['Result'].findall('.//Catalog',
                                           namespaces=NS):
        print('ResourceID: ', catalog.findall('.//ResourceID',
                             namespaces=NS)[0].text)
        print('ResourceName: ', catalog.findall('.//ResourceName',
                                     namespaces=NS)[0].text) 
        print('Description:', 
              catalog.findall('.//Description',
                              namespaces=NS)[0].text[:750],
             '...')

























Result Catalogs:
ResourceID:  spase://NASA/Catalog/Eriksson/WindCurrentSheetExhaustList
ResourceName:  Wind Current Sheet_Exhaust Event List in LMN Coordinates
Description: The dataset contains a list of solar wind current sheet exhaust event data derived from Wind interplanetary magnetic field and solar wind plasma observations. The list is described in a peer reviewed article, Eriksson et al. 2022, titled: Characteristics of Multi-Scale Current Sheets in the Solar Wind at 1 AU Associated with Magnetic Reconnection and the Case for a Heliospheric Current Sheet Avalanche. The data is listed as an ASCII table with eleven columns including the current sheet (CS) start date/time and stop date/time, and the three unit vectors of each CS in a GSE base coordinate system. Here, NGSE is the CS normal direction, LGSE is the direction of the reconnecting component of the magnetic field, and MGSE is the cross-product MGS ...

























Get Collection¶
The following code demonstrates how to get a SPASE Collection document matching the specified criteria.


















In [44]:






query = { 
    'ResourceID': 'spase://NASA/Collection/IRIS/IRIS_SDOAIA',
    'MemberID': 'spase://NASA/NumericalData/IRIS/IRIS/PT1S',
    'Description': 'IRIS AND SDO and AIA'
}   
result = hdp.get_spase_collection(query)
if result['HttpStatus'] == 200:
    for collection in result['Result'].findall('.//Collection',
                                               namespaces=NS):
        print('ResourceID: ', collection.find('.//ResourceID',
                                              namespaces=NS).text)
        print('ResourceName: ', collection.find('.//ResourceName',
                                                namespaces=NS).text)
        print('Collection members:')
        for member in collection.findall('.//Member',
                                         namespaces=NS):
            member_id = member.find('.//MemberID',
                                    namespaces=NS)
            print(f'    {member_id.text}')
else:
    print(result['HttpStatus'])

























ResourceID:  spase://NASA/Collection/IRIS/IRIS_SDOAIA
ResourceName:  Co-aligned Interface Region Imaging Spectrograph (IRIS) and Solar Dynamics Observatory (SDO) Atmospheric Imaging Assembly (AIA) Observations
Collection members:
    spase://NASA/NumericalData/IRIS/IRIS/PT1S
    spase://NASA/NumericalData/SDO/AIA/PT12S

























Get Document¶
The following code demonstrates how to get the SPASE Document matching the specified criteria.


















In [45]:






query = {
    'ResourceID': 'spase://SMWG/Document/HPDE/Policy/HP_DataPolicy_v1.2',
    'DOI': '10.21978/P83P78',
    'Description': '"Program Data Management Plan"'
}
result = hdp.get_spase_document(query)
if result['HttpStatus'] == 200:
    print('Result Documents:')
    for document in result['Result'].findall('.//Document', 
                                             namespaces=NS):
        print('ResourceName: ', document.find('.//ResourceName', 
                                              namespaces=NS).text)
        description = document.find('.//Description', 
                                    namespaces=NS)
        print(f'Description: {description.text}')
        url = document.find('.//AccessInformation/AccessURL/URL',
                           namespaces=NS)
        print(f'URL: {url.text}')

























Result Documents:
ResourceName:  NASA Heliophysics Science Data Management Policy v. 1.2
Description: NASA Heliophysics document that provides guidance on all aspects of HP data, including what needs to go into a Program Data Management Plan, the structure and use of archives, required formats, and a description of the data registry, all in the context of the data lifecycle.
URL: https://cdaweb.gsfc.nasa.gov/pub/documents/HPDE/Heliophysics_Data_Policy_v1.2_2016Oct04_signed.pdf

























Get Software¶
The following code demonstrates how to get the SPASE Software matching the specified criteria.


















In [46]:






query = { 
    'ResourceID': 'spase://CCMC/Software/Kamodo',
    'CodeLanguage': 'Python',
    'Description': '"space weather models and data"'
}

result = hdp.get_spase_software(query)
if result['HttpStatus'] == 200:
    print('Result Software:')
    for software in result['Result'].findall('.//Software', namespaces=NS):
        print(software.findall('.//ResourceID', namespaces=NS)[0].text)
        print(f'    {software.findall(".//ResourceName", namespaces=NS)[0].text}')

























Result Software:
spase://CCMC/Software/Kamodo
    CCMC Kamodo Software

























Get a SPASE document by ResourceID¶
The following code demonstrates how to get the SPASE document with the specified ResourceID.


















In [47]:






resource_ids = [
    'spase://NASA/Collection/IRIS_AIA',
    'spase://SMWG/Service/CCMC/Models'
]
result = hdp.get_spase(resource_ids)
if result['HttpStatus'] == 200:
    print('get_spase Result ResourceIDs:')
    for spase in result['Result'].findall('.//Spase',
                                         namespaces=NS):
        print('ResourceID: ', spase.find('.//ResourceID',
                           namespaces=NS).text)
        print('ResourceName: ', spase.find('.//ResourceName',
                                   namespaces=NS).text)
    if 'Last-Modified' in result:
        last_modified = result['Last-Modified']
        print(f'last_modified = {last_modified.isoformat()}')

























get_spase Result ResourceIDs:
ResourceID:  spase://SMWG/Service/CCMC/Models
ResourceName:  CCMC Modelweb page
ResourceID:  spase://SMWG/Service/CCMC/Models
ResourceName:  CCMC Modelweb page
last_modified = 2024-01-11T11:39:22+00:00

























Conditionally get a SPASE document¶
The following code demonstrates how to get a SPASE document only if it has been modified since the last time it was gotten.


















In [48]:






if last_modified is not None:
    result = hdp.get_spase(resource_ids, if_modified_since=last_modified)

    if result['HttpStatus'] == 304:
        print(f'get_spase if_modified_since {last_modified.isoformat()} return Not Modified')
    else:
        print(f'get_spase if_modified_since returned an unexpected result {result["HttpStatus"]}')

























get_spase if_modified_since 2024-01-11T11:39:22+00:00 return Not Modified

























Error Handling¶
The following code demonstrates how to view an error message from the server in response to a bad request from the client.


















In [49]:






query = {
    'ResourceID': ['spase://NASA/NumericalData/ACE/CRIS/L2/P1D'],
    'Description': '~bad'
}
result = hdp.get_spase_data(types, query, time_range)
if result['HttpStatus'] != 200:
    print(f'hdp.get_spase() failed with status = {result["HttpStatus"]}')
    if 'ErrorMessage' in result:
        print(f'ErrorMessage = {result["ErrorMessage"]}')
        print(f'ErrorDescription = {result["ErrorDescription"]}')
    #else:
        #print(f'HttpText = {result["ErrorText"]}')
        print('Display of actual HTML page returned:')
        display(HTML(result['ErrorText']))

























hdp.get_spase() failed with status = 400
ErrorMessage = Bad Request
ErrorDescription = Invalid Description syntax.
Display of actual HTML page returned:














Bad Request




Message: Bad Request


Description: Invalid Description syntax.




See Apache Lucene
                                for a description of the search syntax.


Contact 📧 NASA-SPDF-Support if additional help is required.




























Additional Documentation¶
View the hdpws API documentation for a description of additional features.


















In [ ]: