This Jupyter notebook demonstrates using the hdpws Python package to access Space Physics Archive Search and Extract (SPASE) metadata documents from the Heliophysics Data Portal (HDP). It assumes some familarity with the SPASE data model. This notebook contains the following sections:
Install the prerequisite software from Python Package Index (PyPI) software repository.
Execute some preliminary code that is necessary before the code that follows.
from hdpws.hdpws import HdpWs
from hdpws import NAMESPACES as NS
from hdpws.resourcetype import ResourceType as rt
from hdpws.spase import AccessURL, HapiAccessURL
from IPython.core.display import HTML
hdp = HdpWs()
The following code demonstrates how to get the list of available /Spase/Software/ApplicationInterface values.
result = hdp.get_application_interfaces()
print('HDP ApplicationInterfaces:')
for value in result['ApplicationInterface']:
print(f' {value}')
HDP ApplicationInterfaces: CLI GUI API
The following code demonstrates how to get the list of available /Spase/MeasurementType values.
result = hdp.get_measurement_types()
print('HDP MeasurementTypes:')
for value in result['MeasurementType']:
print(f' {value}')
HDP MeasurementTypes: ActivityIndex Dopplergram Dust ElectricField EnergeticParticles Ephemeris ImageIntensity InstrumentStatus IonComposition Irradiance MagneticField Magnetogram NeutralAtomImages NeutralGas Profile Radiance Spectrum SPICE ThermalPlasma Waves Waves.Active Waves.Passive
The following code demonstrates how to get the list of available /Spase/SpectralRange values.
result = hdp.get_spectral_ranges()
print('HDP SpectralRanges:')
for value in result['SpectralRange']:
print(f' {value}')
HDP SpectralRanges: CaK ExtremeUltraviolet FarUltraviolet GammaRays Halpha HardXRays He10830 He304 Infrared K7699 LBHBand Microwave NaD Ni6768 Optical RadioFrequency SoftXRays Ultraviolet WhiteLight XRays
The following code demonstrates how to get the list of available /Spase/PhenomenonType values.
result = hdp.get_phenomenon_types()
print('HDP PhenomenonTypes:')
for value in result['PhenomenonType']:
print(f' {value}')
HDP PhenomenonTypes: ActiveRegion Aurora BowShockCrossing CoronalHole CoronalMassEjection EITWave EnergeticSolarParticleEvent ForbushDecrease GeomagneticStorm InterplanetaryShock MagneticCloud MagnetopauseCrossing RadioBurst SectorBoundaryCrossing SolarFlare SolarWindExtreme StreamInteractionRegion Substorm
The following code demonstrates how to get the list of available /Spase/ObservedRegion values.
result = hdp.get_observed_regions()
observed_regions = result['ObservedRegion']
print(f'{len(observed_regions)} HDP Observed Regions:')
for value in observed_regions[0:9]:
print(f' {value}')
print(' ...')
125 HDP Observed Regions: Asteroid Comet Comet.1PHalley Comet.26PGriggSkjellerup Comet.67PChuryumovGerasimenko Earth Earth.Magnetosheath Earth.Magnetosphere Earth.Magnetosphere.Magnetotail ...
The following code demonstrates how to get the list of available /Spase/Observatory/ResourceID values.
result = hdp.get_observatory_ids()
observatory_ids = result['ObservatoryID']
print(f'{len(observatory_ids)} HDP ObservatoryIDs:')
for value in observatory_ids[0:9]:
print(f' {value}')
print(' ...')
2937 HDP ObservatoryIDs: spase://SMWG/Observatory/AE-D spase://SMWG/Observatory/Helios1 spase://SMWG/Observatory/DynamicsExplorer1 spase://SMWG/Observatory/SolarOrbiter spase://SMWG/Observatory/IMP8 spase://SMWG/Observatory/MarsExpress spase://SMWG/Observatory/GIRO spase://SMWG/Observatory/Interball-Tail spase://SMWG/Observatory/IRIS ...
The following code demonstrates how to get a list of available //Keyword values.
result = hdp.get_keywords()
keywords = result['Keyword']
print(f'{len(keywords)} HDP Keywords:')
for value in keywords[0:9]:
print(f' {value}')
print(' ...')
449 HDP Keywords: 60 sec ground magnetometer data electroject ionospheric currents auroral electrojet index 10 sec spherical elementary current amplitudes spherical elementary current systms equivalent ionospheric currents Ionogram ...
The following code demonstrates how to get SPASE Numerical/Display data documents matching the specified search criteria.
query = {
'InstrumentID': 'spase://SMWG/Instrument/ACE/CRIS',
'Cadence': '=PT1H',
'ObservedRegion': 'Heliosphere.NearEarth',
'MeasurementType': 'EnergeticParticles',
'AccessRights': 'Open',
'Format': 'CDF'
}
types = [rt.NUMERICAL_DATA, rt.DISPLAY_DATA]
time_range = ['2022-01-01', '2022-01-02']
result = hdp.get_spase_data(types, query, time_range)
if result['HttpStatus'] == 200:
print('Results:')
for spase in result['Result'].findall('.//Spase', namespaces=NS):
id = spase.find('.//ResourceID', namespaces=NS)
name = spase.find('.//Name', namespaces=NS)
description = spase.find('.//Description', namespaces=NS)
#print(name.text)
print('ResourceID: ', id.text)
print('Description: ', description.text[:60], '...')
display(HTML('<a href="' + hdp.get_spase_url(id.text) +
'" target="_blank">' +
'HTML representation of SPASE</a>'))
ws_access_url_element = \
spase.find('.//AccessURL[Style="WebService"]', namespaces=NS)
if ws_access_url_element is not None:
ws_access_url = AccessURL(ws_access_url_element)
print('Web Service access:')
display(HTML(' <a href="' +
ws_access_url.url + '" target="_blank">' +
'Example data access code</a>'))
#print(' URL: ', ws_access_url.url)
hapi_access_url_element = \
spase.find('.//AccessURL[Style="HAPI"]', namespaces=NS)
if hapi_access_url_element is not None:
hapi_access_url = HapiAccessURL(hapi_access_url_element)
print('HAPI access:')
print(' URL: ', hapi_access_url.url)
print(' Dataset IDs: ', hapi_access_url.product_key)
hapi_display_url = hapi_access_url.get_html_url()
hapi_html_link = " <a href=""" + \
hapi_access_url.get_html_url() + \
""" target=""_blank"">HAPI access information</a>"""
display(HTML(hapi_html_link))
print('=======================================')
Results: ResourceID: spase://NASA/NumericalData/ACE/CRIS/L2/PT1H Description: ACE Cosmic Ray Isotope Spectrometer (CRIS) intensities and c ...
Web Service access:
HAPI access: URL: https://cdaweb.gsfc.nasa.gov/hapi Dataset IDs: ['AC_H2_CRIS']
======================================= ResourceID: spase://NASA/NumericalData/ACE/CRIS/L2/P1D Description: ACE Cosmic Ray Isotope Spectrometer (CRIS) intensities and c ...
Web Service access:
HAPI access: URL: https://cdaweb.gsfc.nasa.gov/hapi Dataset IDs: ['AC_H3_CRIS']
=======================================
The following code demonstrates how to get a SPASE Catalog document matching the specified criteria.
query = {
# 'InstrumentID': 'spase://SMWG/Instrument/ACE/MAG',
# 'PhenomenonType': 'MagneticCloud',
# 'Description': 'ICME'
'Keyword': 'current sheet'
}
#time_range = ['1999-01-01', '1999-01-02']
time_range = ['2005-01-01', '2014-01-01']
result = hdp.get_spase_catalog(query, time_range)
if result['HttpStatus'] == 200:
print('Result Catalogs:')
for catalog in result['Result'].findall('.//Catalog',
namespaces=NS):
print('ResourceID: ', catalog.findall('.//ResourceID',
namespaces=NS)[0].text)
print('ResourceName: ', catalog.findall('.//ResourceName',
namespaces=NS)[0].text)
print('Description:',
catalog.findall('.//Description',
namespaces=NS)[0].text[:750],
'...')
Result Catalogs: ResourceID: spase://NASA/Catalog/Eriksson/WindCurrentSheetExhaustList ResourceName: Wind Current Sheet_Exhaust Event List in LMN Coordinates Description: The dataset contains a list of solar wind current sheet exhaust event data derived from Wind interplanetary magnetic field and solar wind plasma observations. The list is described in a peer reviewed article, Eriksson et al. 2022, titled: Characteristics of Multi-Scale Current Sheets in the Solar Wind at 1 AU Associated with Magnetic Reconnection and the Case for a Heliospheric Current Sheet Avalanche. The data is listed as an ASCII table with eleven columns including the current sheet (CS) start date/time and stop date/time, and the three unit vectors of each CS in a GSE base coordinate system. Here, NGSE is the CS normal direction, LGSE is the direction of the reconnecting component of the magnetic field, and MGSE is the cross-product MGS ...
The following code demonstrates how to get a SPASE Collection document matching the specified criteria.
query = {
'ResourceID': 'spase://NASA/Collection/IRIS_AIA',
'MemberID': 'spase://NASA/NumericalData/SDO/AIA/PT10S',
'Description': 'IRIS AND SDO and AIA'
}
result = hdp.get_spase_collection(query)
if result['HttpStatus'] == 200:
for collection in result['Result'].findall('.//Collection',
namespaces=NS):
print('ResourceID: ', collection.find('.//ResourceID',
namespaces=NS).text)
print('ResourceName: ', collection.find('.//ResourceName',
namespaces=NS).text)
print('Collection members:')
for member in collection.findall('.//Member',
namespaces=NS):
member_id = member.find('.//MemberID',
namespaces=NS)
print(' ', member_id.text)
ResourceID: spase://NASA/Collection/IRIS_AIA ResourceName: Co-aligned Interface Region Imaging Spectrograph (IRIS) and Solar Dynamics Observatory (SDO) Atmospheric Imaging Assembly (AIA) Observations Collection members: spase://NASA/NumericalData/IRIS/IRIS/PT1S spase://NASA/NumericalData/SDO/AIA/PT10S
The following code demonstrates how to get the SPASE Document matching the specified criteria.
query = {
'ResourceID': 'spase://SMWG/Document/HPDE/Policy/HP_DataPolicy_v1.2',
'DOI': '10.21978/P83P78',
'Description': '"Program Data Management Plan"'
}
result = hdp.get_spase_document(query)
if result['HttpStatus'] == 200:
print('Result Documents:')
for document in result['Result'].findall('.//Document',
namespaces=NS):
print('ResourceName: ', document.find('.//ResourceName',
namespaces=NS).text)
description = document.find('.//Description',
namespaces=NS)
print('Description: ', description.text)
url = document.find('.//AccessInformation/AccessURL/URL',
namespaces=NS)
print('URL: ', url.text)
Result Documents: ResourceName: NASA Heliophysics Science Data Management Policy v. 1.2 Description: NASA Heliophysics document that provides guidance on all aspects of HP data, including what needs to go into a Program Data Management Plan, the structure and use of archives, required formats, and a description of the data registry, all in the context of the data lifecycle. URL: https://cdaweb.gsfc.nasa.gov/pub/documents/HPDE/Heliophysics_Data_Policy_v1.2_2016Oct04_signed.pdf
The following code demonstrates how to get the SPASE Software matching the specified criteria.
query = {
'ResourceID': 'spase://CCMC/Software/Kamodo',
'CodeLanguage': 'Python',
'Description': '"space weather models and data"'
}
result = hdp.get_spase_software(query)
if result['HttpStatus'] == 200:
print('Result Software:')
for software in result['Result'].findall('.//Software', namespaces=NS):
print(software.findall('.//ResourceID', namespaces=NS)[0].text)
print(' ', software.findall('.//ResourceName', namespaces=NS)[0].text)
Result Software: spase://CCMC/Software/Kamodo CCMC Kamodo Software
The following code demonstrates how to get the SPASE document with the specified ResourceID.
resource_ids = [
'spase://NASA/Collection/IRIS_AIA',
'spase://SMWG/Service/CCMC/Models'
]
result = hdp.get_spase(resource_ids)
if result['HttpStatus'] == 200:
print('get_spase Result ResourceIDs:')
for spase in result['Result'].findall('.//Spase',
namespaces=NS):
print('ResourceID: ', spase.find('.//ResourceID',
namespaces=NS).text)
print('ResourceName: ', spase.find('.//ResourceName',
namespaces=NS).text)
if 'Last-Modified' in result:
last_modified = result['Last-Modified']
print('last_modified = ', last_modified.isoformat())
get_spase Result ResourceIDs: ResourceID: spase://SMWG/Service/CCMC/Models ResourceName: CCMC Modelweb page ResourceID: spase://NASA/Collection/IRIS_AIA ResourceName: Co-aligned Interface Region Imaging Spectrograph (IRIS) and Solar Dynamics Observatory (SDO) Atmospheric Imaging Assembly (AIA) Observations ResourceID: spase://SMWG/Service/CCMC/Models ResourceName: CCMC Modelweb page last_modified = 2019-05-01T14:06:02+00:00
The following code demonstrates how to get a SPASE document only if it has been modified since the last time it was gotten.
if last_modified is not None:
result = hdp.get_spase(resource_ids, if_modified_since=last_modified)
if result['HttpStatus'] == 304:
print('get_spase if_modified_since ', last_modified.isoformat(),
'return Not Modified')
else:
print('get_spase if_modified_since returned an unexpected result',
result['HttpStatus'])
get_spase if_modified_since 2019-05-01T14:06:02+00:00 return Not Modified
The following code demonstrates how to view an error message from the server in response to a bad request from the client.
query = {
'ResourceID': ['spase://NASA/NumericalData/ACE/CRIS/L2/P1D'],
'Description': '~bad'
}
result = hdp.get_spase_data(types, query, time_range)
if result['HttpStatus'] != 200:
print(f'hdp.get_spase() failed with status = {result["HttpStatus"]}')
if 'ErrorMessage' in result:
print(f'ErrorMessage = {result["ErrorMessage"]}')
print(f'ErrorDescription = {result["ErrorDescription"]}')
#else:
#print(f'HttpText = {result["ErrorText"]}')
print('Display of actual HTML page returned:')
display(HTML(result['ErrorText']))
hdp.get_spase() failed with status = 400 ErrorMessage = Bad Request ErrorDescription = Invalid Description syntax. Display of actual HTML page returned:
Message: Bad Request
Description: Invalid Description syntax.
See Apache Lucene for a description of the search syntax.
Contact 📧 NASA-SPDF-Support if additional help is required.
View the hdpws API documentation for a description of additional features.