Validating ADES Observation Files¶

This tutorial demonstrates how to use the iau-ades Python package to validate ADES-formatted observation files locally, before submitting them to the Minor Planet Center.¶

The ADES (Astrometry Data Exchange Standard) is the standard format for submitting astrometric observations to the MPC. Observations can be formatted as either XML or PSV (Pipe-Separated Values).

The iau-ades Python package provides tools to:

• Validate ADES XML files against the submission schema (submit.xsd)
• Convert between PSV and XML formats

In this tutorial we focus on the valsubmit function that can be used to assess the validity of files that are to be submitted to the MPC. For users interested in the more general valgeneral function, please see the iau-ades documentation.

Validating your ADES files locally before submission is a useful step that can help identify formatting errors early, saving time in the submission process.

Further information and documentation on the ADES standard can be found at:

• https://www.minorplanetcenter.net/mpcops/documentation/ades/
• https://github.com/IAU-ADES/ADES-Master
• https://www.minorplanetcenter.net/mpcops/documentation/valid-ades-values/

Install & Import Packages¶

Here we install the iau-ades package and import the standard Python packages we use in this tutorial.

In [ ]:

!pip install -q iau-ades requests

!pip install -q iau-ades requests

In [ ]:

import requests
import tempfile
import atexit
import os
import io
import contextlib
from dataclasses import dataclass
from ades.valsubmit import valsubmit
from ades.psvtoxml import psvtoxml

import requests import tempfile import atexit import os import io import contextlib from dataclasses import dataclass from ades.valsubmit import valsubmit from ades.psvtoxml import psvtoxml

Helper Function¶

The valsubmit routine will (a) print its result to std.out (b) save its results to a file valsubmit.file in the current working directory.

Your opinion on the appeal of this approach may vary.

Here we define a small helper class that calls valsubmit, optionally suppresses the print-out & removes the created file, and parses the results into class variables.

In [ ]:

@dataclass
class ValidateXML:
    """
    Validate an ADES XML file and capture the result.

    The `valsubmit` routine will
    (a) print its result to std.out
    (b) save its results to a file `valsubmit.file`

    Your opinion on the appeal of this approach may vary. 

    This helper function will 
    (i) capture the result (into a variable)
    (ii) optionally suppress the print statement [defaults to NOT suppressing]
    (iii) optionally remove the results file [defaults to removing]
    
    Parameters
    ----------
    xml_filepath : str
        Path to the XML file to validate.
    """
    xml_filepath: str
    remove_fileoutput: bool = True 
    suppress_print: bool = False

    def __post_init__(self):  # Called automatically after __init__ to perform post-processing.
        self._validatexml()

    def _validatexml(self,):
        
        captured_output = io.StringIO()  # Create a StringIO object
        with contextlib.redirect_stdout(captured_output):  # Use the context manager to redirect stdout
            valsubmit(self.xml_filepath)  # Run the validation
            self.valid = True if "submit is OK" in captured_output.getvalue() else False  # Populate boolean
            self.error = None if self.valid else captured_output.getvalue()  # Capture any error 

        # Optionally remove 'valsubmit.file'
        outputfile = 'valsubmit.file'
        if self.remove_fileoutput and os.path.isfile(outputfile):
            os.remove(outputfile)

        # Optionally print
        if not self.suppress_print:
            print(captured_output.getvalue())

@dataclass class ValidateXML: """ Validate an ADES XML file and capture the result. The `valsubmit` routine will (a) print its result to std.out (b) save its results to a file `valsubmit.file` Your opinion on the appeal of this approach may vary. This helper function will (i) capture the result (into a variable) (ii) optionally suppress the print statement [defaults to NOT suppressing] (iii) optionally remove the results file [defaults to removing] Parameters ---------- xml_filepath : str Path to the XML file to validate. """ xml_filepath: str remove_fileoutput: bool = True suppress_print: bool = False def __post_init__(self): # Called automatically after __init__ to perform post-processing. self._validatexml() def _validatexml(self,): captured_output = io.StringIO() # Create a StringIO object with contextlib.redirect_stdout(captured_output): # Use the context manager to redirect stdout valsubmit(self.xml_filepath) # Run the validation self.valid = True if "submit is OK" in captured_output.getvalue() else False # Populate boolean self.error = None if self.valid else captured_output.getvalue() # Capture any error # Optionally remove 'valsubmit.file' outputfile = 'valsubmit.file' if self.remove_fileoutput and os.path.isfile(outputfile): os.remove(outputfile) # Optionally print if not self.suppress_print: print(captured_output.getvalue())

Sample ADES Files¶

Here we download sample XML and PSV files from the MPC's website and save them to local (temporary) files.

These files are known to be valid submissions to the MPC.

If you have created your own ADES file(s), then this download of sample files is not necessary.

The local, temporary files will be deleted after this notebook exits.

Note: The sample files from the MPC use ADES version 2017. The current iau-ades package validates against the version 2022 schema. Below, we update the version and fix any fields that have changed between versions.

In [ ]:

# Define the files to fetch
mpc_urls = [
    "https://data.minorplanetcenter.net/media/ades/goodsubmit.xml.txt",
    "https://data.minorplanetcenter.net/media/ades/goodsubmit.psv.txt"
]

local_filenames = ["sample.xml", "sample.psv"]
temp_filepaths = {}

for local_filename, mpc_url in zip(local_filenames, mpc_urls):
    content = requests.get(mpc_url).text

    # Update from version 2017 to 2022
    content = content.replace('version="2017"', 'version="2022"').replace('version=2017', 'version=2022')
    # In v2022, RA values must not have a leading '+' sign
    content = content.replace('<ra>+', '<ra>').replace('<dec>+', '<dec>')
    content = content.replace('| +0.1      |+30.0', '|  0.1      | 30.0')

    ext = local_filename[-3:]  # 'xml' or 'psv'
    with tempfile.NamedTemporaryFile(mode='w', suffix='.' + ext, delete=False) as f:
        f.write(content)
        temp_filepaths[ext] = f.name
        print(f"Saved {ext.upper()} sample to: {temp_filepaths[ext]}")
        atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)

# Define the files to fetch mpc_urls = [ "https://data.minorplanetcenter.net/media/ades/goodsubmit.xml.txt", "https://data.minorplanetcenter.net/media/ades/goodsubmit.psv.txt" ] local_filenames = ["sample.xml", "sample.psv"] temp_filepaths = {} for local_filename, mpc_url in zip(local_filenames, mpc_urls): content = requests.get(mpc_url).text # Update from version 2017 to 2022 content = content.replace('version="2017"', 'version="2022"').replace('version=2017', 'version=2022') # In v2022, RA values must not have a leading '+' sign content = content.replace('+', '').replace('+', '') content = content.replace('| +0.1 |+30.0', '| 0.1 | 30.0') ext = local_filename[-3:] # 'xml' or 'psv' with tempfile.NamedTemporaryFile(mode='w', suffix='.' + ext, delete=False) as f: f.write(content) temp_filepaths[ext] = f.name print(f"Saved {ext.upper()} sample to: {temp_filepaths[ext]}") atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)

Validate XML Format Data¶

The iau-ades package validates ADES files in XML format. The valsubmit function checks an XML file against the ADES submission schema (submit.xsd).

XML Validation: Successful¶

Here we validate the sample XML file downloaded above. Since this file contains correctly formatted ADES data, the validation should pass.

In [ ]:

res = ValidateXML(temp_filepaths['xml'], suppress_print=True)
print(f"{res.valid=}")
if not res.valid: 
    print(f"{res.error=}")

res = ValidateXML(temp_filepaths['xml'], suppress_print=True) print(f"{res.valid=}") if not res.valid: print(f"{res.error=}")

XML Validation: Failed (RA out of range)¶

Here we create an intentionally invalid XML file where the Right Ascension (ra) value is set to 999.999, which exceeds the allowed maximum of 360.0 degrees. The validation should fail and report the error.

In [ ]:

# Read the good XML and introduce an error: set RA to an invalid value
with open(temp_filepaths['xml']) as f:
    good_xml = f.read()

bad_xml = good_xml.replace('<ra>0</ra>', '<ra>999.999</ra>', 1)

# Write the bad XML to a temporary file
with tempfile.NamedTemporaryFile(mode='w', suffix='.xml', delete=False) as f:
    f.write(bad_xml)
    bad_xml_path = f.name
    atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)

# Validate the bad XML
res = ValidateXML(bad_xml_path, suppress_print=False)
print(f"{res.valid=}")

# Read the good XML and introduce an error: set RA to an invalid value with open(temp_filepaths['xml']) as f: good_xml = f.read() bad_xml = good_xml.replace('0', '999.999', 1) # Write the bad XML to a temporary file with tempfile.NamedTemporaryFile(mode='w', suffix='.xml', delete=False) as f: f.write(bad_xml) bad_xml_path = f.name atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None) # Validate the bad XML res = ValidateXML(bad_xml_path, suppress_print=False) print(f"{res.valid=}")

XML Validation: Failed (missing required element)¶

Here we create another invalid XML file where the required stn (station/observatory code) element has been removed from an observation. The validation should fail because stn is a required field.

In [ ]:

# Remove the <stn>F51</stn> element from the first observation
bad_xml_nostn = good_xml.replace('        <stn>F51</stn>\n', '', 1)

# Write the bad XML to a temporary file
with tempfile.NamedTemporaryFile(mode='w', suffix='.xml', delete=False) as f:
    f.write(bad_xml_nostn)
    bad_xml_nostn_path = f.name
    atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)

# Validate the bad XML
res = ValidateXML(bad_xml_nostn_path, suppress_print=False)

# Remove the F51 element from the first observation bad_xml_nostn = good_xml.replace(' F51\n', '', 1) # Write the bad XML to a temporary file with tempfile.NamedTemporaryFile(mode='w', suffix='.xml', delete=False) as f: f.write(bad_xml_nostn) bad_xml_nostn_path = f.name atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None) # Validate the bad XML res = ValidateXML(bad_xml_nostn_path, suppress_print=False)

Validate PSV Format Data¶

The iau-ades validation functions operate on XML format. If your observations are in PSV (Pipe-Separated Values) format, they must first be converted to XML before validation can be performed.

The iau-ades package provides the psvtoxml function for this conversion. This mirrors what the MPC does internally when PSV files are submitted.

PSV Validation: Successful¶

Here we demonstrate the two-step process:

1. Convert the PSV file to XML using psvtoxml
2. Validate the resulting XML using valsubmit

In [ ]:

#Step 1: Convert PSV to XML
with tempfile.NamedTemporaryFile(suffix='.xml', delete=False) as f:
    xml_from_psv_path = f.name
    atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)

psvtoxml(temp_filepaths['psv'], xml_from_psv_path)


# # Step 2: Validate the converted XML
res = ValidateXML(xml_from_psv_path, suppress_print=False)

#Step 1: Convert PSV to XML with tempfile.NamedTemporaryFile(suffix='.xml', delete=False) as f: xml_from_psv_path = f.name atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None) psvtoxml(temp_filepaths['psv'], xml_from_psv_path) # # Step 2: Validate the converted XML res = ValidateXML(xml_from_psv_path, suppress_print=False)

PSV Validation: Failed¶

Here we create an invalid PSV file where one observation has a Declination (dec) value of 91 (exceeding the allowed maximum of 90.0 degrees). We convert it to XML and then validate to show that the error is caught.

Note: The psvtoxml function uses module-level state that is not reset between calls. We use importlib.reload to reset this state before the second conversion.

In [ ]:

# Reset psvtoxml module state (required when calling psvtoxml multiple times)
import importlib
import ades.psvtoxml
importlib.reload(ades.psvtoxml)
from ades.psvtoxml import psvtoxml

# Read the good PSV and introduce an error: set Dec to an invalid value
with open(temp_filepaths['psv']) as f:
    good_psv = f.read()

bad_psv = good_psv.replace('| 90        |', '| 91        |', 1)

# Write the bad PSV to a temporary file
with tempfile.NamedTemporaryFile(mode='w', suffix='.psv', delete=False) as f:
    f.write(bad_psv)
    bad_psv_path = f.name
    atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)

# Step 1: Convert bad PSV to XML
with tempfile.NamedTemporaryFile(suffix='.xml', delete=False) as f:
    bad_xml_from_psv_path = f.name
    atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)

psvtoxml(bad_psv_path, bad_xml_from_psv_path)
print("PSV converted to XML successfully (conversion does not validate content).")

# Step 2: Validate the converted XML
res = ValidateXML(bad_xml_from_psv_path, suppress_print=True)
print(f"{res.valid=}")
if not res.valid: 
    print(f"{res.error}")

# Reset psvtoxml module state (required when calling psvtoxml multiple times) import importlib import ades.psvtoxml importlib.reload(ades.psvtoxml) from ades.psvtoxml import psvtoxml # Read the good PSV and introduce an error: set Dec to an invalid value with open(temp_filepaths['psv']) as f: good_psv = f.read() bad_psv = good_psv.replace('| 90 |', '| 91 |', 1) # Write the bad PSV to a temporary file with tempfile.NamedTemporaryFile(mode='w', suffix='.psv', delete=False) as f: f.write(bad_psv) bad_psv_path = f.name atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None) # Step 1: Convert bad PSV to XML with tempfile.NamedTemporaryFile(suffix='.xml', delete=False) as f: bad_xml_from_psv_path = f.name atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None) psvtoxml(bad_psv_path, bad_xml_from_psv_path) print("PSV converted to XML successfully (conversion does not validate content).") # Step 2: Validate the converted XML res = ValidateXML(bad_xml_from_psv_path, suppress_print=True) print(f"{res.valid=}") if not res.valid: print(f"{res.error}")

Summary¶

In this tutorial we demonstrated how to use the iau-ades Python package to validate ADES-formatted observation files locally:

• XML files can be validated directly using valsubmit from ades.valsubmit
• PSV files must first be converted to XML using psvtoxml from ades.psvtoxml, and then validated
• Validation checks the data against the ADES submission schema (submit.xsd) and reports errors such as out-of-range values or missing required elements

We recommend validating your ADES files locally before submitting them to the MPC. For information on how to submit validated files to the MPC, see the Submit Observations tutorial.