Converting 80-Column Observations to ADES Format¶
This tutorial demonstrates how to (a) convert observations from the legacy 80-column (MPC1992) format to the ADES format, and (b) how to add RA & Dec measurement uncertainties to the ADES format file produced in (a).
The 80-column format is the legacy format historically used for submitting astrometric observations to the MPC.
The ADES (Astrometry Data Exchange Standard) is the current standard for observation submissions.
One important limitation of the 80-column format is that it cannot represent RA & Dec measurement uncertainties (rmsRA and rmsDec). These fields are supported by ADES and are increasingly important for orbit determination. After converting from 80-column to ADES format, you can add these uncertainty values as a separate step.
The iau-ades Python package provides tools for:
- Converting 80-column format to ADES XML (
mpc80coltoxml) - Converting ADES XML to PSV format (
xmltopsv) - Validating ADES XML files (
valsubmit)
Further information can be found at:
Install & Import Packages¶
Here we install the iau-ades package and import the packages used in this tutorial.
! Environment note¶
This notebook installs the iau-ades package using %pip.
In Jupyter, packages must be installed into the same Python environment as the running kernel.
If you see import errors:
- Restart the kernel after installation, and
- Ensure you’re running the intended environment (e.g., virtualenv/conda).
%pip install -q iau-ades
import tempfile
import atexit
import os
import xml.etree.ElementTree as ET
from lxml import etree as lxml_etree
from ades.mpc80coltoxml import mpc80coltoxml
from ades.xmltopsv import xmltopsv
from ades.valsubmit import valsubmit
Sample 80-Column Data¶
The 80-column format uses fixed-width fields in exactly 80 characters per line. Each line encodes one observation, including the object designation, date/time, RA, Dec, magnitude, and observatory code.
Full documentation of the format can be found at https://www.minorplanetcenter.net/iau/info/OpticalObs.html
Here we create a sample file containing three observations of the same object from observatory 413 (Siding Spring Observatory).
# Three sample 80-column observation lines (each exactly 80 characters)
obs80_lines = [
"COD 310\n",
"CON N. Copernicus\n",
"OBS G. Galilei\n",
"MEA A. J. Cannon\n",
"TEL 2.00-m f/10 Ritchey-Chretien + CCD\n",
"ACK Uninformative string\n",
"NET Gaia DR2\n",
"AC2 an.email@gmail.com\n",
" K18Q99B C2016 08 19.53259 13 57 33.13 -09 12 14.3 18.4 R 310\n",
" K18Q99B C2016 08 20.52234 13 58 12.45 -09 10 22.1 18.6 R 310\n",
" K18Q99B C2016 08 21.51567 13 59 01.78 -09 08 30.5 18.5 R 310\n",
]
# Write to a temporary file
with tempfile.NamedTemporaryFile(mode='w', suffix='.obs80', delete=False) as f:
f.writelines(obs80_lines)
obs80_path = f.name
atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)
print(f"Sample 80-column file: {obs80_path}")
print(f"Each line is {len(obs80_lines[0].rstrip())} characters wide.")
print()
for line in obs80_lines:
print(line, end='')
Convert 80-Column to XML¶
The mpc80coltoxml function from the iau-ades package converts 80-column format observations to ADES XML.
Note:
- The output from
mpc80coltoxmlplaces the 80-character observations lines into the<optical>elements within the<obsData>section of the ADES file. - The shorter "header" lines describing the observer/measurer/email/etc, are placed into an
<obsContext>block at the start of the ADES file. - It also includes metadata elements (
subFmt,precTime,precRA,precDec) that are not valid in the submission schema.- We will fix these issues in the next step.
- Obviously it would be better to not have to fix these at all, so an update to
mpc80coltoxmlwould seem to be in order.- https://github.com/IAU-ADES/ADES-Master/pull/94 has been prepared & opened, allowing
mpc80coltoxml(obs80_path, raw_xml_path, submission=True), i.e. supplying asubmissionoption that omits elements such asubFmt,precTime,precRA,precDec
- https://github.com/IAU-ADES/ADES-Master/pull/94 has been prepared & opened, allowing
# Convert 80-column to XML
with tempfile.NamedTemporaryFile(suffix='.xml', delete=False) as f:
raw_xml_path = f.name
atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)
mpc80coltoxml(obs80_path, raw_xml_path)#, submission=True)
# Display the raw output
with open(raw_xml_path) as f:
print(f.read())
Post-Process for Submission¶
The raw XML output needs to be restructured before it can pass ADES submission validation:
I.e. we remove invalid elements (subFmt, precTime, precRA, precDec) that are not part of the submission schema
Below we define a helper function that performs this restructuring.
We also demonstrate that the restructured file is valid for submission to the MPC. For more details on validation, see the Validating ADES Observation Files tutorial.
def prepare_for_submission(raw_xml_path, fixed_xml_path):
"""
Restructure raw mpc80coltoxml output into valid ADES submission format.
Parameters
----------
raw_xml_path : str
Path to the raw XML from mpc80coltoxml.
fixed_xml_path : str
Path to write the restructured XML.
"""
tree = ET.parse(raw_xml_path)
root = tree.getroot()
# Clean optical elements
remove_tags = {'subFmt', 'precTime', 'precRA', 'precDec'}
for optical in root.findall('.//optical'):
for child in list(optical):
if child.tag in remove_tags:
optical.remove(child)
# Write with pretty formatting using lxml
lxml_tree = lxml_etree.ElementTree(
lxml_etree.fromstring(
ET.tostring(root, encoding='unicode')
)
)
lxml_etree.indent(lxml_tree, space=' ')
lxml_tree.write(fixed_xml_path, xml_declaration=True, encoding='UTF-8', pretty_print=True)
# Create a temp file to hold the output
with tempfile.NamedTemporaryFile(suffix='.xml', delete=False) as f:
prepared_xml_path = f.name
atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)
# Prepare the XML for submission
prepare_for_submission(raw_xml_path,prepared_xml_path)
# Demonstrate that the original was NOT valid, but that the corrected one is now valid
print("\nValidating raw_xml_path...")
valsubmit(raw_xml_path)
print("\nValidating prepared_xml_path...")
valsubmit(prepared_xml_path)
Add RA & Dec Uncertainties¶
The 80-column format has no fields for RA and Dec measurement uncertainties. In ADES, these are represented by the rmsRA and rmsDec elements (in arcseconds), which capture the random component of the astrometric uncertainty (1-sigma).
To allow the MPC to account for your uncertainties, it is very important for you to add RA & Dec uncertainties whenever you know them. It is very important that you do not create fake or invalid (too small) uncertainties to make your observations look better. No uncertainties is better than wrong ones.
After converting from 80-column format, you can add these values based on your knowledge of the measurement precision. The elements must be placed in a specific position within each <optical> element: after <dec> and before <astCat>.
Below we define a helper function that inserts rmsRA and rmsDec values into each observation.
We also demonstrate that the restructured file is not valid for submission to the MPC. For more details on validation, see the Validating ADES Observation Files tutorial.
def add_uncertainties(input_xml_path, output_xml_path, rms_ra, rms_dec):
"""
Add rmsRA and rmsDec elements to each optical observation in an ADES XML file.
Parameters
----------
input_xml_path : str
Path to input ADES XML file.
output_xml_path : str
Path to write the updated XML file.
rms_ra : float
RA uncertainty in arcseconds (e.g. 0.25).
rms_dec : float
Dec uncertainty in arcseconds (e.g. 0.20).
"""
tree = ET.parse(input_xml_path)
root = tree.getroot()
for optical in root.findall('.//optical'):
# Find the <dec> element to insert after it
children = list(optical)
dec_idx = None
for i, child in enumerate(children):
if child.tag == 'dec':
dec_idx = i
break
if dec_idx is not None:
rmsRA_elem = ET.Element('rmsRA')
rmsRA_elem.text = f"{rms_ra:0.12}"
rmsDec_elem = ET.Element('rmsDec')
rmsDec_elem.text = f"{rms_dec:0.12}"
optical.insert(dec_idx + 1, rmsRA_elem)
optical.insert(dec_idx + 2, rmsDec_elem)
# Write with pretty formatting using lxml
lxml_tree = lxml_etree.ElementTree(lxml_etree.fromstring(ET.tostring(root, encoding='unicode')))
lxml_etree.indent(lxml_tree, space=' ')
lxml_tree.write(output_xml_path, xml_declaration=True, encoding='UTF-8', pretty_print=True)
# Create a temp file to hold the output
with tempfile.NamedTemporaryFile(suffix='.xml', delete=False) as f:
final_xml_path = f.name
atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)
# Add RA & Dec uncertainties (in arcseconds)
add_uncertainties(prepared_xml_path, final_xml_path, rms_ra='0.25', rms_dec='0.20')
# Demonstrate that both prepared_xml_path & final_xml_path are valid
print("\nValidating raw_xml_path...")
valsubmit(prepared_xml_path)
print("\nValidating prepared_xml_path...")
valsubmit(final_xml_path)
with open(final_xml_path) as f:
print(f.read())
Convert to PSV Format¶
ADES observations can also be represented in PSV (Pipe-Separated Values) format, which is more human-readable than XML. The xmltopsv function converts the final XML to PSV.
The PSV output will include the rmsRA and rmsDec columns that we added above.
# Convert the final XML to PSV
with tempfile.NamedTemporaryFile(suffix='.psv', delete=False) as f:
final_psv_path = f.name
atexit.register(lambda path=f.name: os.unlink(path) if os.path.exists(path) else None)
xmltopsv(final_xml_path, final_psv_path)
with open(final_psv_path) as f:
print(f.read())
Summary¶
In this tutorial we demonstrated the full workflow for converting 80-column format observations to ADES:
- Convert 80-column to raw XML using
mpc80coltoxml - Post-process the XML to remove elements not valid in the submission schema
- Add RA & Dec uncertainties (
rmsRA,rmsDec) which cannot be represented in the 80-column format - Validate the result using
valsubmit(see Validating ADES Observation Files) - Convert to PSV format if desired using
xmltopsv
For information on submitting your ADES files to the MPC, see the Submit Observations tutorial.