introduction

In the world of radio frequency (RF) analysis, managing and interpreting large datasets can be a daunting task. Recently, I found myself faced with such a challenge, requiring the aggregation of huge datasets of RF frequencies, exported from ROVER (a geo-spatial mapping tool). The need for a tool that could efficiently parse, round, and group this frequency data led me to develop the ROVER-KML-Frequency-Aggregator.

The problem

Rover data is often exported as KML (Keyhole Markup Language). However, these files can become bloated and cumbersome to analyse due to numerous issues:

1. Slight variations in the frequency values aren’t considered the same when imported
2. The frequency list couldn’t be sorted against a known frequency list to identify outliers
3. The lack of any tool that could easily re-process this data with transformations

Due to the size of the datasets, manual processing of these files is functionally impossible, being too time-consuming and error-prone. Data integrity wasn’t necessary for this project, so transformations could be made without worry.

the solution

The ROVER-KML-Frequency-Aggregator is a Python script, utilising Streamlit as a UI. It addresses the above issues by providing:

1. Automatic parsing and rounding of frequency data
2. Aggregation of data point from the same rounded frequency
3. Optional second-pass filtering against a JRFL (Joint Restricted Frequency List)
4. An opertor-centric interface for file upload and configuration
5. Direct download of the processed file

This tool is designed to be extremely simple and accessible to non-technical users, whilst still being effective.

technical deep dive

The main component of this project is the “parse_kml()” function. Lets take a look:

1. XML Parsing: Using the “xml.etree.ElementTree” library, parsing of the KML file is handled in the same fashion as an XML file
2. Frequency Rounding: The “round_frequency” function handles the rounding of the frequency values to the user-specified number of decimal places
3. Folder Aggregation: Iterating through the KML folders, they are grouped based on their rounded frequency values. Duplicate folders are removed, preserving the LOB (Line of Bearing) data in the new folder
4. Name Updating: The “update_all_names()” function ensures that all references to a freq is updated to its newly rounded value
5. CSV Filtering: If provided, the remaining data is filtered against a CSV file to remove any conflicts

A key aspect of this implementation is the preservation of the original KML format, ensuring it’s ability to be re-imported. Additionally, high-value data (LOB Data) is also preserved, being consolidated into the rounded folders.

code showcase

Here is a section of the frequency rounding and folder aggregation logic:

def round_frequency(freq_str, decimal_places):
    try:
        freq = float(freq_str)
        rounded_freq = round(freq * (10 ** decimal_places)) / (10 ** decimal_places)
        return f"{rounded_freq:.{decimal_places}f}"
    except ValueError:
        return freq_str

# ... (in parse_kml function)
for folder in folders:
    name = folder.find('{http://www.opengis.net/kml/2.2}name')
    if name is not None and "MHz" in name.text:
        original_freq = re.search(r'(\d+\.\d+)', name.text).group(1)
        rounded_freq = round_frequency(original_freq, decimal_places)

        if rounded_freq not in folder_dict:
            folder_dict[rounded_freq] = folder
            update_all_names(folder, original_freq, rounded_freq)
        else:
            # Aggregate data into existing folder
            # ...

In this snippet, you can see how the frequency data is extracted, rounded, and placed into folders.

challenges faced

A key issue that was run into during the development of this project was the handling of KML files, especially regarding the preservation of some data (like LOBs) and deletion of others (Like duplicated frequencies). The crux of the problem actually lied with the re-importation of the data back into ROVER, and ensuring the data structure remained the same, as any changes with break the importing process.

Additionally, a key consideration was user-friendliness. The tool was originally CLI-based as I had no issues with using it this way. However, when I tried show my team how to use it, it proved to be potentially either too complicated or slow to use, thus developing the need for a more friendly UX. Having worked with Streamlit on previous projects, I decided this was a simple UI, that still included all of the key features required for the project.

Future improvements

Whilst the current implementation is functional, it is hyper-specific, and has much room for improvement:

1. Supporting other export types or other programs
2. Implementing more advanced filtering options
3. Add the ability to select the persisted data instead of it being hard-coded to only-ever include LOBs
4. Optimise for larger input files or lower-end systems

conclusion

Developing the ROVER-KML-Frequency-Aggregator was an excellent exercise in how simple scripts can dramatically change a workflow, optimising a several-hour-long timeline of identification, classification, and restructuring, into a few seconds. Whether you’re a RF engineer working with ROVER, or just performing XML processing, this tool is an extremely valuable resource. Check out this project on GitHub, Give it a go and a star, if you find and problems, make an issue. Or, if you have any improvements, make a pull request