Skip to content

MaatDex

Overview

MaatDex is a Python application designed to fetch and process network topology data from Maat. The application supports saving the fetched data in JSON or CSV format.

Installation

  1. Clone the repository: 

    git clone <repository_url>
cd project

  2. Create a virtual environment and activate it: 

    python -m venv venv
source venv/bin/activate  # On Windows use `venv\Scripts\activate`

  3. Install the required dependencies: 

    pip install -r requirements.txt

Configuration

The configuration file /config.conf contains settings for the API, output paths, and logging. Update this file with the appropriate values for your environment.

Usage

To fetch and process network core link data, run the fetch_data.py script with the required arguments: 

python scripts/fetch_data.py -a <address> -o <output_file> -f <format>

Arguments

Required Arguments:

  • -a, --address: The address of the Maat API.
  • -o, --output: The path to the output file.
  • -f, --format: The output format (json or csv).

Optional Arguments:

  • --modified: Modifies coordinates of overlapping nodes
  • --only_nodes: Fetches only nodes with geo-coordinates
  • --nmp_database: Enables fetching traffic metrics for NMP (for example VictoriaMetrics) for devif and enriches output with in_bits and out_bits. Requires nmp_database section in config.conf.
  • --nmp_topology: Saves an additional NMP topology file.
  • --cat_links: Category name for links.
  • --cat_sap: Category name for service access points.
  • --cat_sap_interfaces: Category name for service access point interfaces.
  • --cat_interfaces: Category name for interfaces.
  • --cat_nodes: Category name for nodes.
  • --cat_sites: Category name for sites.

Usage Examples

Basic topology fetching in JSON format: 

python scripts/fetch_data.py -a https://maat.example.com/api -o topology.json -f json

Fetching with coordinate modification: 

python scripts/fetch_data.py -a https://maat.example.com/api -o topology.json -f json --modified

Fetching only GPS nodes: 

python scripts/fetch_data.py -a https://maat.example.com/api -o nodes.csv -f csv --only_nodes

Saving in CSV format with custom categories: 

python scripts/fetch_data.py -a https://maat.example.com/api -o data.csv -f csv --cat_nodes custom_nodes --cat_links custom_links

Saving Network Map Panel topology: 

python scripts/fetch_data.py -a https://maat.example.com/api -f json -o export_data/network_map_panel/topology_for_nmp.json --nmp_topology

Saving Network Map Panel measurements to json format: 

python scripts/fetch_data.py -a https://maat.example.com/api -f json -o export_data/network_map_panel/database_for_nmp.json --nmp_database

Application Architecture

Directory Structure

project/
├── api/
│   └── api_client.py               # API client for Maat communication
├── config/
│   ├── config_loader.py            # Configuration loading
│   └── log_config.py               # Logging configuration
├── scripts/
│   ├── fetch_data.py               # Main data fetching script
│   ├── process_data.py             # Data processing and normalization
│   └── modify_data.py              # Data modification
├── utils/
│   ├── file_utils.py               # File saving utilities
│   └── metrics_client.py           # NMP/VictoriaMetrics client
├── export_data/                    # Directory for exported data
│   └── additional_topologies/      # Generated city-based topologies
├── doc/
│   └── images/
│       └── mapgl_data_diagram.png  # Data model diagram used in README
├── config.conf                     # Configuration file
├── requirements.txt                # Python dependencies
└── README.md

Modules

  • api/api_client.py Handles Maat API communication, connection management, and error handling.
  • scripts/fetch_data.py
  • main(): Orchestrates fetching, processing, enrichment, and saving.
  • parse_args(): Parses CLI arguments including --nmp_database and --nmp_topology.
  • scripts/process_data.py
  • fetch_network_core_link(): Fetches network link data
  • fetch_nodes_with_gps(): Fetches nodes with GPS coordinates
  • modify_coordinates(): Modifies coordinates of overlapping elements
  • normalize_polish_chars(): Normalizes Polish diacritical characters
  • scripts/modify_data.py
  • checking_vertexb(): Verifies and corrects vertex data
  • checking_vertexb_for_cities(): Checks vertex data for city grouping
  • utils/file_utils.py
  • save_as_json(): Saves data in JSON format
  • save_as_csv(): Saves data in CSV format
  • group_connections(): Groups connections by cities
  • save_grouped_topologies(): Saves grouped topologies
  • enrich_with_traffic_data(): Enriches items with NMP metrics (in_bits, out_bits) by devif
  • utils/metrics_client.py
  • class MetricsClient: VictoriaMetrics client for NMP integration
  • query_metric(): Executes PromQL query for a given metric and devif
  • get_traffic_data(): Returns current in_bits/out_bits for an interface
  • config/config_loader.py Loads application configuration (includes nmp_database section)
  • config/log_config.py Configures application logging

Output Formats

JSON Format

Data is saved as an array of objects with the following fields:

{
    "vertexA": "device-1",
    "vertexB": "device-2",
    "edgeId": "12345678-1234-1234-abcd-abc123def456",
    "lon": 16.92516810000000,
    "lat": 52.40637400000000,
    "devif": "device-1 et-0/0/0",
    "bandwidth": 100000000000,
    "thickness": 5,
    "description": "Network core link",
    "city": "Poznań",
    "link": "device-1 <-> device-2",
    "dst_lng": 19.455020577406692,
    "dst_lat": 51.75439257919774
}

CSV Format

CSV columns correspond to JSON fields, with comma separator.

City-Grouped Topologies

The application automatically generates separate JSON files for each city in the export_data/additional_topologies/ directory.

Special Features

  • Coordinate Modification: The --modified option allows automatic separation of overlapping nodes by slightly shifting their geographical coordinates.
  • Polish Character Normalization: Automatic conversion of Polish diacritical characters to ASCII equivalents for better compatibility.
  • City-Based Grouping: Automatic grouping of connections by cities and generation of separate topology files.

Logging

The application uses a logging system with configurable levels. Logs include:

  • Data fetching progress information
  • API connection errors
  • Data issue warnings
  • Processed record statistics

Maat Data Model for MaatDex

This section describes the required data model that must exist in the Maat application for proper operation of MaatDex. This model is utilized by MaatDex to export the necessary data for creating network topology visualizations in the Mapgl plugin. The model defines the structure of network elements, their relationships, and the specific requirements for data extraction and processing.

Mapgl Data Diagram

  1. Link Category (Service)
  2. Type: Service
  3. Category: network.core.link
  4. Description: Main object representing network connection between devices
  5. SAP Structure: Link MUST contain exactly 2 SAPs in specific order:
    • serviceRelationship[0] = SAP1 (first geographical point)
    • serviceRelationship[1] = SAP2 (second geographical point)
  6. Example required relationship: relationshipType = "ref:network.core.sap"
  7. Order requirement: For link "Poznan - Warszawa":
    • SAP1 → device in Poznan
    • SAP2 → device in Warszawa
  8. Service Access Points SAP (Service)
  9. Type: Service
  10. Category: network.core.sap
  11. Order: Maintained according to geographical order of the link
  12. Example required relationship: relationshipType = "bref:interface"
  13. Relationship: Each SAP has resourceRelationship to interface
  14. Physical Interfaces (Resource)
  15. Type: Resource
  16. Category: interface
  17. @type: PhysicalResource
  18. Example required relationship: relationshipType = "bref:device.router"
  19. Relationship with SAP: Each SAP has backward reference to interface
  20. Speed characteristics: resourceCharacteristic[name="speed"] in format "100G", "400G", "10G" ONLY format with "G" (not mb, gb, GB)
  21. Logical/Aggregated Interfaces (Resource)
  22. Type: Resource
  23. Category: interface
  24. @type: LogicalResource
  25. Example required relationship: relationshipType = "bref:device.router"
  26. Special relationships: Aggregated interface has resourceRelationship to two PhysicalResource
  27. Bandwidth calculation: Sum of speeds of two physical interfaces
  28. Example: 
    LogicalResource (ae0) → relationshipType: cat_interfaces
├── PhysicalResource (et-0/0/0) → speed: "100G"
└── PhysicalResource (et-0/0/1) → speed: "100G"
Result: bandwidth = 200G
  29. Network Devices (Resource)
  30. Type: Resource
  31. Category: device.router
  32. Example required relationship: relationshipType = "bref:site"
  33. Relationship with interface: Each interface has reference to device
  34. Geographical relationship: Each device MUST have reference to location
  35. Site (Resource)
  36. Type: Resource
  37. Category: site
  38. Relationship with device: Each device has reference to location
  39. Required characteristics:
    • gps - coordinates in format "latitude;longitude"
    • city - city name

Required Relationships Map

  • Link → SAP: relationshipType = "ref:network.core.sap"
  • SAP → Interface: relationshipType = "bref:interface"
  • Interface → Device: relationshipType = "bref:device.router"
  • Device → Site: relationshipType = "bref:site"

This model ensures full geographical consistency, correct bandwidth calculation for aggregated interfaces, and preservation of point order in network topology with clear definitions of required relationships.