Data Migration from Graphite to ATSD

Use the migrate.py utility to copy data from a Graphite server, specifically from its Whisper database, into ATSD.

The migrate.py utility extracts stored historical data from Whisper files and transfers it in Graphite format into ATSD.

Download the migrate.py utility here.

Below are the available configurations and examples.

migrate.py supports only Whisper files (.wsp).

Data sent into ATSD by the migrate.py utility is parsed according to rules specified in the graphite.conf file.

Using migrate.py:

migrate.py [-h] [--whisper-base BASE] [-R] path atsd_hostname atsd_tcp_port

migrate.py settings

Setting Required Description
-h OR --help no Show help message and exit.
path yes Path to folder with .wsp files or .wsp file that is exported to ATSD.
Path must be specified either:
– directly to the .wsp file (without -R setting)
OR
– to the folder containing .wsp files (with -R setting).
Note that the ~ symbol cannot be used when specifying path.
-R no Export recursively all files in the specified folder; searches are sub folders and directories for .wsp files.
If -R is not specified then you must specify the direct path to the .wsp file.
--whisper-base BASE no Base path to which all metric names are resolved.
Recommended to set Whisper base directory.
Default value: . (current directory).
atsd_hostname yes ATSD hostname or IP.
atsd_tcp_port yes ATSD TCP listening port. Default ATSD TCP port is 8081.

Examples

Base path to the Whisper database directory is set with -R to migrate all the data and metrics.

Command:

./migrate.py -R --whisper-root=/var/lib/graphite/whisper/ /var/lib/graphite/whisper/ atsd_hostname 8081

Messages sent to ATSD:

carbon.agents.NURSWGDKR002-a.avgUpdateTime 9.41753387451e-05 1436859240
carbon.agents.NURSWGDKR002-a.avgUpdateTime 9.3019925631e-05 1436859300
carbon.agents.NURSWGDKR002-a.avgUpdateTime 9.33683835543e-05 1436859360
...
collectd.NURSWGDKR002.users.users 4.0 1436878560
collectd.NURSWGDKR002.users.users 4.0 1436878620
collectd.NURSWGDKR002.users.users 4.0 1436878680

Direct path to a specific .wsp file is set without -R to migrate only the contained metric.

Command:

./migrate.py --whisper-root=/opt/graphite/whisper/ /opt/graphite/whisper/collectd/NURSWGDKR002/memory/memory-free.wsp atsd_hostname 8081

Messages sent to ATSD:

collectd.NURSWGDKR002.memory.memory-free 31467552768.0 1436867280
collectd.NURSWGDKR002.memory.memory-free 31480631296.0 1436867340
collectd.NURSWGDKR002.memory.memory-free 31409938432.0 1436867400
collectd.NURSWGDKR002.memory.memory-free 31384133632.0 1436867460

Test Data Migration

Launch nc in server mode to record incoming commands to the test.txt file.

nc -lk 8081 > test.txt &

Substitute --whisper-base, the path, atsd_hostname, and atsd_tcp_port with actual parameters and launch the migration script.

date +%s && ./migrate.py -R --whisper-base=/var/lib/graphite/whisper/ /var/lib/graphite/whisper/ atsd_hostname atsd_tcp_port && date +%s

The resulting test.txt file contains all metrics migrated by the migrate.py script.

9 seconds
730236 lines
49246203 bytes