Introduction
Axibase Collector is a Java-based ETL application that queries external data sources on a defined schedule and uploads the data as series, properties, messages, and files into Axibase Time Series Database.
Collector supports data markers to minimize the load on the source systems as well as list-based automation to efficiently copy data from multiple sources with re-usable configurations.
Use Cases
- Download a CSV/TSV file from a URL or an FTP server and upload the file into ATSD.
- Offload data from a relational database for long-term storage in ATSD.
- Historize metrics by querying current values and retaining their history in ATSD.
- Query a web service, convert JSON document into CSV format, and upload the file into ATSD.
- Collect performance metrics from Java applications (JMX), network devices (SNMP), etc.
- Copy data from operational support systems and plant historians for advanced analytics and visualization.
Supported Databases
- Oracle
- IBM DB2
- Microsoft SQL Server
- MySQL
- PostgreSQL
- Sybase
- Apache Derby
- Vertica
- ATSD
- SAP HANA
- OSIsoft PI Data Archive
- OSIsoft PIOleDBEnt
- Custom
Supported Data and Network Protocols
- JDBC
- SNMP
- JMX
- ICMP
- TCP
- HTTP
- MQTT
Supported File Formats
- CSV
- TSV
- Text
- JSON
Supported Schemas
Supported APIs
- Kafka
- Amazon Web Services CloudWatch
- Docker Engine
- HP OVPM (Performance Manager)
Installation
- Requirements
- Installation:
- Installation on host
- Installation in Docker container
- Installation on Kubernetes
- Administrator Account
- Collector account in ATSD
- ATSD Server Connection
Jobs
Automation
Job Types
| Type | Description |
|---|---|
| AWS | Collects AWS statistics using CloudWatch API. |
| Docker | Collects container, image, and volume statistics using Docker Engine API. |
| File | Downloads CSV/TSV files from remote servers or local file system and uploads them into ATSD for parsing. Supports HTTP/s, FTP, SFTP, SCP, and FILE protocols. |
| HTTP | Executes HTTP requests or Web Driver scripts and stores response status, response times, and SSL certificate status as metrics. |
| ICMP | Pings hostnames/IP addresses and stores response status. |
| JDBC | Executes SQL queries against relational databases. Converts rows into series, property, or message commands. |
| JMX | Collects MBean attribute values from Java applications. |
| JSON | Downloads JSON files, parses the documents and converts JSON fields into series, property, and message commands using JSONPath. |
| Kafka | Reads JSON messages from Kafka brokers, parses the messages and converts them into series, property, and message commands. |
| MQTT | Reads JSON, CSV or plain messages from MQTT brokers, parses the messages and converts them into networks commands. |
| OVPM | Offloads statistics from HP OpenView Performance Manager. |
| PI | Extracts PI points archive data via JDBC driver. |
| SNMP | Queries SNMP devices using built-in and custom MIB files. |
| Socrata | Downloads JSON documents published in Socrata schema and converts them into series, property, and message commands. |
| TCP | Connects to hostnames/IP addresses and stores connection status. |
Administration
Examples
| Name | Job Type | Description |
|---|---|---|
| ActiveMQ | JMX | Collect metrics about brokers, queues, pub/sub topics |
| Derby Database | JMX | Collect uptime metrics for the database |
| File | File | Download CSV files |
| HP OpenView | OVPM | Offload CODA metrics from OVPM |
| Jetty | JMX | Collect sessions, requests, status from the Jetty server |
| JSON | JSON | Download JSON file and convert the file to CSV format |
| JVM | JMX | Collect key JVM performance metrics |
| MySQL | JDBC | Collect database performance metrics |
| nginx | File | Collect key web server metrics for nginx |
| nginx-plus | File | Collect extended web server metrics for nginx-plus |
| Oracle EM | JDBC | Offload incremental database and application metrics collected by Oracle EM |
| PI | PI | Copy incremental tag values from PI Data Archive |
| PostgreSQL | JDBC | Collect database performance metrics |
| SCOM | JDBC | Offload incremental server metrics collected by SCOM |
| Socrata | Socrata | Download and parse datasets published in Socrata format |
| SolarWinds | JDBC | Offload incremental network and server metrics collected by SolarWinds |
| Tomcat | JMX | Collect key container metrics exposed by Tomcat |
| VMware | JDBC | Offload incremental cluster, host, VM metrics collected by VMware vCenter |