# Storage Driver Configuration

# Overview

Data retrieved by Axibase Collector from external data sources is persisted by Collector in one or multiple ATSD databases, configured on the Admin > Storage Drivers page.

By default, all jobs running in Collector transmit data into the same ATSD instance using a single shared storage driver.

The storage driver sends data into ATSD over HTTP protocol.

Via advanced configuration, each job can store data in a specific ATSD instance by using multiple storage drivers.

# Requirements

To store data in a specific ATSD instance, Collector uses a restricted user account defined in the target ATSD instance and authorized to write data and metadata for all or specific entities.

Create a Collector account on the Settings > User Accounts page in the ATSD web interface.

# Initial Configuration

If no storage driver is configured for an ATSD instance, the database prompts you to create one upon login.

• Specify hostname or IP address of the target ATSD server.
• Specify https service port address, default port is 8443.
• Enter Collector account credentials.
• Click Validate to verify settings. In case of permission errors, ensure that specified Collector account has both API_DATA_WRITE and API_META_WRITE roles.
• Upon successful validation, click Save to create the storage driver. Collector automatically creates an HTTP connection pool and storage driver based on submitted settings.

# Storage Driver

Storage drivers are listed on the Admin > Storage Drivers page.

To create a new storage driver, open the Storage Drivers page and click Add.

Choose an existing HTTP connection pool from the HTTP Pool drop-down list or create a new pool by clicking .

For a high-availability configuration, specify a failover driver to deliver commands to a secondary ATSD database while the primary database is not available.

Click Test to verify settings.

If Collector contains multiple storage drivers, each unique job can be configured to send collected data into a specific ATSD instance.

# HTTP Pool

Collector transmits data into ATSD via HTTP protocol. HTTP connection pool settings contain limits and timeouts which apply to HTTP connections initiated by Collector when sending data into ATSD.

# Create a New Pool

• Open the Data Sources > HTTP Pools page from the top menu and click Add.
• Enter a pool name.
• Enter hostname or IP address of the target ATSD server.
• Specify port 8443 and select https protocol.
• Clear Ignore SSL Certificate Errors if the target ATSD SSL Certificate is trusted (not self-signed).
• Set timeouts and limits based on the recommended settings below.
• Select Authentication Type=BASIC, enter Collector account credentials.
• Set Test Path to /ping.
• Click Test to verify settings.

If the settings are valid, the test returns 200 OK status.

In case of 4xx status codes, refer to ATSD Authentication Codes for troubleshooting.

Sample HTTP Pool used by storage driver:

# Recommended Pool Settings

Setting	Value
Max Connections	16
Use Cookies	false
Connection Pool Timeout	15
Connect Timeout	15
Close Idle Connections interval	60
Read Timeout	15
Socket Linger	0
Socket Reuse	true
Socket Keep-Alive	true

# Delivery

Collector implements a file-based cache to temporarily store commands that are not delivered to the target database due to network and other errors.

The pending commands are retained for the duration of time specified in the Command Retention Interval setting on the Admin > Application Settings page.

Name	Description
Command Retention Interval	Interval after which failed commands are deleted from file-based cache. Default: 86400 seconds (1 day). Minimum: 60 seconds. Must exceed the Failover Timeout. Restart is required to apply changes.

# Failover

Collector supports an automated failover mechanism to ensure uninterrupted data collection.

If the primary database becomes unavailable and the failover driver is specified, Collector switches to the secondary database. While sending data to the secondary database, Collector periodically checks the status of the primary database and switches back once the connectivity is re-established.

The following steps describe the switch-back procedure:

• Collector determines that the current database to which there is a connection is a secondary database.
• Collector starts a background periodic task to re-connect to the primary database.
• The default periodic re-connect interval is 300 seconds.
• The periodic re-connect task attempts to connect to the primary server. This task continues until the primary database is online or Collector shuts down.
• If the primary database is available, Collector initiates a switch-back procedure:
  • Sends a message to the secondary database that Collector is now connecting to the primary database.
  • Sets the primary database as the current database.
  • Establishes a connection to the primary database.
  • If the above connection is successful, the periodic re-connect task is stopped.

# Failover Parameters

Name	Description
Failover Timeout	Duration of time, starting with the first error, after which the driver switches from the primary to the secondary database. Default: 300 seconds. Minimum: 15 seconds.
Failover Switchback Interval	Interval for checking availability of the primary storage driver. Activated when sending command to the secondary driver in failover mode. Default: 300 seconds. Minimum: 60 seconds. Restart is required to apply changes.

The failover timeout can be specified for each storage driver separately whereas the switch-back interval applies to all drivers and is specified on the Admin > Application Settings page.

# Failover Logging

The switch-back procedure states are logged with the following messages, each containing message tags primary_driver_url and secondary_driver_url.

• Collector fails to send commands to the primary driver and attempts to switch to the secondary driver.

  Failed to send commands to the primary driver. Switching to secondary.
  

• Secondary driver is not available.

  Failed to send commands to the primary driver. Failed to connect to the secondary driver. Continue using primary driver.
  

• Secondary driver is available.

  Failed to send commands to the primary driver. Switched to secondary driver.
  

• Re-connect procedure started for activated secondary drivers.

  Primary driver re-connect check procedure started.
  

• Primary driver validation result.

  Primary driver re-connect check result: ${VALIDATION_RESULT}
  

• Secondary storage driver continues to be in active state.

  Primary driver is not available. Continue sending data using secondary driver.
  

• Primary driver is available.

  Primary driver is available. Switching from secondary.
  

• Switched to primary driver.

  Primary driver is available. Switch from secondary complete.
  

• Failed to switch from secondary driver.

  Primary driver is available. Failed to switch from secondary. Continue sending data using secondary driver.
  

• Re-connect procedure stopped for all drivers.

  Primary driver re-connect check procedure stopped. All drivers are sending data to primary drivers.
  

# Failover Logging Example

Database	Severity	Date	Entity	Tags	Message
Secondary	CRITICAL	2018-09-12 08:33:00.122	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Failed to send commands to the primary driver. Switching to secondary.
Secondary	CRITICAL	2018-09-12 08:33:00.190	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Failed to send commands to the primary driver. Switched to secondary driver.
Secondary	NORMAL	2018-09-12 08:33:17.760	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Primary driver re-connect check procedure started.
Secondary	CRITICAL	2018-09-12 08:33:17.830	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Primary driver re-connect check result: [Unexpected error: org.apache.http.conn.HttpHostConnectException: Connect to atsd.example.org [atsd.example.org/203.0.113.0] failed: Connection refused (Connection refused)]
Secondary	WARNING	2018-09-12 08:33:17.893	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Primary driver is not available. Continue sending data using secondary driver.
Secondary	NORMAL	2018-09-12 08:34:17.757	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Primary driver re-connect check procedure started.
Secondary	CRITICAL	2018-09-12 08:34:17.824	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Primary driver re-connect check result: [Unexpected error: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
Secondary	WARNING	2018-09-12 08:34:17.889	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Primary driver is not available. Continue sending data using secondary driver.
Secondary	NORMAL	2018-09-12 08:35:17.755	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Primary driver re-connect check procedure started.
Primary	NORMAL	2018-09-12 08:35:17.890	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Primary driver re-connect check result: SUCCESSFUL
Primary	WARNING	2018-09-12 08:35:17.917	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Primary driver is available. Switching from secondary.
Primary	NORMAL	2018-09-12 08:35:17.932	collector01	primary_driver_url = https://atsd.example.org , secondary_driver_url = https://198.51.100.1:8443	Primary driver is available. Switch from secondary complete.