ATSD Cluster Migration
These instructions describe how to upgrade an ATSD instance running on the Cloudera CDH cluster.
For non-distributed installations, refer to the following migration guide.
Versioning
| Code | ATSD Revision Number | Java Version | Cloudera Manager Version | CDH Version |
|---|---|---|---|---|
| Old | 16999 and earlier | 1.7 | 5.1 - 5.11 | 5.1 - 5.9 |
| New | 17000 and later | 1.8 | 5.12 | 5.10 |
Requirements
Disk Space
The migration procedure requires up to 30% of the current atsd_d table size to store new records before old data can be deleted.
Open the Clusters > Cluster > HDFS > Status page in Cloudera Manager.
Be sure that enough configured capacity is available in HDFS.
Memory
The migration task is implemented as a Map-Reduce job and requires at least 4 GB of RAM available on each HBase region server.
Check Record Count for Testing
Log in to ATSD and open the SQL tab.
Count rows for one of the key metrics.
SELECT COUNT(*) FROM mymetric
The number must match the results after the migration.
Prepare ATSD For Upgrade
Stop ATSD.
/opt/atsd/atsd/bin/stop-atsd.sh
Take note of the table prefix specified in the /opt/atsd/atsd/conf/server.properties file.
cat /opt/atsd/atsd/conf/server.properties | grep "hbase.table.prefix" | cut -f 2 -d "="
This prefix is required in subsequent steps.
Install Java 8
Install Java 8 on the ATSD server.
Upgrade Cloudera Cluster
Upgrade Cloudera Manager to version 5.12.
Upgrade CDH to version 5.10.
Start HDFS, HBase, YARN and HistoryServer services.
Configure Migration Map-Reduce Job
Log in to the server where YARN ResourceManager is running.
Locate the yarn.keytab file.
sudo find / -name "yarn.keytab" | xargs ls -la | tail -n 1
-rw------- 1 yarn hadoop 448 Jul 29 16:44 /run/cloudera-scm-agent/process/7947-yarn-RESOURCEMANAGER/yarn.keytab
Switch to the yarn user.
sudo su yarn
Initiate a Kerberos session
Obtain the fully qualified hostname of the YARN ResourceManager server.
hostname -f
Authenticate with Kerberos using the located yarn.keytab file and the full hostname of the YARN ResourceManager.
kinit -k -t /run/cloudera-scm-agent/process/7947-yarn-RESOURCEMANAGER/yarn.keytab yarn/{yarn_rm_full_hostname}
Download the migration.jar file to the temporary /tmp/migration/ directory.
mkdir /tmp/migration
curl -o /tmp/migration/migration.jar \
https://axibase.com/public/atsd-125-migration/migration-hbase-1.2.0-cdh5.10.0.jar
Check that the current Java version is 8.
java -version
Add migration.jar, HBase configuration files, and HBase classes used by the migration job to Java and Hadoop classpaths.
export CLASSPATH=$CLASSPATH:$(hbase classpath):/tmp/migration/migration.jar
export HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hbase/bin/../conf:$(hbase mapredcp):/tmp/migration/migration.jar
Modify Map-Reduce settings using parameters recommended by Axibase support based on the Data Reporter logs.
Run Migration Map-Reduce Job
Execute the below steps under the yarn user on the YARN ResourceManager server.
Backup atsd_d Table
Run the TableCloner task to rename atsd_d table as atsd_d_backup.
java com.axibase.migration.admin.TableCloner --table_name=atsd_d
If a custom table prefix is specified in the server.properties file, for example, if the prefix is set to atsd_custom_, change the table_name parameter accordingly:
java com.axibase.migration.admin.TableCloner --table_name=atsd_custom_d
Migrate Records
Launch the migration job with the nohup command.
The job creates a new data table, convert data from the backup table to the new format, and store the data in the new table.
The optional drop-annotations setting discards duplicate series annotations when copying the data.
Custom Table Prefix
nohup yarn com.axibase.migration.mapreduce.DataMigrator --forced \
--drop_annotations --source=atsd_custom_d_backup \
--destination=atsd_custom_d &> /tmp/migration/migration.log &
Default Table Prefix
nohup yarn com.axibase.migration.mapreduce.DataMigrator --forced \
--drop_annotations --source=atsd_d_backup \
--destination=atsd_d &> /tmp/migration/migration.log &
Monitoring Job Progress
The job takes several hours to complete.
Open Clusters > Cluster > YARN > Web UI > ResourceManager Web UI in Cloudera Manager.
Monitor the job progress.
tail -f /tmp/migration/migration.log
Note that the Yarn interface is stopped automatically once the job is finished.
Once the job is complete, the migration.log file contains the following message:
17/08/01 10:44:31 INFO mapreduce.DataMigrator: HFiles loaded, data table migration job completed, elapsed time: 45 minutes.
Deploy ATSD Coprocessors
Copy New Coprocessors into HDFS
Log in to HDFS NameNode server or another server with the DFS client.
Switch to the hdfs user.
sudo su hdfs
Copy atsd-hbase.17326.jar to the HDFS hbase.dynamic.jars.dir directory.
The path to this directory is set to ${hbase.rootdir}/lib by default in HBase.
hadoop fs -ls /hbase/lib/ # check existence
hadoop fs -mkdir /hbase/lib/ # if not exists
curl -O https://axibase.com/public/atsd-125-migration/atsd-hbase.17326.jar
hadoop fs -put -f atsd-hbase.17326.jar /hbase/lib/atsd-hbase.jar
hadoop fs -ls /hbase/lib
Found 1 items
-rw-r--r-- 3 hdfs hbase 547320 2017-08-23 13:03 /hbase/lib/atsd-hbase.jar
This path must match the coprocessors.jar setting specified in the /opt/atsd/atsd/conf/server.properties file in the ATSD server as outlined below.
Remove Old Coprocessors
ATSD coprocessors added to HBase CoprocessorRegion Classes have been loaded automatically and therefore must be removed from HBase settings in Cloudera Manager.
Remove old jar files from the local file system on each HBase Region Server.
Remove Coprocessor Settings
Open the Clusters > Cluster > HBase-2 tab in Cloudera Manager.
Open the Configuration tab.
Search for the hbase.coprocessor.region.classes setting.
Remove all ATSD coprocessors and save settings:
Remove Coprocessor jar Files
Find ATSD coprocessors jar files on each Region Server:
sudo find /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/ -name "atsd*.jar"
Remove files found by the above search.
Restart HBase.
Configure ATSD
Log in to the ATSD Server.
Switch to the axibase user.
sudo su axibase
Remove deprecated settings.
sed -i '/^hbase.regionserver.lease.period/d' /opt/atsd/atsd/conf/hadoop.properties
Add path to co-processor jar file.
echo "coprocessors.jar=hdfs:///hbase/lib/atsd-hbase.jar" >> /opt/atsd/atsd/conf/server.properties
Upgrade jar files and start-up scripts.
rm -f /opt/atsd/atsd/bin/*
curl -o /opt/atsd/atsd/bin/atsd.17370.jar https://axibase.com/public/atsd-125-migration/atsd.17370.jar
curl -o /opt/atsd/scripts.tar.gz https://axibase.com/public/atsd-125-migration/scripts.tar.gz
tar -xf /opt/atsd/scripts.tar.gz -C /opt/atsd/ atsd
rm /opt/atsd/scripts.tar.gz
rm -rf /opt/atsd/hbase
rm -rf /opt/atsd/collectors
Update JAVA_HOME in the start-atsd.sh file:
jp=`dirname "$(dirname "$(readlink -f "$(which javac || which java)")")"`; sed -i "s,^export JAVA_HOME=.*,export JAVA_HOME=$jp,g" /opt/atsd/atsd/bin/start-atsd.sh
Edit the /opt/atsd/atsd/conf/atsd-env.sh file.
Increase Xmx memory to 50% of available RAM memory on the ATSD server:
JAVA_OPTS="-server -Xmx4096M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="$atsd_home"/logs"
Start ATSD.
/opt/atsd/atsd/bin/start-atsd.sh
Review the start log for any errors:
tail -f /opt/atsd/atsd/logs/atsd.log
Watch for ATSD start completed message at the end of the start.log.
Web interface is now accessible on port 8443 (HTTPS).
Check Migration Results
Open the SQL tab in ATSD.
Count rows for the previously selected metric and compare the results.
SELECT COUNT(*) FROM mymetric
The number of records must match the results prior to migration.
Delete Backup Table
Log in to HBase shell.
Delete atsd_d_backup table. Adjust the table name if a custom prefix is specified in the server.properties file.
/usr/lib/hbase/bin/hbase shell
hbase(main):001:0> disable 'atsd_d_backup'
hbase(main):002:0> drop 'atsd_d_backup'
hbase(main):003:0> exit
Delete Temporary Migration Folder
rm -rf /tmp/migration