AWS Job in Axibase Collector lets you collect Amazon CloudWatch metrics into ATSD for advanced analytics and long-term retention. For example, you can use this job to gather the following performance metrics for your EC2 instances:
The complete up-to-date list of namespaces, metrics, and dimensions (identifiers and keys) is available at https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/CW_Support_For_AWS.html.
Once the data starts flowing, you can visualize it, build forecasts, and create alerts using windowing statistics in the ATSD rule-engine. Entities collecting Amazon CloudWatch metrics will be automatically added to the AWS EC2 entity group and will have a default visualization portal assigned.
The job executes scheduled HTTP API requests to the Amazon CloudWatch monitoring service and stores the metrics in the specified ATSD instance. The job stores markers for each metric so that API requests load only the most recent incremental data. These granular markers allow the AWS job to built optimized queries and avoid data gaps in case of temporary network or service outage. When AWS job is started for the first time, it will load all of the available historical data from the CloudWatch service, currently retained by CloudWatch for 2 weeks.
You can configure the AWS job to collect metrics for all or a subset of namespaces. For each metric, AWS job retrieves the following period statistics: average, minimum, maximum, sum, sample count. The length of the period is typically 5 minutes, although 1 minute period is available if detailed monitoring is enabled for EC2 instances. AWS customers can switch between basic and detailed monitoring directly in the AWS console. AWS charges additional fees for detailed monitoring. The detailed monitoring also provides lower delay, typically 1 or 2 minutes whereas basic monitoring can have a delay between 5 and 10 minutes.
How to create an AWS job in Collector:
1. Create a dedicated read-only user account in Amazon IAM to query CloudWatch service.
This would enforce granular security and prevent user-based API quota limit errors. See notes here
2. Create HTTP Pool:
- Click [Data Sources] -> HTTP Pools and create a new pool.
- Specify hostname of the endpoint without “monitoring.” part, e.g. us-east-1.amazonaws.com for endpoint monitoring.us-east-1.amazonaws.com. Endpoint is URL of the CloudWatch monitoring service that AWS job will be using. For a complete list of endpoints see: https://docs.aws.amazon.com/general/latest/gr/rande.html#cw_region We recommend selecting an endpoint with maximum affinity to the availability zone where most of your hosted resources are provisioned. If data stops flows from this endpoint, this would server as another early warning for your operational monitoring needs.
- Click [Test] to verify that the service returns 200 response code.
- Save the pool.
3. Create AWS Job:
- Open Jobs page, click on [Add Job] button, select AWS Job.
- Set Status to Enabled.
- Enter job name and a cron expression, for example ‘0 0/5 * * * ?’ to execute the job every 5 minutes
- Click [Save] and add AWS configuration.
- Enter configuration name
- Select HTTP pool created in the previous step 2.
- Enter region endpoint, for example monitoring.us-east-1.amazonaws.com
- Enter username, Access key id, and Secret access key for the read-only user created in Step 1.
- Specify which metrics to collect by selecting particular namespaces, note that Custom Metrics can be collected.
- Save the job.
4. Verify metrics in ATSD:
- Click on [Metrics] tab to view available AWS* metrics. Metric naming convention is [namespace].[metric].[statistic], for example, aws_ec2.cpuutilization.avg or aws_billing.estimatedcharges.sum. Note that ATSD treats avg, min, max, sum statistics received from AWS as raw data, which means that ATSD can compute aggregated statistics on top of AWS statistics as if it were detailed data.
- Click on [Entities] tab to view EC2 instances. Click on Last Insert column to view list of metrics available for an instance. Click on Chart link to view metric values.
- Note that it may take AWS job a while to load backlogged historical data for all metrics after AWS job was created.
5. Built-in Entity Groups and Portals in ATSD
ATSD comes with two built-in Entity Groups for Entities gathering AWS performance statistics:
Each Entity in the AWS Entity Groups is assigned a relevant visualization portal:
AWS EBS VOLUME
AWS EC2 Portal
AWS EBS Portal