Placeholders

Overview

Collector supports the following placeholders to format strings, calculate dates, and download multiple files.

Name Description
${ITEM} Current element in the Item List.
${PATH} URL path or the file's absolute path.
${FILE} Name of the file *.
${DIRECTORY} Parent directory of the file.
${TIME()} Text output of the TIME function.
${DATE_ITEM()} Current element in the Date Item list.
  • * In the case of HTTP protocol, the placeholder ${FILE} returns part of the URL after the last slash and before query string. For example: http://examples.com/data/stats.csv?city=Denver -> ${FILE} = stats.csv

Usage

Name Supported Fields Supported Protocols
${ITEM} Path, First Line Contains, Default Entity, Metric Prefix, Custom Tags, Success Directory, Error Directory All
${PATH} First Line Contains, Default Entity, Metric Prefix All
${FILE} First Line Contains, Default Entity, Metric Prefix All
${DIRECTORY} First Line Contains, Default Entity, Metric Prefix FILE, FTP, SFTP, SCP
${TIME()} First Line Contains, Path, Success Directory, Error Directory All
${DATE_ITEM()} Path All
  • To URL-encode placeholder value, for example, if the placeholder may contain special characters and is included in the Path, apply the url function as follows: ${ITEM?url}.

Syntax

The placeholder is prefixed with $ and enclosed in curly brackets {}.

${PLACEHOLDER}

Examples:

${ITEM}

file:///opt/files/inbound/${TIME("previous_day", "yyyy-MM-dd")}/daily.csv

Functions

TIME Function

The TIME function calculates time based on the calendar expression. The syntax outputs the value in the specified time_format.

Syntax: ${TIME("end_time_syntax", "time_format")}

Example: ${TIME("previous_hour", "yyyy-MM-dd/HH")}

Specify the time format using y, M, d, H, m, or s. See SimpleDateFormat for reference.

If the TIME function returns a number, use the function in addition or subtraction expressions:

Example: ${TIME("2016-01-01","M") - 1} (subtract 1 from month).

DATE_ITEM Function

The DATE_ITEM function returns an array of strings.

Function creates an array of dates between start and end time arguments, and formats these dates into strings with the specified date format.

If the path contains the ${DATE_ITEM()} placeholder, the function executes a separate file request for each string in the array.

Syntax: ${DATE_ITEM(startDate, endDate, periodCount, periodUnit, timeFormat)}

startDate and endDate support calendar expressions.

Example: ${DATE_ITEM("current_day - 7 * DAY", "now", 1, "HOUR", "yyyy/MM/dd/HH")}

String Functions

Placeholder values can be further modified with built-in string functions.

${PLACEHOLDER?function(arguments)}

Example: ${FILE?keep_before("_")}

Function Description
keep_after Removes part of the string before the first occurrence of the given substring.
keep_after_last Removes part of the string before the last occurrence of the given substring.
keep_before Removes part of the string that starts with the first occurrence of the given substring.
keep_before_last Removes part of the string that starts with the last occurrence of the given substring.
replace Replace all occurrences of the given string in the original string with another string.
remove_beginning Removes the given substring from the beginning of the string.
remove_ending Removes the given substring from the end of the string.

Multiple functions can be chained (executed from left to right):

${PLACEHOLDER?functionA(arguments)?functionB(arguments)}

Example: ${FILE?keep_before("_")?replace(".csv", "")}

Input: ftp.example.com/data/95014_Cupertino_20160625_DAILY.csv

Expression Result
${FILE} 95014_Cupertino_20160625_DAILY.csv
${FILE?keep_before('_')} 95014
${FILE?keep_after('_')?keep_before('_')} Cupertino
${FILE?split('_')[0]} 95014
${FILE?split('_')[1]} Cupertino
${FILE?split('_')[n]} Nth token after split by underscore
${FILE?keep_after_last('_')?keep_before('.')} DAILY

String Function Examples

The following examples are based on the Path field. The Path field can be used to define Default Entity.

keep_after

  • file:///opt/files/cpu_busy.*
  • ${PATH?keep_after('.')}
Matching Paths Output
/opt/files/cpu_busy.nurswgvml.106
/opt/files/cpu_busy.nurswgvml.107
nurswgvml.106
nurswgvml.107

keep_after_last

  • /2.2/tags/docker/info?key=privateKey((&site=${ITEM}
  • ${ITEM?keep_after_last("-")}
ITEM value Output
so-stackoverflow stackoverflow

keep_before

  • ftp://user:password@10.10.0.10:21/home/user/nurswgvml106_*
  • ${FILE?keep_before('_')}
Matching Paths Output
/home/user/nurswgvml106_temperature.csv nurswgvml106

keep_before_last

  • file:///opt/files/*_busy.csv
  • ${FILE?keep_before_last('_')}
Matching Paths Output
/opt/files/nurswgvml106_cpu_busy.csv
/opt/files/nurswgvml107_cpu_busy.csv
nurswgvml106_cpu
nurswgvml107_cpu

replace

  • file:///opt/files/*
  • ${FILE?replace(' ','.')}
Matching Paths Output
/opt/files/nurswgvml106 cpu_busy nurswgvml106.cpu_busy

remove_beginning

  • file:///opt/files/*
  • ${PATH?remove_beginning('/opt/files/')}
Matching Paths Output
/opt/files/nurswgvml106
/opt/files/nurswgvml107
nurswgvml106
nurswgvml107

remove_ending

  • file:///opt/files/*.cpu_busy.csv
  • ${FILE?remove_ending('.cpu_busy.csv')}
Matching Paths Output
/opt/files/nurswgvml106.cpu_busy.csv
/opt/files/nurswgvml107.cpu_busy.csv
nurswgvml106
nurswgvml107

LOOKUP function

The LOOKUP function provides key-to-value mapping in the specified Item List. Use the function in JSON, JDBC, PI jobs to resolve entities.

  • Syntax

The LOOKUP function has several implementations.

LOOKUP(itemList, key, separator='=')

Treat Item List as a properties file. Key is a substring of item list row before the separator, value is a substring after the separator. If Separator parameter is not specified, = is the separator.

Parameter name Parameter type Description
itemList String Name of the Item List
key String Lookup Key
separator String (optional) Key/value separator

LOOKUP(itemList, key, keyColumnIndex, valueColumnIndex, separator=',')

Treat Item List as a CSV structure.

Parameter name Parameter type Description
itemList String Name of the Item List
key String Lookup Key
keyColumnIndex Integer Index of column in a CSV row used as the key column, starting with 1
valueColumnIndex Integer Index of column in a CSV row used as the value column, starting with 1
separator String (optional) CSV columns separator

LOOKUP Examples

Consider the Item List 'us-regions' with items:

1=New-England
2=Middle-Atlantic
3=East-North-Central
4=West-North-Central
5=South-Atlantic
6=East-South-Central
7=West-South-Central
8=Mountain
9=Pacific
${LOOKUP('us-regions', '9')} => Pacific
${LOOKUP('us-regions', '2', '=')} => Middle-Atlantic

Consider the Item List 'meteo-stations':

94274,QLD,IDQ60801
94341,QLD,IDQ60801
94683,SA,IDS60801
94677,SA,IDS60801
94141,NT,IDD60801
94131,NT,IDD60801
94461,WA,IDW60801
94319,WA,IDW60801
${LOOKUP('meteo-stations', '94461', 1, 2)} => WA
${LOOKUP('meteo-stations', '94683', 1, 3, ',')} => IDS60801
${LOOKUP('meteo-stations', '94141', ',')} => NT,IDD60801