You can't use this until you register with us!

Description

When upsight places files on s3 for they are in token format. This engine copies shuffles the files to a new folder path allowing for more intuitive data loading.

Available Actions

Copy Files

Take files from the list of keys token definitions provided and moves them to the target with the new specified pattern

Variables

config_list required

Configuration list with parameters for files you want to copy.


Connection Information
tracking_db required

Database where you store which files you have processed and their status

tracking_table required

Tracking schema and tablename where the file statuses will be stored

source_aws_connection

AWS Credentials with access to the source and target s3 locations.


Source Info
source_s3_pattern required

Pattern required to match for any files to be moved.

source_bucket

S3 Bucket where you want to unload the data to.


Target Info
target_s3_pattern required

Pattern required to match for any files to be moved Tokens will automatically be placed in deepest folder level.

target_bucket

S3 Bucket where you want to unload the data to.


Custom Arguments
lookback required

Number of days based on the pattern to lookback

reprocess required

If True reprocess files even if you have already successfully processed those files before.


Copy Files

Take files from the list of keys token definitions provided and moves them to the target with the new specified pattern

Variables

config_list required

Configuration list with parameters for files you want to copy.


tracking_table required

Tracking schema and tablename where the file statuses will be stored


Tracking Database
connection_info

Database where you store which files you have processed and their status

source_aws_connection

AWS Credentials with access to the source and target s3 locations.


Source Info
source_s3_pattern required

Pattern required to match for any files to be moved.

s3_bucket

S3 Bucket where you want to unload the data to.


Target Info
target_s3_pattern required

Pattern required to match for any files to be moved Tokens will automatically be placed in deepest folder level.

s3_bucket

S3 Bucket where you want to unload the data to.


Custom Arguments
lookback required

Number of days based on the pattern to lookback

reprocess required

If True reprocess files even if you have already successfully processed those files before.