You can't use this until you register with us!




Description

Used for setting up a redshift database for nomnom loading and then loading data tables into it.



Available Actions

Initialize Cluster

Initialize a redshift cluster to allow tracking of loaded data.


Variables

redshift_connection required

Entry in the configs for the specified AWS redshift cluster you want to Initialize. Connection needs admin permissions.


Register Dataset

Register a data_table on a redshift cluster. This allows NomNomData to load the data_table into an initialized cluster.


Variables

Metadata Info
metadata required

Metadata location your data is coming from. We support a few types and generally companies only have a few of these.

data_table required

Data table that you want to load into your redshift cluster. As identified from your metadata catalog.

data_table_uuid required

UUID of the Data table that you want to load into your redshift cluster. As identified from your metadata catalog.


Redshift Info
redshift_connection required

Entry in the configs for the specified AWS redshift cluster you want to Initialize

target_table required

Table name you want to load into specified via the notation schema_name.table_name. To specify special characters in table name you need to quote the table_name. Note special characters in table names is not recommended.


Load Data Table

Load a data_table into a redshift cluster. Only allowed after a data_table has been registered on an initialized cluster.


Variables

Metadata Info
metadata required

Metadata location your data is coming from. We support a few types and generally companies only have a few of these.

data_table required

Data table that you want to load into your redshift cluster. As identified from your metadata catalog.

data_table_uuid required

UUID of the Data table that you want to load into your redshift cluster. As identified from your metadata catalog.


Redshift Info
redshift_connection required

Entry in the configs for the specified AWS redshift cluster you want to Initialize. Needs write permissions to the table you are loading to as well as staging tables.

target_table required

Table name you want to load into specified via the notation schema_name.table_name. To specify special characters in table name you need to quote the table_name. Note special characters in table names is not recommended.


S3 Info
aws_connection

AWS Credentials with access to the s3 bucket where the data table lives. Should also have access to a temp directory.

aws_iam

AWS IAM Credentials with access to the s3 bucket where the data table lives. Should also have access to a temp directory.

s3_temp_space

Path to a location where the s3 credentials have write access and read access. Needs to be in the same region/account/bucket where the redshift files live. If left blank the sytem will use the temp path defined in the aws connection if one exists.


Loading Options
batch_size required

Number of files to load per batch. Min of 1, Max of 100,000

path_filter

RegEx of filter to apply for loading files. Can use <YYYY> sytax to load time ranges for current dates


Register & Load

Verifies registration and load a data_table into a redshift cluster. Only allowed after a cluster has been initilized initialized cluster.


Variables

Metadata Info
metadata required

Metadata location your data is coming from. We support a few types and generally companies only have a few of these.

data_table required

Data table that you want to load into your redshift cluster. As identified from your metadata catalog.

data_table_uuid required

UUID of the Data table that you want to load into your redshift cluster. As identified from your metadata catalog.


Redshift Info
redshift_connection required

Entry in the configs for the specified AWS redshift cluster you want to Initialize. Needs write permissions to the table you are loading to as well as staging tables.

target_table required

Table name you want to load into specified via the notation schema_name.table_name. To specify special characters in table name you need to quote the table_name. Note special characters in table names is not recommended.


S3 Info
aws_connection

AWS Credentials with access to the s3 bucket where the data table lives. Should also have access to a temp directory.

aws_iam

AWS IAM Credentials with access to the s3 bucket where the data table lives. Should also have access to a temp directory.

s3_temp_space

Path to a location where the s3 credentials have write access and read access. Needs to be in the same region/account/bucket where the redshift files live. If left blank the sytem will use the temp path defined in the aws connection if one exists.


Loading Options
batch_size required

Number of files to load per batch. Min of 1, Max of 100,000

path_filter

RegEx of filter to apply for loading files. Can use <YYYY> sytax to load time ranges for current dates