Used for setting up a redshift database for nomnom loading and then loading data tables into it.
Initialize a redshift cluster to allow tracking of loaded data.
Variables
Entry in the configs for the specified AWS redshift cluster you want to Initialize. Connection needs admin permissions.
Register a data_table on a redshift cluster. This allows NomNomData to load the data_table into an initialized cluster.
Variables
Metadata location your data is coming from. We support a few types and generally companies only have a few of these.
Data table that you want to load into your redshift cluster. As identified from your metadata catalog.
UUID of the Data table that you want to load into your redshift cluster. As identified from your metadata catalog.
Entry in the configs for the specified AWS redshift cluster you want to Initialize
Table name you want to load into specified via the notation schema_name.table_name. To specify special characters in table name you need to quote the table_name. Note special characters in table names is not recommended.
Load a data_table into a redshift cluster. Only allowed after a data_table has been registered on an initialized cluster.
Variables
Metadata location your data is coming from. We support a few types and generally companies only have a few of these.
Data table that you want to load into your redshift cluster. As identified from your metadata catalog.
UUID of the Data table that you want to load into your redshift cluster. As identified from your metadata catalog.
Entry in the configs for the specified AWS redshift cluster you want to Initialize. Needs write permissions to the table you are loading to as well as staging tables.
Table name you want to load into specified via the notation schema_name.table_name. To specify special characters in table name you need to quote the table_name. Note special characters in table names is not recommended.
AWS Credentials with access to the s3 bucket where the data table lives. Should also have access to a temp directory.
AWS IAM Credentials with access to the s3 bucket where the data table lives. Should also have access to a temp directory.
Path to a location where the s3 credentials have write access and read access. Needs to be in the same region/account/bucket where the redshift files live. If left blank the sytem will use the temp path defined in the aws connection if one exists.
Number of files to load per batch. Min of 1, Max of 100,000
RegEx of filter to apply for loading files. Can use <YYYY> sytax to load time ranges for current dates
Verifies registration and load a data_table into a redshift cluster. Only allowed after a cluster has been initilized initialized cluster.
Variables
Metadata location your data is coming from. We support a few types and generally companies only have a few of these.
Data table that you want to load into your redshift cluster. As identified from your metadata catalog.
UUID of the Data table that you want to load into your redshift cluster. As identified from your metadata catalog.
Entry in the configs for the specified AWS redshift cluster you want to Initialize. Needs write permissions to the table you are loading to as well as staging tables.
Table name you want to load into specified via the notation schema_name.table_name. To specify special characters in table name you need to quote the table_name. Note special characters in table names is not recommended.
AWS Credentials with access to the s3 bucket where the data table lives. Should also have access to a temp directory.
AWS IAM Credentials with access to the s3 bucket where the data table lives. Should also have access to a temp directory.
Path to a location where the s3 credentials have write access and read access. Needs to be in the same region/account/bucket where the redshift files live. If left blank the sytem will use the temp path defined in the aws connection if one exists.
Number of files to load per batch. Min of 1, Max of 100,000
RegEx of filter to apply for loading files. Can use <YYYY> sytax to load time ranges for current dates