Create the Pipeline

Step 1: Create a new Pipeline

Once you have Pipeline enabled, go ahead and Create a new Pipeline. You will be prompted to name your Pipeline, call it what you will but we will go with Snowflake-Integration. After creation, you will be taken to the following blank canvas

Step 2: Add the Source

This parts easy. Go to the pipeline you created previously and click Add Source

From there, just select HTTP, give it a Title like Edge Devices, set Decoding Method to JSON and click Save.

Now, we need to create an access key corresponding to the new HTTP source. Click on the HTTP source to bring up the following panel.

Go ahead and click the Create new key button in the Access Key Management section. Here you can give the new access key a name of Edge Device Key and click the Create button.

A new key will be generated for you to use and is displayed on the source details, as well as the HTTP address to send the data to. Be sure to copy this Access Key somewhere safe for later reference as you will not be able to view it again once the source node is updated. Click Update to save your changes.

You now have an endpoint defined that can receive any data. If you run into trouble here, please checkout out our comprehensive Mezmo Platform workshop to learn how to utilize the sytem in depth.

Step 3: Add the S3 Destination

Next we will add S3 as a destination.

You will need some information from you AWS account here. Specifically, you will need the following you created during step 2 of the previous section.

• AWS Access Key ID
• AWS Secret Access Key
• AWS Bucket name
• AWS Region

With access information in hand

• Add a new Destination and select AWS S3
• Give this Destination the title Snowflake Bucket
• Enter your Access Key ID and Secret Access Key
• Enter Bucket name (we will go with mezmo-use1-snowflake-demo)
• Enter the prefix of device-sim/event_date=%F/event_name={{ .event }}/. This prefix allows for dynamic location routing and will store data including date and event name coming from the event field. Example device-sim/event_date=2022-11-09/event_name=transaction/
• Chose text for the Encoding, with a compression of gzip
• Select the Region where you created your bucket (in this example we use us-east-1)
• Click the save button

Your destination should look similar to the image below.

Note: Make sure messages going to this S3 destination contain the dynamic field as part of the path. Any events that do not have the fields will not go to S3.

Now let’s connect the Edge Device Source to the Snowflake Bucket Destination. For this example, we are not going to use any processors.

Now, simply Deploy your Pipeline.

• ← Previous
• Next →