Connect to S3 and Deploy

Step 1: Add S3 Financial Destination

With our data cleaned we can get fancy with how we route the financial transactions here (see the workshop on S3 to Snowflake to learn more), but we will keep it simple for now. Let’s dump all this data into a single S3 bucket for our data engineering teams.

You will need the following information from you AWS account:

• AWS Access Key ID
• AWS Secret Access Key
• AWS Bucket Name
• AWS Region

With those in hand, add a new Destination and select AWS S3 (docs)

• Give the title S3 Fin Transactions
• Enter your Access Key ID
• Enter your Secret Access Key
• Enter your Bucket name (we will go with mezmo-pipeline-financial-transactions)
• Select JSON for the Encoding
• Enter your Region (we will go with us-east-1
• Click Save when yours looks similar to the image below.

Then, connect up the last Encryption Processor for the CC data to this destination like so

Step 2: Add S3 General Destination

The last step before deploying is to funnel the cleaned data and those that were unmatched to the teams general S3 bucket.

Follow a similar procedure to step one, but this time create it using a new bucket (say mezmo-pipeline-financial-all) and a new name of S3 General.

Once done, connect up that Destination to the same final Encryption Processor from Step 1 as well as the Unmatched Route from earlier. You should end up with something like this

Step 3: Deploy

Now, simply Deploy pipeline in the top right. After the Pipeline should no longer be a draft (if you hadn’t deployed earlier) and look like this

Watch as data comes into both S3 buckets. Looking at the bucket connected to S3 Fin Transactions, you should begin seeing files like so

Note that it will take up to 5 min to first see data flowing in to S3. This is due to batching and our durable queues, no data will be dropped.

• ← Previous
• Next →