Encrypt and Filter

Overview

There is a lot we may want to do with the data as you may have saw while looking at live data.

For this workshop we are going to encrypt PII, drop useless information from events and then route the financial transaction data to a specific S3 bucket while sending that and everything else to the teams general S3 bucket.

While we are going with S3 for this workshop, we have many other Destinations available today and others, like Mezmo Log Analysis, that are experimental. If interested in access to experimental features, reach out to your account representative or support@mezmo.com.

But, let’s take this one step at a time.

Step 1: Drop the Unnecessary Buffer

We don’t need the buffer, so let’s drop it. First select Add Processor which will pull up a dialog like so

• Select Remove Fields from the list (docs)
• Give it a title like Drop buffer
• Enter the field .buffer to drop it
• Click Save.

Then connect this to the Source processor by hovering over the Source till you see a gray half circle. Click and drag to the right edge of the Drop buffer node. Release the mouse and things are linked up. Data will now flow left to right, from the Source to the Processor. Also note that things will rearrange themselves as you go.

You may have noticed we are referencing the buffer key via .buffer. This is syntax is slightly different than you may be used to but its very straight forward. To learn more, check out our docs here.

Step 3: Route Transaction Data

We want to send only the transaction events to S3, to do this we can use a Route processor (docs). Go ahead and add one with the Title Transactions.

We could group successful and failed transactions (.transaction.result) but let’s seperate the routes. To do this, we will create two Outputs.

For the first route:

• Give it the name Transaction Success
• Select an IF and enter .transaction.result equals true
• To weed out any anomalies for later analysis, lets also ensure .transaction.total_price is greater_or_equal to 0 via Add Expresion.

Similarly, for the second output:

• Select Add route and enter the name Transaction Fail
• Configure the IF with .transaction.result equals false
• Eliminate anomalies with the .transaction.total_price expression from above.

Click Save.

Connect the Drop Buffer processor from Step 1 to the Route processor you just created.

Note that we will leave the Unmatched route untouched for this workshop. But there are many things that could be done with this data: send to Log Analytics, send to a SIEM, etc.

Step 4: Encrypt the Credit Card Information

Now, let’s encrypt each of the credit card fields individually to ensure security and compliance. The fields we want to encrypt are

• .transaction.cc.cc_number
• .transaction.cc.cc_exp
• .transaction.cc.cc_cvv
• .transaction.cc.cc_zip
• .transaction.cc.cc_name

Since each are unique, order doesn’t matter so much here. For each:

• Add an Encrypt Field processor (docs)
• Choose the AES-256-CFB algorithm with a 32 character Encryption Key (checkout AllKeysGenerator.comto generate each key)
• Add an Initialization Vector and name it whatever you like. Note that every encryption processor needs to add a key like this to the event itself for decryption down the road.
• Click Save.

Once you do this for each of the above fields (or don’t, it’s just a demo pipeline afterall), you should have 5 floating processors like so

Now, connect each one sequentially and then link the fail and success routes to the first processor in this group in parallel. The Pipeline should now look similar to

Now that the transformations have been defined, it’s time to sink this all up to the S3 and start gathering data.

• ← Previous
• Next →