In many cases you may expect to run a particular pipeline if a condition is met. To Evaluate the condition and perform some action is most common scenario in Data Engineering field.
ADF/Synapse pipeline offers an activity to achieve this, The activity is called the SWITCH activity. This activity can evaluate the condition and perform the action which is expected to run for that particular event.
I am assuming a situation where I have storage location in ADLS Gen 2 which accepts different files let’s say a customer.csv and product.csv. For these two you have different mapping and some other transformation while loading from file to a SQL table or to any other type of sink.
You have to run the pipeline which is dedicated to customer and product depending on which file is present in the blob.
Here you can use the SWITCH activity which will evaluate the filename and will process the file according to that.
We will use a storage event trigger so that when a file is created then from the payload we can extract the filename and will use that for evaluation.
Lets design the Pipeline.
Steps:
- Click on the Integrate tab
- Click on the + symbol to create new pipeline.
- Search for the SWITCH activity and drag the Switch activity to the canvas and name the activity.
- Create a parameter(filename). This will be used to hold the filename from the payload and will be used for evaluation.
- Name the pipeline as per your environment naming convention.
Once the above steps are done you can click on the activity and in the expression.
- We have to provide an expression that will be used for the evaluation. Here we will first take the filename from the payload that is saved in parameter, convert into the lower case to avoid the any comparison issue.
#Expression example @replace(toLower(pipeline().parameters.filename),'.csv','')
It is taking the filename parameter then converting it to lower case and at last it is removing the extension to take the filename without extension.
Note: The expression is case sensitive so you must have a uniform name else it will not get evaluated.
- Once we converted the name of the file with lower case, we will now remove the extension from the file name to get the filename.
- Since we are going to use switch activity, we need multiple case statements. I have kept two case since I am expecting two file to be processed. I have kept the first case as lowercase customer and second case as lowercase product. These two will be evaluated with the expression output and if anyone of these cases matches with the expression output then the activities inside that will be triggered.
- If none of the condition is True, then the default case will be executed.
Attach the storage event trigger to the pipeline and pass the parameter to take the file from the payload i.e. @trigger().outputs.body.fileName
Now you can publish the pipeline and drop a file with name Customer.csv or Product.csv.
I have dropped a file with the name ProducT.csv, I have kept the name in upper and lowercase to see if the expression if working properly.
It worked fine and the copy activity ran perfectly.
Similarly, if you drop a file which cannot be evaluated in the case condition then the default script will run.
Default Email:
