There could be the scenario where you want to copy the file that gets upload in SharePoint to your Azure Storage account. I have a scenario where the file from any source will get uploaded in SharePoint folder (it could be a manual upload of dumped by any automatic process) once it is uploaded, we…
In many cases you may expect to run a particular pipeline if a condition is met. To Evaluate the condition and perform some action is most common scenario in Data Engineering field. ADF/Synapse pipeline offers an activity to achieve this, The activity is called the SWITCH activity. This activity can evaluate the condition and perform…
Today we are going to discuss on how to clone any lower environment dedicated SQL pool with the upper environment dedicated SQL pool or vice versa within a subscription. There could be the situation if you have lower environments which does not get refreshed every day, but you must clone them with the PROD…
We saw how to create a database in the synapse SPARK pool in our previous blog. Today we are going to understand the types of Delta table in SPARK. There are mainly two types of SPARK tables. There is slight difference while creating these tables, in a nutshell “Managed” table does not need any Location…
We have seen the overview of the Delta Lake solution Architecture and the operation on the Delta Lake needs to be done using SPARK. So today, we will discuss on the creation of the SPARK db that will serve as the layers in the Delta Lake. It is always good to segregate the layers to…
Let’s understand the concept of a Delta Lake Architecture so that later on we can go ahead and understand the concept of the Delta tables in the Delta Lake and their capabilities. The Delta Lake is optimized advance storage layer. The Delta Lake follows the medallion architecture where the tables are segregated in different categories…
This topic is quite rare because no one wants to delete their application or resource’s connection string secret from the key vault especially in PROD where the number of dependencies are quite high. But by mistake it can be mishandled. Always make sure that the below processes are followed as best practice: The main key…
So today, we have an amazing topic to discuss on triggering refresh on Power BI Dataset in a workspace using Rest API. In many Business scenarios, the Power BI reports needs to be Triggered/refreshed once the batch load is completed every day.The batch load finish time can be dynamic and cannot be believed to finish…
We have seen the implementation of the Lake Database in our previous blog, and we will see that the object created using the SPARK in the Lake Database can be easily queried using the Serverless SQL Pool. This gives a benefit that the tables can be queried on the Ad-hoc basis without turning the SPARK…
The Lake Database can be expressed as Database on top of a Storage which got added to the Synapse workspace Studio in 2022 as a new feature by Microsoft. At first glance, if you read about it then it seems a database which you can create from a template that is prebuilt and exposed to the user from the variety of industries. Well, that’s true that you can just start to design your own data model on top of the Data Lake and it gives you a GUI to map the relationship between the tables you want to create on top of your Azure Storage using Database Designer.
