We have seen the solution architecture of the delta lake in our Blog and how we can register a delta table in the Delta Lake. However, this blog will go around the working of the Delta table from the time you register and perform the DML operation on it. We will see internal architecture of…
Today we are going to see, how to read an excel file which is stored in the Azure Storage from Synapse notebook using python OpenPyxl Library. Openpyxl is the library that is designed to specially work with the Excel files, and it provides rich list of methods to work with the Excel data. However, the…
We have seen how to clone the Synapse Dedicated SQL pool using the GUI in our blog https://techdiw.com/cloning-restoration-dedicated-sql-pool-formerly-sql-dw-in-synapse/. We can perform the same action using the PowerShell and make it more generic where it can be used for the Automation and can reuse the code to clone from any source to any target within the…
There could be the scenario where you want to copy the file that gets upload in SharePoint to your Azure Storage account. I have a scenario where the file from any source will get uploaded in SharePoint folder (it could be a manual upload of dumped by any automatic process) once it is uploaded, we…
In many cases you may expect to run a particular pipeline if a condition is met. To Evaluate the condition and perform some action is most common scenario in Data Engineering field. ADF/Synapse pipeline offers an activity to achieve this, The activity is called the SWITCH activity. This activity can evaluate the condition and perform…
Today we are going to discuss on how to clone any lower environment dedicated SQL pool with the upper environment dedicated SQL pool or vice versa within a subscription. There could be the situation if you have lower environments which does not get refreshed every day, but you must clone them with the PROD…
We saw how to create a database in the synapse SPARK pool in our previous blog. Today we are going to understand the types of Delta table in SPARK. There are mainly two types of SPARK tables. There is slight difference while creating these tables, in a nutshell “Managed” table does not need any Location…
We have seen the overview of the Delta Lake solution Architecture and the operation on the Delta Lake needs to be done using SPARK. So today, we will discuss on the creation of the SPARK db that will serve as the layers in the Delta Lake. It is always good to segregate the layers to…
Let’s understand the concept of a Delta Lake Architecture so that later on we can go ahead and understand the concept of the Delta tables in the Delta Lake and their capabilities. The Delta Lake is optimized advance storage layer. The Delta Lake follows the medallion architecture where the tables are segregated in different categories…
This topic is quite rare because no one wants to delete their application or resource’s connection string secret from the key vault especially in PROD where the number of dependencies are quite high. But by mistake it can be mishandled. Always make sure that the below processes are followed as best practice: The main key…
