In this blog, we will discuss the movement of the Delta tables from one environment to another environment within a same subscription. There may be a scenario where you need to transfer Delta Table or the Lake Database from the upper environment to the lower environment and vice versa. Currently, there is no comprehensive documentation…
Z-Order is an optimization technique used in the Delta Lake. However, before discussing what is Z-Order, we need to first understand what happens when we write data to a delta table. Whenever we write data, statics gets collected, which are used when querying the table. When filter is applied to the column, the statistics are…
In our last blog, we saw the workings of the Delta Table. Today, we have a topic that will help optimize our delta table. In the real time scenarios, the number of data files increases at a very high rate due to the multiple transactions performed on the table during day-to-day work. The size of…
We have seen the overview of the Delta Lake solution Architecture and the operation on the Delta Lake needs to be done using SPARK. So today, we will discuss on the creation of the SPARK db that will serve as the layers in the Delta Lake. It is always good to segregate the layers to…
Let’s understand the concept of a Delta Lake Architecture so that later on we can go ahead and understand the concept of the Delta tables in the Delta Lake and their capabilities. The Delta Lake is optimized advance storage layer. The Delta Lake follows the medallion architecture where the tables are segregated in different categories…
We have seen the implementation of the Lake Database in our previous blog, and we will see that the object created using the SPARK in the Lake Database can be easily queried using the Serverless SQL Pool. This gives a benefit that the tables can be queried on the Ad-hoc basis without turning the SPARK…
The Lake Database can be expressed as Database on top of a Storage which got added to the Synapse workspace Studio in 2022 as a new feature by Microsoft. At first glance, if you read about it then it seems a database which you can create from a template that is prebuilt and exposed to the user from the variety of industries. Well, that’s true that you can just start to design your own data model on top of the Data Lake and it gives you a GUI to map the relationship between the tables you want to create on top of your Azure Storage using Database Designer.
