In this blog, we will discuss the movement of the Delta tables from one environment to another environment within a same subscription. There may be a scenario where you need to transfer Delta Table or the Lake Database from the upper environment to the lower environment and vice versa. Currently, there is no comprehensive documentation…
In our earlier blogs, we discussed the optimization of Delta tables through the use of OPTIMIZE and ZORDER commands. We have learned about their functionality and their impact on the performance of the tables. However, it is also important to consider the number of files that are invalidated when we run OPTIMIZE. As an example,…
Z-Order is an optimization technique used in the Delta Lake. However, before discussing what is Z-Order, we need to first understand what happens when we write data to a delta table. Whenever we write data, statics gets collected, which are used when querying the table. When filter is applied to the column, the statistics are…
In our last blog, we saw the workings of the Delta Table. Today, we have a topic that will help optimize our delta table. In the real time scenarios, the number of data files increases at a very high rate due to the multiple transactions performed on the table during day-to-day work. The size of…
We have seen the solution architecture of the delta lake in our Blog and how we can register a delta table in the Delta Lake. However, this blog will go around the working of the Delta table from the time you register and perform the DML operation on it. We will see internal architecture of…
We saw how to create a database in the synapse SPARK pool in our previous blog. Today we are going to understand the types of Delta table in SPARK. There are mainly two types of SPARK tables. There is slight difference while creating these tables, in a nutshell “Managed” table does not need any Location…
We have seen the overview of the Delta Lake solution Architecture and the operation on the Delta Lake needs to be done using SPARK. So today, we will discuss on the creation of the SPARK db that will serve as the layers in the Delta Lake. It is always good to segregate the layers to…
Let’s understand the concept of a Delta Lake Architecture so that later on we can go ahead and understand the concept of the Delta tables in the Delta Lake and their capabilities. The Delta Lake is optimized advance storage layer. The Delta Lake follows the medallion architecture where the tables are segregated in different categories…
