Today, we are going to look into one of the commonly faced issues in the field of Data Engineering i.e. slow running queries in the production environment. I encountered this issue in production where the queries were running over 13 hours whereas initially used to take 22-25 mins. Use case: One weekend, we saw the…
In today’s blog, we will explore how to call a Logic App using Python code. I have a scenario where I need to send an email to the desired recipient with some information in the middle of the data processing. This can also be achieved by using the SMTP connection in Python for sending email….
Hope you all are doing good, Today’s blog we will be diving into the PowerShell Script that we will use to toggle any Triggers ON and OFF within the synapse workspace using the DevOps Pipeline. This script is tailored to toggle the specific triggers OFF and ON from the DevOps pipeline pre and post deployment…
FileDataLakeClient is the python package to communicate with the azure Hierarchical Namespace enabled storage(ADLS GEN2). This API is added to the storage SDK specifically for ADLS Gen2. This package can be used to do many operations like (creating, deleting, renaming) filesystem, Directory, Files, lease and many more like setting ACLs, acquiring Leases etc. The containers…
Today, I encountered an issue with the Production Self-Hosted Integration Runtime (SHIR) that involved a version mismatch. In our production environment, we typically utilize multiple nodes of the Integration Runtime to ingest data from various sources. Microsoft frequently releases upgrades for the Integration Runtime, containing fixes and additional functionalities. If you have enabled auto-update for…
Today, we are going to learn how to refresh an Azure Analysis Service Models from the Azure Synapse pipeline. There may be scenarios where you want to refresh the AAS model dependent on the Azure ETL pipeline runs. It is crucial to initiate the refresh as soon as the data load is completed to reduce…
In this blog, we will discuss the movement of the Delta tables from one environment to another environment within a same subscription. There may be a scenario where you need to transfer Delta Table or the Lake Database from the upper environment to the lower environment and vice versa. Currently, there is no comprehensive documentation…
In our earlier blogs, we discussed the optimization of Delta tables through the use of OPTIMIZE and ZORDER commands. We have learned about their functionality and their impact on the performance of the tables. However, it is also important to consider the number of files that are invalidated when we run OPTIMIZE. As an example,…
Z-Order is an optimization technique used in the Delta Lake. However, before discussing what is Z-Order, we need to first understand what happens when we write data to a delta table. Whenever we write data, statics gets collected, which are used when querying the table. When filter is applied to the column, the statistics are…
In our last blog, we saw the workings of the Delta Table. Today, we have a topic that will help optimize our delta table. In the real time scenarios, the number of data files increases at a very high rate due to the multiple transactions performed on the table during day-to-day work. The size of…
