Databricks Temp Views and Caching

Search for a command to run...

No comments yet. Be the first to comment.
June 17, 2024: Databricks - Issues with Excel Library in Clusters An issue was encountered with the crealytics:spark-excel library in Databricks. This Spark plugin is essential for reading and writing Excel files within Databricks. However, we observ...

The importance of abstraction, reusability, error handling, efficient data manipulation, robust string handling, and performance optimization. Adopting these principles leads to cleaner, more maintainable, and high-performance code that becomes cruci...

As we accumulate library of sample and reference code through various Databricks notebooks finding the code is not easy. The Purpose behind this notebook is to find the code/text in the Databricks' workspace easier and navigate to it. You can downloa...
Streaming data can be enriched using the following scenarios: Static References Dynamic Data Sets Another Streaming Data Source This post covers two of the above scenarios: 1 and 2. First setup the streaming reader outputPath = f'{working_dir}/outp...
There are two kinds of temp views:
The temp views, once created, are not registered in the underlying metastore. The non-global (session) temp views are session based and are purged when the session ends.
The global temp views are stored in system preserved temporary database called global_temp.
There are two ways to created a temp view from a DataFrame:
# Python
spark.read \
.format("delta") \
.load(batch_source_path) \
.createOrReplaceTempView(batch_temp_view)
# .createOrReplaceGlobalTempView(batch_temp_view)
The Delta Engine gains some of the optimization through the caching layer that sits between the execution layer and the cloud object store.
There are also two ways to cache a temp view:
# Python
# Cache using the spark catalog
spark.catalog.cacheTable(batch_temp_view)
To cache a DataFrame object
# Python
df = spark.read \
.format("delta") \
.load(batch_source_path)
df.cache()