June 17, 2024: Databricks - Issues with Excel Library in Clusters An issue was encountered with the crealytics:spark-excel library in Databricks. This Spark plugin is essential for reading and writing Excel files within Databricks. However, we observ...
The importance of abstraction, reusability, error handling, efficient data manipulation, robust string handling, and performance optimization. Adopting these principles leads to cleaner, more maintainable, and high-performance code that becomes cruci...
As we accumulate library of sample and reference code through various Databricks notebooks finding the code is not easy. The Purpose behind this notebook is to find the code/text in the Databricks' workspace easier and navigate to it. You can downloa...
Streaming data can be enriched using the following scenarios: Static References Dynamic Data Sets Another Streaming Data Source This post covers two of the above scenarios: 1 and 2. First setup the streaming reader outputPath = f'{working_dir}/outp...
There are two kinds of temp views: Session based Global The temp views, once created, are not registered in the underlying metastore. The non-global (session) temp views are session based and are purged when the session ends. The global temp views ...
Delta Lake Technologies A Delta Table is a collection of components kept together using Delta Lake technologies: Delta files (Parquet files) Delta transaction logs (stored in object storage) The metastore (optional) Delta Transaction Logs The logs ...
