Week of June 17 2024 - Mindmap Recap

Search for a command to run...

No comments yet. Be the first to comment.
The importance of abstraction, reusability, error handling, efficient data manipulation, robust string handling, and performance optimization. Adopting these principles leads to cleaner, more maintainable, and high-performance code that becomes cruci...

As we accumulate library of sample and reference code through various Databricks notebooks finding the code is not easy. The Purpose behind this notebook is to find the code/text in the Databricks' workspace easier and navigate to it. You can downloa...
Streaming data can be enriched using the following scenarios: Static References Dynamic Data Sets Another Streaming Data Source This post covers two of the above scenarios: 1 and 2. First setup the streaming reader outputPath = f'{working_dir}/outp...
There are two kinds of temp views: Session based Global The temp views, once created, are not registered in the underlying metastore. The non-global (session) temp views are session based and are purged when the session ends. The global temp views ...

An issue was encountered with the crealytics:spark-excel library in Databricks. This Spark plugin is essential for reading and writing Excel files within Databricks. However, we observed that it tends to run out of memory when handling large datasets. This limitation necessitates exploring alternative libraries for better performance and reliability.
Alternative Libraries:
Apache POI with SXSSF: Ideal for streaming writes, providing efficient memory usage
Openpyxl
xlsxwriter
Databricks Runtimes (at the time of the writing of this post)
14.3 LTS: Now includes Apache Spark 3.5.0 and will be supported until February 1, 2027.
Old: 9.1 LTS: Includes Apache Spark 3.1.2, with support ending on September 23, 2024.
To check the current Spark version in your runtime, use:
Angular Flex Layout:
For those working with Angular, understanding the Flex Layout library is crucial for responsive designs. Key directives include:
fxLayout: Defines a flex container, e.g., row or column.
fxFlex: Specifies the size and growth of elements within a flex container.
The Flex Layout library is not included by default. To install and import it, run:
Then, import the module in AppModule:
Software Development:
Emphasizing use-case-driven development for efficient project execution.
TSQL Tips:
SQL Table Types (UDTT): Since you cannot pass record result sets to stored procedures directly, use UDTT to pass structured data.
UNION vs. UNION ALL: Use UNION to eliminate duplicates, and UNION ALL to include all rows.
Data Protection in Azure Storage Accounts:
Secure In-Transfer: Ensure all requests use HTTPS or secure SMB 3.0. HTTP requests will be rejected.
Secure At-Rest: Data is encrypted using server-side encryption (SSE), providing comprehensive data protection and compliance.
Angular Development:
ng build --prod: Command for production builds.
@HostListener: A decorator to listen and handle events on host elements of a directive or component without directly accessing the DOM.
SQL Server:
DTU (Database Transaction Units): Monitor and optimize database performance.
TSQL SELECT 1 FROM ...: Often used in EXISTS conditions or CASE statements for efficient querying.