Tag: Technical Subject

Principle of multipurpose traversal When traversing the data table on external storage, most of the time is spent on reading data from the hard disk. Therefore, we would hope to do as many things as possible over a single reading, that is, the data read out during the traversal process can be used to the maximum extent. For example, we hope to perform two calculations when grouping and aggregating the orders table. The first one (compute1) is to count the sales by product group to get the result (result1), and the second one (compute2) is to count the maximum order

In structured data query, the requirement for full-text retrieval often arises. For example, we need to search for the countries whose name starts with Ch in the country table, or search for the records that contain water in the “content” field of “post” table. Although the search-oriented full-text retrieval can solve this problem, it is not suitable for structured data computing task for the reason that it will face the large-scale web page with huge amount of data, and hence the index is also huge, and it may also involve the understanding of very professional natural language and the word

Established outside the original table, the external storage index is an association table between the value of to-be-searched (TBS) field and the physical position of original table record. When searching, we can use a specified value to quickly find the physical position of original table from the association table, and then read the original table record. Thus, the index will store many TBS field values. To find the specified value in these values, the search action originally used for the original table will also be utilized. Therefore, we need to design a special index structure to improve the search speed

The index is similar to the “content” of the original table, which is a storage structure additionally created outside the original table. When searching and computing, we can first search the index to find the location of the original table in the “content”, and then go to the original table to find the corresponding record. The more fast the index query is than the original table query, the more obvious the speedup effect of the index query will be. Index creation is a relatively independent process, which can be done in one go during the system initialization, and we only

When two large tables are associated, performance problems often occur. One of the more common situations is to use the non-primary key field of one large table to associate the primary key of the other. For example, the order table orders and the customer table customer are both large, and the memory cannot hold them. Using the customer No. cid field of order table (hereinafter referred to as fact table) to associate the primary key of customer table (hereinafter referred to as dimension table), the size relationship between these two tables and the memory can be intuitively shown in Figure 1: In

Ordered storage is to write the data into external storage (mainly refers to the hard disk) continuously after sorting them by some fields (usually refer to the primary keys or part of primary keys). Not only can the ordered storage achieve low-cost data compression, but it can also avoid reading the hard disk in a frequent jumping manner. Moreover, since the data continuously read from hard disk during calculation are already ordered, it can implement high-performance algorithms designed specifically for ordered data, such as the binary search, fast filtering, ordered grouping, ordered association. In this article, we will introduce the

Big data usually requires external storage. To implement parallel computing in external storage, there must be better data segmentation technology, i.e., a technology that can easily divide the data into several parts so that each thread or process can process separately. SPL adopts the double increment segmentation technology, which achieves the appendable blocking scheme on a single file, and has a particularly prominent advantage in parallel computing. In this article, we will analyze the principle of this technology in detail by simulating the process of writing data to a binary file, and will focus mainly on multi-thread cases, as the multi-process is similar. To implement