Tag: Technical Subject

SPL variables have three scopes: local variable, global variable, and job variable. The default variables are all local variable, and the scope is within the scope of the same SPL script (excluding the main sub thread code). The usage is relatively simple, which is not covered in this article. The following will focus on the other two variables and their related locks. Global variable The scope of the global variable in SPL is the life cycle of the same JVM (computing node\ SPL service), which can be accessed in all SPL scripts run on this JVM. The global variable is

In addition to common static code, sometimes dynamic code is also needed to solve problems, such as generating code (or part of code) based on parameters and dynamically executing it. For programming languages that lack dynamic coding mechanisms, it is usually necessary to write the variable parts of the code in string form. For example, when referencing dataset field names in Python, it is necessary to write them as strings to achieve the effect of dynamic code. However, this will make it inconvenient to read and write more common static code. SQL, on the other hand, can directly write field

The main goal of Lambda syntax is to quickly and conveniently define and use temporary functions, reduce code amount, and improve development efficiency. SPL focuses on the field of structured data computing and often faces complex computational logic and lengthy computational code. In order to better adapt to specific application environments, SPL makes appropriate modifications to Lambda syntax. Loop variable The conventional Lambda syntax does not have built-in loop variables, and loop variables are used almost everywhere, and programmers need to define them every time. Example 1: To filter out even numbers in the set of integers A=[2,1,2], the conventional

Based on an ordered set, multiple operations can be extended, with the simplest being access by position, i.e. retrieving members of a record set by subscript or position. On the other hand, there is also positioning operation, which obtains the position of members in the record set in a certain way, such as the position of the record where the field Max/Min value is located, the position of records that meet certain conditions, and the position of sorted records in the original set, etc. The variety and powerful functions of positioning operations enhance and expand the expressive ability of structured

Sequence number is a natural number that increases sequentially from 1 (or 0), and can be used as the index of array and sequence, the row number of data set and table sequence, and the natural number field. SQL doesn’t support sequence number well. The record set (table), as SQL’s basic data type, is unordered, and has no row number, and hence SQL programmers do not have the awareness of actively using sequence number to solve problems. When row number has to be used in special cases, it needs to be temporarily generated witha function or pseudo-field such as row_number()

Partial pre-aggregation The backend operation of the multi-dimensional analysis is essentially the grouping and aggregating, which can be performed directly on the data. However, when the amount of data is very large, it is difficult to achieve an instant response. To solve this problem, an easy way to think of is the pre-aggregation, that is, calculate every aggregation result in advance and save them as an intermediate result (we will hereinafter call it cube). When the front-end application requests, we can directly search the cube and return the result. For example, pre-aggregate the order table in Figure 1 below: In

When associating the big data tables, conditional filtering is sometimes performed on the associated tables first. For the case where the associated fields are the primary keys or part of primary keys, SPL provides the association location algorithm on the basis of the ordered merge association algorithm (refer to: SPL Order-based Merge Join ）to improve the computing performance of the association after filtering. Let’s first look at the case where the associated tables are joined by all primary keys (we call such tables are the homo-dimension table from each other). Assuming that there are two large tables, table a and

In a large-table association scenario, if two tables are associated through their primary keys or by relating the primary key in one table to part of the primary key in the other table, we can optimize performance using SPL attached table. The storage mechanism combines tables to be associated and stores them together. Take customer table and contact table as an example, below shows their relationship: The two tables are associated through their primary keys. For two tables related in this way, each one is the homo-dimension table of the other. Below is the structure of the stored customer table

It is no rare thing that we are handling an association between very large tables where performance becomes an issue. SPL specifically offers an order-based merge algorithm for primary-key-based associations and associations based on the part of a composite primary key. The order-based merge algorithm requires that every table to be associated be sorted by their primary key in advance. The sorting is costly but once-for-all, always letting us use the algorithm to perform the same type of associations later and thus increasing performance considerably. Learn how to achieve order-based storage in SPL Ordered Storage. Order-based merge join Suppose we

In big data computing scenarios, many complicated algorithms produce concrete intermediate results that result in performance-reducing hard disk reads/writes. Here’s an example. The account trading table trades has three fields – id (account id), dt (trading date) and amount (trading amount), and we are trying to find records having transactions for n days in a row and sum amounts by the weekday when the transaction happens. Let’s call the first half of the computation compute1 and the second half compute2. For compute1, we sort records by account and trading date and store the sorted data before we start to get