Category: 1 SPL&Code

The screening method is an ancient way of finding all prime numbers that do not exceed the natural number N (N>1). Arrange natural numbers 1 to N. The smallest prime number is 2, so first cross out 1. Starting from 2, leave 2, and cross out all numbers after 2 that can be divided by 2. Then the first number after 2 that is not crossed out is 3, leave 3 behind, and then cross out all numbers after 3 that can be divided by 3. The first number after 3 that is not crossed out is 5. Leave 5

Columnar storage is an important technique to improve data processing & analysis. If a data table has a large number of columns but the processing/analysis only involves a very small number of them, columnar storage lets you read the target columns only. This helps reduce disk accesses and boost performance. One column generally contains data of the same type. Sometimes it even has values that are close to each other. By storing such data in order, it can be compressed more efficiently. But setting up a column-oriented data warehouse for use in real-world applications is too complicated. All the commonly

Using csv files to store data is very common. Similar formats include text files, tsv files, etc. These files all have a plain structure—no hierarchy—and store data as text characters. Some xls files also have a similar plain structure. CSV files have characteristics such as simplicity and strong versatility, but they also have significant drawbacks. Text characters cannot directly participate in calculations, and parsing them to in-memory data types such as dates and real numbers is a very time-consuming process, especially when checking for invalid cases, which makes it even more complex. Therefore, the performance of csv files is consistently

Using SQL for data anlysis is actually quite basic SQL is widely used for data analysis and often considered a default skill of data analysts. Indeed, being able to write SQL in a database environment is very convenient; it might seem as if you can solve any query by just writing a single SQL statement. For example, when you want to find sales grouped by user, the SQL code written out is as simple as English: However, data analysis tasks are not always that simple. For example, to calculate the next-day retention rate of new users, the SQL code will

The puzzles facing Excel data analysts In the world of data, data analysts who work with Excel are like a group of “knights holding the ‘Excalibur’”. Relying on Excel’s powerful interactive ability and ease of use, Excel knights cut through obstacles easily and perform data analysis efficiently. However, potential challenges are surging continuously in the said world. As the scale of data continues to grow and the requirements for data analysis are increasingly complex, relying solely on the ‘Excalibur’ seems insufficient to cope with the ever-emerging challenges. Excel is good, but it seems powerless when faced with complex ‘data enemies.’

Talk is cheap; let’s show the codes 1. User Session Count User behavior data table userid action_type action_time U1059 login 2023-12-01 18:00:10 U1092 login 2023-12-01 18:00:17 U1069 login 2023-12-01 18:00:22 … … … A session is considered over if a user does not take any action within 10 minutes, or if they do not log in within 5 minutes after logging out. Calculate the number of sessions for each user. SPL A 1 =file(“session_data.csv”).import@tc() 2 =A1.group(userid;~.group@i((action_type[-1]==“exit”&&interval@s(action_time[-1],action_time)>300)||(interval@s(action_time[-1],action_time)>600)).len():session_num) SQL Python 2. Count the players who score 3 times in a row within 1 minute Score table of a ball game team player

Yes! You read that right. SPL (abbreviation of Structured Process Language) is an open-source programming language that codes in a grid and that specifically designed to process structured data.As almost all programming languages code in the form of text, what does the SPL code written in the grid like? And what differences does it make? Let’s first take a look at SPL’s programming environment. Features of SPL Code written in the grid In the above screenshot, the middle part is SPL’s grid code. But what are the advantages of writing code in the grid?Coding always involves intermediate variables, to which

SQL is wasting lives of data scientists SQL is difficult to write Nearly all data scientists use SQL for data exploration and analysis. SQL appears deceptively simple and offers a certain degree of interactivity, making it a seemingly ideal choice for these purposes. To perform filtering and grouping operations, for example, SQL just needs one line of code: But this is only limited to simple cases. When the computation becomes complex, SQL code becomes complicated, too. For example, to count the longest consecutive rising days of each stock, the SQL statement is as follows: And to perform the commonly seen

Global variable In SSC environment, sometimes information needs to be shared between computing tasks (QVM scripts) on multiple QVMs. In this case, SSC global variable (simply written as global variable in the following part) is useful. Managed by QVS, the global variable is available to every script run on each QVM applied through QVS. Qenv function is used to assign value to the global variable: Qenv(“gv1”,1) #gv1 is the variable name Qenv(“gv1”) The function is also used to obtain the global variable: Qenv(“gv1”) Unlike an ordinary SPL global variable, related data will be transmitted between QVS and QVM once the

An application on SSC has two parts: the server side, which consists of QVA and QVM, and the application side, which consists of the application itself (simply written as APP in the following part) and QVS. Both parts involve the use of SPL scripts. The SPL script on the server side is executed on QVM and thus called QVM script. On the application side, another SPL script is written to call the QVM script and it is called APP script. The basic procedure is like this: APP calls the APP script that sends a computation request to QVS; QVS then