Category: 2-SPL-Application-Development
How to Implement Hot-Cold Data Mixed Computing with esProc
Separating hot and cold data into different databases makes performing mixed queries quite troublesome. Many databases lack support for such queries; those with partial support offer limited capabilities and often require extensive data replication, resulting in low efficiency. Separately retrieving the cold and hot data and performing calculations within Java would be overly complex. esProc, a database-independent computing engine, supports various data sources, provides rich computation libraries, and can be embedded in Java, making it ideally suited for performing hot-cold database hybrid computation in applications. First, download esProc (Standard Edition recommended): https://www.esproc.com/download-esproc After installation, configure the database connection. Here, we
How to Implement Mixed Computations Between Oracle and MySQL with esProc
Logical data warehouses can implement multi-source mixed computations, but require configuring views and preprocessing data, resulting in an overly heavyweight architecture. DuckDB offers a lightweight solution but lacks a native Oracle connector, rendering custom development too complex. esProc, as another lightweight solution, supports the common JDBC interface, enabling mixed computations between any RDBs. First, download and install esProc (Standard Edition recommended): https://www.esproc.com/download-esproc/ Place the database’s JDBC driver in the directory ‘[Installation Directory]\common\jdbc,’ which is one of esProc’s classpath directories. The following are the JDBC drivers for MySQL and Oracle: Open the esProc IDE. From the menu bar, select Tool ->
How to Implement Merge Operations Across Tables from Multiple Databases with esProc
Due to business requirements, data is stored by year in two structurally identical databases. To perform statistical analysis on this data, it will involve cross-database computation. Implementing this directly through databases or hardcoding is cumbersome. esProc simplifies such operations. Data Structure of orders table: Where order_id is the primary key. Sample data: Now, we want to merge and calculate data from the two databases. How can we do this using esProc? Installing esProc First, download esProc Standard Edition at https://www.esproc.com/download-esproc/ After installation, configure a connection to the MySQL database. Begin by placing the MySQL JDBC driver package in the directory
How to Operate nested JSON from RESTful with esProc
After receiving JSON from RESTful services, further calculations are often required. Processing such data in Java is cumbersome. In contrast, esProc simplifies this significantly, as it can connect to RESTful interfaces, excels at processing multi-layered data, and can be embedded in Java applications as an in-application computing engine. First, download esProc Standard Edition at https://www.esproc.com/download-esproc/. After installation, try accessing RESTful data. Accessing RESTful Direct access REST services offer various data access interfaces. For example, accessing http://192.168.2.52:8503/orders can retrieve order data. The Orders data uses nested JSON to store order, product, and related information, as structured below: Write an esProc script
How to query CSV as a database table in Java with esProc
The code for calculating CSV files in Java is too cumbersome, and relying on databases can lead to complex structures. esProc provides JDBC drivers and computation libraries, which can embed SPL statements in Java and directly query CSV files as data tables, making it much more convenient.Download esProc first, recommend standard edition: https://www.esproc.com/download-esproc/Download and install the corresponding version.Before configuring the Java environment, try whether the installation of esProc is successful. Prepare a standard CSV file: Open the esProc IDE, create a new script, and write the SPL statement in cell A1 to read the CSV file: Press ctrl-F9 to execute,
How to operate large CSV files with esProc SPL
esProc SPL provides cursor operations, which can operate large CSV files with very simple code. With slight modifications, it can be converted into parallel computing. It also has a graphical interface, which is much more convenient than Python.First, go here to download esProc SPL: https://www.esproc.com/download-esproc/If you don’t want to bother with the source code, the standard edition can be used., download and install it.Prepare a large CSV file: Open the esProc IDE, create a new script, write SPL code in cell A1, and read the first 100 entries: The function cursor represents opening the text file with a cursor, and
How to execute SQL on CSV files with esProc SPL
esProc SPL supports simple SQL and can directly execute SQL statements on structured text files such as CSV, allowing for SQL calculations without the need for a database.Download esProc SPL first: https://www.esproc.com/download-esproc/If you don’t want to deal with the source code, you can simply download and install the standard edition.Prepare a CSV file of standard format. Start esProc IDE, create a new script, and write SQL in cell A1: Note that the preceding $ indicates that this statement is SQL, otherwise it is assumed to be a native SPL statement by default.Press Ctrl-F9 to execute, and on the right side,
This Is Likely the Computing Technology that Supports the Most Data Sources
Today, the data sources of enterprises have evolved from “a few tables” to a diverse range such as databases, files, APIs, streaming data, object storage and NoSQL. The ability to handle “multi-source computation” has become one of the critical criteria for data processing technologies. When it comes to multi-source computation, the ‘logical data warehouse’ is probably the most mainstream approach. And it sounds very appealing: no need to synchronize data in advance, no need to struggle with traditional ETL, and the ability to perform cross-database queries using SQL. However, reality falls short of ideals. These logical data warehouses, despite claiming
This Is Probably the Most Lightweight Alternative Technology to Logical Data Warehouses
Contemporary business analytics applications (such as common reporting applications) often need to extract data from different systems. The data are stored in different locations, have different formats, use different storage formats, and are updated at different intervals. They can be sales data in CRM, financial data in the accounting system, inventory information in ERP, and even external data in the cloud. These data sources are like jigsaw puzzle pieces, which have different shapes and contain different types of information. And how to put them together appropriately to fit for data analytics has been a constant challenge in terms of efficiency
