Category: 2-SPL-Application-Development

Logical DW (data warehouse) offers users the ability to logically integrate a variety of different type of data sources without moving the original data, presenting itself as a physical DW. Logical DW can address traditional DW’s problem of the inability to respond to real-time data processing needs due to long data chain caused by data moving, and hence logical DW can well meet the fast business change scenarios. Moreover, logical DW has the cross-source computing ability. However, due to the absence of physical storage, logical DW needs to map the data of each source as SQL table so as to

In order to obtain better computing performance, the MPP databases, such as Greenplum, Vertica, IQ, TD and AsterData, are often adopted. Although MPP can achieve better performance, the cost is high. Specifically, MPP consumes a large amount of hardware resources, resulting in high hardware cost, and it needs to pay expensive license fee if a commercial software is used. Moreover, it is very complicated to operate and maintain MPP, as each node needs to be maintained separately, and the uniform distribution and the consistency assurance of data under distributed framework will increase the O&M complexity. In short, it is heavy

Usually, the streaming data sources are dynamic and unbounded, and appear quite different from the static and bounded batch data source. For framework reasons, it is difficult for traditional database technologies to directly process streaming data source, so programmers have to resort to later technologies. The computing frameworks such as heron\samza\storm\spark\flink were the first to make breakthroughs and gained first-mover advantage in stream computing technology. These frameworks are so successful that as soon as a stream computing is involved, the application programmers will naturally turn to one of them. On the contrary, for those computing technologies that do not claim

Q1. Operating environment esProc is currently a pure Java software that can run on any operating system with JVM of JDK1.8 or above version, including common VMs and Containers. After normal installation, esProc occupies less than 1GB of space, but the vast majority are referenced third-party external data source driver packages. Its core package is less than 15M and can even run on Android. Except for the JVM, esProc has no other special requirements for the operating environment. The capacity requirements for hard disk and memory are related to computing tasks, and different computing tasks vary greatly. When performing the

We know that the early databases do not distinguish between TP and AP, and all tasks are handled in one database. When dealing with TP business, it is important to ensure the consistency of data, and consistency makes sense only when the data are limited within a certain range, which gives rise to the concept of “base”. The data to be loaded into database should meet some constraints, otherwise it cannot be loaded. There is a clear distinction between the data inside and outside the database, and this characteristic is called the closedness. In addition to guaranteeing the consistency of

Currently the vast majority of data warehouses employ SQL to process data. Following decades of development, SQL has become the standard language in the database world, and amassed a large user population, so it is normal to support SQL for data warehouses. However, in the context of contemporary big data, as the business complexity keeps increasing, the abilities of SQL seem to be increasingly inadequate in the data warehouse scenario where computation is the primary task. A typical manifestation is that some data warehouses begin to integrate non-SQL languages like Python. Not to mention whether the two languages with very

The data interfaces on the Web are mainly restful and WebService, and the data formats usually are hierarchical JSON and XML. Hierarchical data can carry more general and richer information, yet it is more complex in structure and more difficult to compute than traditional two-dimensional data. There seem to be many tools or engines available for Web computing, but each has its own shortcomings. Specifically, the libraries like JsonPath/XPath are good at parsing but not good at computing; Python Pandas is relatively powerful in computing, but it is difficult to integrate it into Java and, since its data object DataFrame

The files of open formats like txt\csv\json\xml\xls are commonly used in our daily work, and need to be calculated and processed sometimes. There seem to be many tools that can calculate and process such files, but in reality, these tools all have their own shortcomings. Specifically, the libraries like OpenCSV\JsonPath are good at parsing but weak in computing; spark is relatively strong in computing, but too heavy in framework and steep in learning curve; embedded databases such as SQLite\HSQLDB have light framework, yet the process of loading data into database is long and complicated, and delay would be fatal; Other

Many small and micro applications also require certain data processing and computing abilities. Integrating a database will make the application too heavy. In this case, SQLite is a good choice for its simple framework, easy-integration, ability of storing data persistently and, it provides SQL to implement calculation. However, for some more complex scenarios, SQLite still falls short. The shortcomings of SQLite in the face of complex scenarios Data source support SQLite is like a database, and can provide good support for its own database files, but sometimes applications need to process other forms of data, such as text files, Excel,

Data business in applications usually involves access to persistent data, data calculation and flow processing. The persistent data in database can be calculated in SQL, and the loop/if statement of stored procedures can perform flow processing, and JDBC (including ODBC) can integrate SQL in applications. Therefore, complex SQL (including stored procedures) is often used for data business development. However, since complex SQL is deeply bound to database, there are defects in framework, resulting in a failure to meet the requirements of modern applications. The defects of complex SQL (and stored procedures) Difficult to scale When using complex SQL or stored