Tag: Data Source

InfluxDB is a time series database, where data is stored in its bucket and multiple buckets form an organization. Every piece of data consists of measurement, multiple dimensions, multiple filed values, and a timestamp:

Kafka is a distributed flow information platform, of which there are two kinds of clients, producers and consumers respectively. The former submit data, while the latter read data. There is also no fixed format for the event message delivered in Kafka.

Cassandra is a distributed NoSQL database. In JAVA, DataStax can be used to execute the CQL of Cassandra. This language can maintain and access data in a simple way, but does not support operations such as association, grouping and aggregation, etc.

FTP is a commonly-used file transfer service. Although JAVA provides APIs for reading and writing files from an FTP server, it is not convenient enough to use. Therefore, SPL provides some functions for further convenience.

Redis is a K-V database, in which values can be String, Hashmap, List, Set, and Sortedset. There are a variety of operating commands on each type of value, which amount to over three hundred with some administrative commands in total. However, these commands return the results of different formats and can only perform some simple operations such as accessing data.

HBase is the K-V database based on HDFS with similar concepts to the relational database: namespace, table, row, column, field value, primary key and so on.

Hadoop Distributed File System (HDFS) provides the Java API to read and write its files, and esProc encapsulates the JAVA methods for reading and writing HDFS files to easier-to-use SPL functions. With these functions, HDFS file can be loaded directly into esProc for calculation. This article describes in detail the usage of the functions.

MongoDB is a non-relational database storing data in BSON format, which is similar to JSON format, and it provides a full set of commands to process data. SPL, the lightweight computation engine language, supports nested data structure and is very easy to load JSON data. Therefore, we can extend the computational abilities of MongoDB with SPL. SPL provides the mongo_shell function, through which we can execute Mongo commands to read and write Mongo data.

The SPL provides the httpfile function to access webpages, WebService and Restful services on HTTP server, passing parameters required by the service to get the return data or download files.

XML is a commonly used data format. The format features multilevel nodes. In each node, you can define multiple pairs of attribute values. The structure is more complicated than that of a two-dimensional database table. There are difficulties if we are trying to compute XML directly. SPL (Structured Process Language) provides xml() function to parse and generate XML. It is convenient to arrange and compute data using it. Parsing XML Basic data types XML does not define explicit data types, so we treat both tag values and attribute values as strings. In the following person.xml, for instance, root is the