Category: TalkingData

Hibernate, Mybatis, and emerging ORM technologies such as JOOQ can easily map database tables into Java objects and provide automatic read and write methods. ORM technology makes developing database applications in Java more efficient. However, mapping data tables is only the most basic part, and the development of business logic also involves a lot of process control and data calculation work. Process control is a strong point of Java, without pressure; However, batch structured data computing has always been a weakness of Java, and programming directly in Java can be very cumbersome. Unfortunately, the computing capability provided by these ORM

Distributed data warehouses (such as MPP) in the era of big data are a popular technology, even to the point where data warehouses are referred to as distributed. But is distributed data warehousing really necessary? After all, these distributed data warehouse products are not cheap, with high procurement and maintenance costs. Is there a low-cost and lightweight solution? In fact, the amount of data involved in structured data computing tasks (the main goal of data warehouses) is usually not very large. For example, a bank with tens of millions of accounts has a transaction volume of hundreds of millions of

The usage frequency of historical data often decreases over time, and how to store and calculate it becomes a problem. Low frequency of use does not mean that it is no longer used at all. If all these data are archived from the database, they must be imported into the database again when querying statistics, it is time-consuming and labor-intensive. There is often an awkward situation where the query takes only 3 minutes but the import takes 3 hours. It’s not feasible to keep these data occupying database space. Many enterprises use commercial databases, which are not cheap in terms

Microservices have been quite trendy in recent years, and using Java to replace SQL and stored procedures to develop business logic can indeed gain framework advantages. We will not delve into the details here. Of course, there is a reason why microservices can become popular. But are microservices really “micro”? We know that when faced with the same business logic, the code written in Java is much longer than SQL, about 10-20 times longer. Even with the assistance of new libraries and languages such as Stream, Kotlin, and Scala, when faced with slightly more complex computational requirements, the code length

ClickHouse has been popular in recent years as an analytic database. It is noted for the high speed and is often used as the benchmark by vendors of many other analytic databases whose selling points are high performance. Many users will consider turning to ClickHouse when they encounter database performance problems. ClickHouse has an outstanding feature. Its column-oriented wide table runs fast, thanks to the excellent compression probably. However, it does not have any other merits. Users who hope to use it to solve database performance problems will likely feel disappointed. We compare the TPC-H 100G performance of ClickHouse with

Java has not provided a direct dataset operation syntax for a long time, and writing a simple SUM requires many lines, let alone complex operations such as grouping and sorting. The code with exactly the same data processing function is much longer than SQL, resulting in low development efficiency. Starting from Java 8, Stream has been introduced, providing set operation classes that support Lambda syntax. Programmers no longer write many lines for SUM, and conventional grouping and sorting have ready-made functions. The sorting on the dataset is written in this way, which is indeed much more convenient than early Java:

At the beginning of information system construction, usually only one database is used, and the database combines TP (transaction processing) and AP (analytical processing) together. As the scale of business and the amount of data continue to grow, the database faces increasing pressure. In order not to affect transaction, a common practice in the industry is to move data (usually cold data) out of the database and use a dedicated database to handle AP business. This method effectively reduces the burden on TP database and ensures the smooth operation of transaction business. Normally, a professional AP database runs fast and

More use of Java to avoid stored procedures and complex SQL is a trend in current application development, which will bring advantages in architecture. However, implementing SQL like operations in Java is not very convenient, and many tasks need to be written from scratch, which actually reduces development efficiency. Hibernate/Mybatis/JOOQ and Stream/Kotlin, among other pure Java open source technologies, provide structured data objects and some computing class libraries, which can reduce development workload to a certain extent without compromising architectural advantages. However, the functionality of these open source technologies is still far from SQL, and when the requirements are slightly

The stored procedure is a love-hate thing. Its significance is obvious as all old-brand databases support it. Newcomer databases are often slighted if they do not support stored procedures. Yet, the stored procedure has noticeable shortcomings. Many development teams make painstaking efforts to remove stored procedures and try to implement all business logics in applications. The stored procedure is already at the crossroads. The stored procedure’s biggest advantage is to enhance SQL’s ability to compute in a stepwise fashion. SQL supports writing the whole logic only in a single statement. But certain logics need to be implemented step by step,

Not all SQL statements are migratable. Different databases have different functionalities. When the syntax of certain does not have their counterpart in another database, migration becomes impossible. If we do not use the databases’ special functionalities but only use their common functionalities – which actually are a rich collection – within the range of the international standards, does the migration work? Basically, yes. For example, the following statement Works normally for all databases. However, there are still many SQL functions whose syntax vares with databases, particularly string and date-related functions, for example: The international standards do not impose any rules