Category: TalkingData

It seems that one mile distance isn’t very long, and that a cubic mile isn’t that big if compared with the size of the earth. You may be surprised if I tell you the entire world’s population can fit into a cubic mile of space. The statement is not from me, Hendrik Willem van Loon, a Dutch-American writer, once wrote this in his book. Teradata is a famous data warehouse product. Over 30 years ago, such a brand name aimed to impress people with its ability of handling massive amounts of data. Today, TB is already the smallest unit many

When learning SQL, we often hear the saying that SQL is a declarative language. You just need to tell it what to do, without telling it how to do it, it will find its own implementation method. That is to say, you need to use it only to describe the task objectives without explaining the computational process, which is fundamentally different from conventional procedural languages. Obviously, this programming language sounds much easier to learn and use. Is it really that good? Let’s take an example. We use SQL to query the number of female employees in the sales department. This

Many modern applications will move data computation and processing tasks away from databases and implement them in Java, which can gain framework benefits. Moreover, Java has comprehensive process processing capabilities and is more adept at handling increasingly complex business logic than SQL (although the code is not short). However, we often find that the performance of these Java codes in computing and processing data is not satisfactory, and they cannot even match the performance of SQL in the database. Normally, as a compiled language, Java may not be as good as C++in terms of performance, but it should have an

Early applications usually only connected to one database, and calculations were also performed in the database, with little or no problem of mixed calculations from multiple data sources. The data sources of modern applications have become very rich, and the same application may also access multiple data sources, such as various SQL and NoSQL databases, text/XLS, WebService/Restful, Kafka, Hadoop…. Mixed computing on multiple data sources is an unavoidable problem that needs to be addressed. Direct hard coding to implement in applications is very cumbersome, and commonly used application development languages such as Java are not good at doing such things.

Customer profiling is very fashionable in current business analysis. Simply put, it means putting various tags on customers, using these tags to define different customer groups (so-called profiles), and then calculating the quantity (and changes) of customers in each customer group. Logically speaking, tags are dimensions or fields of a data table. Tags are fields with relatively simple values, and there are generally two types: binary tags, which have only two values and are usually represented by 0/1, such as marital status or gender. Another type is enumeration tags, with values ranging from a few to a few hundred, which

JSON is a good thing that can carry rich structured data information in a common text format. Many modern technologies prefer to use JSON as a data transmission format, such as Elastic Search, Restful, Kafka, etc. Mongodb, which is more concerned about performance, uses binary JSON. Structured data is often in bulk and often requires recalculation. However, JSON related class libraries are not very convenient to use for calculations. JSONpath is fine to parse JSON, but it doesn’t have much computing power. Simple filtering and aggregation are fine, but it cannot handle slightly complex operations such as grouping and summarization.

It is almost universally accepted in the industry and has been successfully practiced for many years that when the TP database is too bloated, then add an AP database. Is there anything to discuss? The use of AP database can indeed alleviate the computational pressure of the TP database, and the AP database usually has better computational performance and can provide users with a better experience, which is indeed a common practice in the database field. However, it does not mean that this road is incredibly smooth. The first is the cost. The operation and maintenance of databases have never

That’s right, it’s the famous JOIN. JOIN has always been a major challenge in database computing, and the industry has come up with many ways to calculate it. If no optimization is done, it would be to loop through and traverse two associated tables, which is a multiplication level complexity that can be unbearable if the data volume is slightly larger. Mature databases are certainly not so foolish. For the most common equivalence JOIN (with an association condition of equal key values), the HASH JOIN method is usually used, which can reduce the computational complexity by K times (K is

It is the very inconspicuous account de-duplication count, written in SQL as COUNT (DISTINCT…). Account de-duplication count is common and has important business significance in business analysis. The account here may be a user ID, bank account, phone number, license plate number…. The calculation logic is basically the same, which is to calculate how many accounts meet a certain condition from the historical data of a certain period of time. For example, how many cars have been to NewYork last month? How many phones had calls between 2:00 am and 4:00 am last week? How many bank accounts have received

Oracle is a widely used database, and it is not a professional analytical database. When the data volume is large, it often results in poor computational performance, which affects the user experience. Nowadays, there are many new analytical databases that are faster or even far superior in performance than Oracle. So when Oracle performance is insufficient, replacing to these databases seems to solve the problem. The matter is not so simple. Firstly, these new databases generally cannot completely replace Oracle. Oracle is not only used for OLAP, but also for OLTP, and these faster analysis databases do not have the