How Twitter and Machine Learning (KDE + LDA) help to predict Crime?

He (Gerber) uses, in addition to KDE, topic modeling on these messages, in particular LDA.

As a brief summary, LDA is a generative probabilistic topic model method that discovers a preset number of topics that pervade a text corpus.

The number of topics that we would like to extract represent a model hyperparameter (K) and needs to be provided a priori.

These K-topics are arranged in descending order of probability of occurrence, and in this case the corpus is represented by all the tweets messages considered.

For instance, if a person twits from the airport, most likely will include words such as flight, plane, gate, airline, american, …, and therefore we can infer that the tweet originates from the geo-location provided by the twit itself, in this case the location takes place from the airport (in the region under study).

In the case for locating a crime, let’s suppose that the highest probability topic (among 500) for the prostitution crime is, “lounge studios continental village ukrainian …”, then the location provided by the twitter message will indicate where such crime may be taking place.

Figure 1.

Division of Chicago Illinois USA city neighborhood boundaries for tweet-based topics, only the green cells were considered in Gerber’s analysis.

In his paper, Gerber considers two ML techniques: KDE + LDA on Twitter.

For performing his analysis, he considers the city of Chicago, Illinois, U.



where he has divided the city into square-cells (neighborhoods) of 1000 meters by 1000 meters (See Fig.


How does this method works?The functioning of the predicting method is as follows: For each neighborhood the tweets posted undergo topic modeling using LDA.

LDA will contribute with the highest probability of the K tweet-topics, and the twit carries the geo-location of the twit itself.

At the same time, KDE will quantify the historical density of the crimes (by type and cell).

By combining KDE and Twitter contents treated with LDA, he formulated a prediction model, and after its application Gerber observed that in the majority of the crime types, an improvement in the prediction accuracy was attained.

In order to present the gains and loses that he discovered in the predicting accuracy of this technique, I have summarized them in 3 schematic representations for 3 types of crimes (B, W and S), where type B represents those crimes where a better prediction accuracy (than using solely KDE) was achieved, Gerber claims having obtained an improvement of up to 0.

10% in AUC (See Fig 2.

), type W for those crimes where the prediction accuracy was worse (See Fig 3.

) and type S crimes, where the prediction accuracy was similar (See Fig.


I have arbitrarily selected from Gerber’s findings the following ranks for classifying the crimes according to its type (see Table 1).

Table 1Figure 2Figure 3Figure 4For clarity, the three figures are presented all together (See Fig.

5), as explained, his model reports gains in prediction accuracy for crimes type B, a poorer prediction accuracy for crimes type P, and for crimes type S, a similar prediction accuracy than using only KDE.

Figure 5The following images show (a) a heat map generated by using only KDE, while (b) shows a heat map generated using KDE + Twitter topic features, where we can appreciate a more defined and denser prediction, that even resembles the neighborhood division (cells) that has been made for analyzing the region.

See Figure 6.

Figure 6In my opinion, this paper represents a good effort for improving the crime prediction technique (based solely KDE), by adding Twitter analysis using LDA.

However, the gains in prediction accuracy that the author claim do not extend to every crime type.

The author recognizes that it is difficult to explain why some type crimes benefited more or less from the addition of Twitter topics, maybe this would be a question for further research on this matter.

Along with the geo-location of the Twitter messages perhaps other features might be included to improve the prediction accuracy of crimes, such as the Twitter sentiment, or even influences crime, like the weather?Further sources:Predicting crime using Twitter and kernel density estimationMS Gerber — Decision Support Systems, 2014 — Elsevier[BOOK] Density estimation for statistics and data analysisBW Silverman — 2018 — taylorfrancis.

com[PDF] Speeding up calibration of Latent Dirichlet Allocation model to improve topic analysis in Software EngineeringJA López — 2017 — digital.



caAbout the author of this article: https://www.


com/.. More details

Leave a Reply