Data visualization matters: a quick review on alternatives to plot market dataM.
EmmanuelBlockedUnblockFollowFollowingMar 29Data visualization is highly overlooked in quantitative trading.
In an area dominated by algorithms, it seems that visual inspection of the data is no longer relevant and that it belongs to discretionary trading, but in fact, it is one of the key differences between a proper analysis and an amateur approach.
Data visualization is also a broad area, and it covers several chart and plot types.
In financial markets, there is plenty of room to deal with classic histograms and scatter plots diagrams (those would be the most useful ones according to my experience).
Plotting such graphs is extremely easy and do not pose any special need, as almost any plot/chart library can easily cope with them.
This post deals specifically with the plot of market time series, which are usually a bar or candlestick charts covering open, high, low and close prices — plus volume and/or open interest — .
The nature of financial markets as pure time series creates specific needs and particularities which are not found in any other areas.
Time series plots — which as we mentioned will be mainly candlestick and bar charts — have a major role in market speculation and investing.
They normalize the way we present and analyse asset price evolution.
It is specifically relevant in technical price analysis where most of the analysis relies on the specific patterns and trends that can be found in charts.
«In quantitative trading, data visualization is easily overlooked as all focus is put on algorithms.
It is a huge mistake, as data visualization plays an extremely important role in both defining and validating strategies.
»In this post, I will cover the different alternatives we have to display data.
We will basically cover the standard charting software packages, custom high level standalone charting libraries and packages, to finally cover low-level graphics libraries.
For each area, I will describe the benefits and issues found with each approach, and I will highlight the areas that I consider that are relevant to build a professional approach to analyse market data.
Standard charting software packagesThere are many well-known charting platforms and software packages that can be used to display and analyse market charts.
Any investor or speculator will have his/her own charting software used regularly to carry on trading decisions.
Most active traders will also use the software to submit market orders directly from the chart, which allows to easily setup profit and loss levels, alarms, limited orders, etc.
While these charting platforms provide a superb degree of flexibility they usually deal with an interactive, discretionary and nonstandard way of analyzing data.
Among two of the most common platforms used by small and retail investors, we find ProRealTime and Metatrader.
Metatrader is a lightweight and simple tool.
Despite it has many limitations and the fact that it is only being used in OTC markets it has become the most popular tool among retail active investors; this is probably due to the fact that it has been adopted by a large number of retail brokers around the globe.
Metatrader 4 is one of the most widely used chartings and trading platform for CFD instruments.
Pro RealTime is another widely used tool among small and retail investors.
Capabilities are functionalities are much better than Metatrader and it is used for regulated markets.
It has professional functionalities such as order book depth, volume profile and many analysis tools.
Pro RealTime is a well-known platform for regulated markets among retail investors.
These are just two examples of the many tools available in the industry.
I have selected these two ones just because they are the ones I am more familiar with.
But the market counts with many other well-known platforms.
While charting software is appropriate for discretionary market analysis and trading, they focus more on traditional trading and they are not so suitable to approach historical data analysis.
They expect someone behind the screen interacting with the tool and the degree of flexibility they provide is outstanding.
It is so outstanding that I consider it too much for quantitative trading and for systematic market analysis.
Complex charts: when more is lessA charting software that can do anything, in every possible scale seems like a good option, but it is not indeed.
There is a common pattern in the technical analysis journey that every trader seems to go through.
During the initial months – while technical indicators and methodologies are discovered –, there is a tendency to overcomplicate chart analysis with multiple indicators, trend line, auxiliary lines, etc.
It is easy to think that more is better and that a full set of indicators will lead to safer operations.
This takes place until a plateau of complexity is reached.
From that point, it is common to start simplifying the charts to the minimum.
The process might experience a consolidation bouncing period until a fairly simple scenario is reached.
The level of simplification depends on each person and on the specific methodology being used, while some people prefer to draw a chart with nothing but price and a couple of support or resistance levels, others will prefer to use also volumes, one or two indicators — depending on the specific strategy being followed — , and some auxiliary Fibonacci levels.
The goal is to have only the lines that are strictly necessary, but a couple of extra indicators and some auxiliary lines can help sometimes.
Some strategies such as Elliott wave analysis require more extensive drawings for both price and time extensions and target areas.
Regular price architecture trading usually promotes fewer drawings to avoid distractions on what price is doing.
A further level of detail — such as using a grid or not, averages, few indicators or no indicators, etc.
— still depends more on particular preferences of each person.
Scales: nobody seems to pay attention to themOne of the areas where flexibility can be an issue is scales.
While discretionary trading encourages having the flexibility to deal with almost any potential time & price scale combination, quantitative trading encourages a more structured approach.
While analyzing any event — a market scenario in this particular case — from a more scientific point of view, it is required that the measurement scale does not change or changes within a controlled environment and conditions.
It makes no sense to observe the market with a different price scale for each day, the same that it makes no sense to evaluate half a day one time, day and half a second time and three days and a half a third time.
Additionally, when visualizing the market with charting software, unless you know extremely well the market you are following — something that shall be done nevertheless — you will not be able to understand if a given daily price movement is large or not, and how often that movement usually takes place.
Quantitative analysis deal with so many possibilities that it is needed to define the perimeter where analysis takes place.
In doing so, it is recommended to always deal with the same timeframe setups with clear and identified price scales.
I am not promoting a single and fixed setup for every analysis, but to professionalize the way that charts are looked at.
We can set up an analogy with cartography.
There is potentially an infinite number of potential scales for maps, but at the end, professional cartographers define a small set of available scales, widely used among the industry, and later choose the more appropriate scale for each particular map.
By doing that it is easier to evaluate each map.
If a different scale is used for every map or a different scale is used each time you analyse a specific map, it would be really difficult for cartographers to grasp easily an understanding on how large or small are the distances.
«Any scale can be used for a map, but a few common scales have been settled on for use by most organizations»Using a standard charting software usually promotes this different scale scenario, where the user can select any potential price and time scale, while scientific approach encourages the opposite: to limit the number of ways you can look at a chart.
My personal approach to charting is to determine price movements with a price movement histogram and later select — within a limited set — , which scale fits each particular chart.
I have no statistical evidence or proof that this approach reduces risk or increases profitability, but it sounds like a fair approach.
Regarding timescales, I also like to deal with specific, fixed time periods.
Intraday is 24 hours — I lately tend to use US Eastern Time to reflect the deep impact that US stock market has on global markets — , 48 hours to cope with larger movements extending more than one day, 5 day period to cope with weekly movements, then bi-weekly, monthly and yearly.
If larger periods are required I use standard charting software.
This approach of fixed time periods clearly contradicts the flexible nature of charting software, where the period is usually defined by bar widths, screen size and zoom.
They basically display as much information as can be fit in the screen as possible.
This might be seen as an advantage but it is basically changing your point of view and distance every time you want to observe an object or event.
Not a very systematic approach indeed.
Historical data: standard charting software just do not handle it wellWhile charting software is good to analyse what happened recently — or not so recently if you are charting larger time periods —, it is usually an extremely poor tool to systematically observe what has happened on intraday periods for longer periods.
Getting what happened any particular day two years ago it is not as easy as one might think, and chances are that it is not possible at all, as charting software relies on real-time data provided by the broker, and brokers usually limit the amount of intraday data available.
More often than not, you will not be able to retrieve information for periods older than two weeks.
In some charting tools, it is even actually really difficult to navigate the chart backwards in time, they are simply tools intended to be used as trading platforms, not as historical and strategy analysis platforms.
Custom charting with existing librariesThese conditions lead to the point where many people involved in quantitative analysis or intensive backtesting or historical analysis end up dealing with custom charting software.
Although it is possible to exploit existing charting software through their APIs —potentially saving development time —, there are important limitations on this approach and it tightly couples our software and methodology to a third party platform which might change their API or licensing terms in the future.
The initial approach to solve this problem is to use one of the many existing charting libraries or modular software packages that are already available.
While this seems a good approach — and it is usually the first one you address — , it poses limitations and at the end, it is easy to reach the conclusion that leveraging on an existing software package is not enough.
I have specifically tested Gnuplot and Matplotlib for this need.
If there is a well known, old-school plotting software, that is Gnuplot.
It was designed more than 30 years ago, and it is still one of the simplest and easiest ways to incorporate plotting capabilities to a project.
Gnuplot follows perfectly the UNIX philosophy of acting as a separate tool that can be piped or integrated at OS level with our software.
It is simple, powerful, easy to integrate and well documented.
Being a specific charting tool, it is extremely simple to use and integrate, but it counts with extremely poor interactive options.
While it is possible to integrate mouse and keyboard events with an external program — Gnuplot can launch external commands on events and get commands from standard input console which effectively enables a rudimentary interface implementation— , the integration is not natural nor straightforward.
It lacks the speed and flexibility required to implement complex user interfaces, and it is extremely clumsy to properly integrate it with external software.
During my tests, I prototyped a simple Gnuplot implementation.
Displaying charts with all data was easy but implementing an interactive user interface to draw trend lines or Fibonacci proved to be complex and slow.
Event handling was hard as a hard weight interface needs to be implemented between your software and Gnuplot.
It simply does not perform well and it lacks smoothness in its operation.
Scales were also an issue.
Finding that isolated tools such as Gnuplot lack the integration required to implement a minimum user interface — while charting, we would like to be able to at least draw trend lines, Fibonacci and support/resistance levels — , the second level of complexity is using a plotting library.
Matplotlib is a well-established library, it has wrappers for all major programming languages and it counts with a richer API to handle mouse and keyboard events.
Matplotlib is widely used and recommended in data science as a plotting library, but again it lacks the speed to implement complex user interfaces.
Matplotlib prototype implemented using Python wrapper for Matplotlib.
The example shows an intraday chart for Telefonica stock (SIBE TEF ES0178430E18).
A complex user interface will require embedding Tkinter — or any other lower level widget toolkit — as shown in the included figure.
I implemented a prototype using Tkinter, and while it is possible to display charts I found again that dealing with mouse and keyboard events was not easy.
Tkinter does not use GPU hardware acceleration either, which will pose a performance issue in complex charts.
Also, using Matplotlib creates a certain degree of complexity while dealing with the layout, as you are dealing with an embedded library which was not really designed to be integrated with Tkinter.
It is also worth to mention that Matplotlib counts with a specific financial extension which can plot candlestick charts out of the box.
However, this extension is no longer maintained and it is actually harder to use than regular Matplotlib bar plots (bars can be vertically offset so it is easy to draw candlesticks with a combination of bar and line charts).
My recommendation is to stay away from that extension, according to my personal experience is outdated, it has poor functionality, it lacks proper documentation and it is not supported in recent versions.
Low-level graphics librariesOnce that we have found that charting libraries are great for static charts, but that their interactive capabilities are far from being useful, it is clear that we need a low-level library to deal with the charts, and that our implementation will have to cover basic aspects such as scaling.
In this area, I have tried two approaches: Cairo Graphics and Java2D/AWT.
We do not cover Tkinter as we included the prototype which used Matplotlib, although another option would be to directly implement charts using a Canvas widget in Tkinter.
This implies implementing all chart functionality.
As there are no affine transformations and GPU hardware optimization in Tkinter response is a bit slower, but current hardware could cope with that.
I also prototyped this option — indeed that approach was the first prototype I did a long time ago — , at that time I find it too complex to be useful.
Our last prototype round for our quantitative project included Cairo and Java2D.
We found that GPU usage was required and Cairo looked like a good alternative on the paper, but I abandoned the idea the same day I started trying it after finding the complexities of running Cairo on Windows computers.
It was extremely complex to get it running.
Java2D/AWTAWT is not recommended for new developments, it is outdated and it was included in Java in the 90s.
Swing is presented as a modern replacement and more recently JavaFX is the recommended way for new developments.
This advice might be worth for regular user interfaces, but when dealing with financial charts we will spend more of our time and effort dealing with panel and canvas than with actual widgets.
As Swing and JavaFX are bulkier, going with Java2D and AWT is a perfectly valid solution.
Java2D uses hardware acceleration under the hood and it supports affine transformations.
Usage of affine transformations simplifies dealing with the actual price and bar scales, effectively decoupling part of the software from the window dimensions.
Our first prototype was a success, it was relatively easy to separate functionalities in different classes and the achieved performance was good.
We plan to further extend this prototype into a fully working solution, including a full encapsulation of features, custom user interface and the ability to provide volume profiles and selectable indicators/panels.
SummaryDespite existing charting software has many features and flexibility, its usage in quantitative trading and systematic backtesting is not the best alternative and a tailored solution is recommended.
Among the challenges found while dealing with this need, we can highlight the need for simplifying the information included in the chart, the use of proper and consistent scales (both for prices and time) and the complexities of systematically analysing historical data, especially when this data is stored as a proprietary repository.
These environments often require a certain degree of automation, which makes the use of standard software complex.
Leveraging on existing plotting libraries or modular software (such as Matplotlib or Gnuplot) seems a good option when the need is just displaying static charts, but their user interface capabilities are far from being perfect and the libraries were not built for the specificities of financial time-series charts.
Using low-level libraries such as Cairo and Java2D seems a good option as we can fully customize the way charts are presented, and despite they require more development effort their functionalities are extremely good as they will provide hardware acceleration and affine transformation capabilities which, at the end, reduce the development time as both price and time scales can be managed effortlessly.
Among those two options, I found Java2D as the simplest option to deal with this need, although there will be for sure other alternatives.
A strong point with Java2D is that it is truly oriented to user-space 2D vector graphics, which is what you really need when dealing with this kind of charts.
We have not covered other alternatives such as web-based libraries (such as TechanJS), but they pose the same complexities as the standard plotting libraries and their customization, integration and development are not straightforward — they are usually built on top of standard plotting libraries — .
They might be a good option though if a web user interface poses any benefit in your project.
Another option for this goal would be to use SVG, which is a powerful graphics language and it makes managing scales and user interface as simple as it can be on the web.
.. More details