Machine Learning is the new game. Machine learning entails writing algorithms that can learn from experience and then implement those algorithms in anyone of the programming languages like C++, Java, Python, R etc. Now R is a powerful statistical software that is open source. It has got more than 3000 packages that implement various machine learning and data science algorithms. The important question that comes to mind is how we can connect R with MT4 so that we can use its predictive power in our trading.Take a look at the following EURUSD H1 chart.
In the above screenshot can you see the red line ahead of price action? Well this is the predicted closing price for the next 20 candles. We have used R to make this prediction. This R indicator is using autoregression on the closing price to make the above predictions. Now if you have a way to know in which direction price is going to move, you can drastically improve your trading performance. In the above screenshot you can see that EURUSD has been ranging for the past 15 hours. The solid red line ahead of price action also indicates that price will most probably range for the next few bars. This is a simple indicator. Once we are able to connect R with MT4, we will be able to use its data science and machine learning power to make much more powerful indicators.
Let me give you an example. First read this post in which I have give you the MACD Signal Line Trailing Stop Loss EA MQL4 code. Now this EA works when the trend is very strong. Otherwise it gives false signals. If we could somehow connect R with MT4 and use its machine learning power to reduce the false signals that are being generated by this EA we can make a lot of pips. Now you can understand how much powerful R is and how we can use its machine learning and data science algorithms to improve our existing EA and indicators.
R Is An Open Source Project
So let’s discuss how to connect R with MT4. First you can download R. As said R is an open source project that has been started by a few professors in New Zealand some 2 decades back. Soon this R project had became popular and many other people from all across the world joined this open source project. Today as said in the beginning of this post there are more than 3000 R packages that implement many machine learning and data science algorithms in it. Python is also another open source scripting language that is rapidly becoming a very powerful data science and machine learning language. Google is behind Python. In this post we discuss how to connect R with MT4. In some future post we will discuss how to connect Python with MT4. Python being supported by Google will definitely surpass R in the next few years as the most powerful machine learning and data science project. Python is some 5-10 times faster than R. So you can well imagine what we can do with it once we are able to successful connect it with MT4.
For now we connect R with MT4. Develop a few indicators and EA. Once we get very good results with those indicators and EAs, we can then start working on the Python project that will connect it with MT4. C++ is the fastest language. After than comes Java and then comes Python. This is what we are planning to do. First we connect R with MT4 as said above. Implement a few machine learning and data science algorithms using R on MT4 in a few indicators and EAs. Check the performance of these indicators and EAs. If we get very good performance, then we will use C++ to develop the fastest versions of those indicators and EAs. The philosophy is to start simple. Test a few ideas in R. If we get very good results then we become serious and develop those indicators and EA in C++. C++ development is laborious and time intensive. Once we have tested our ideas using R, we than know we have succeeded and we can invest in C++ development.
Connecting R with MT4
Bernd Kruess from Germany was the first person who successfully connected R with MT4. He made a post on the Forex Factory Forum and shared the DLL file and the header file that is necessary when you connect MT4 with another platform for data sharing. You might be wondering what is a DLL file? DLL file is a Dynamic Link Library that becomes the interface between 2 software which are MT4 and R in our case. MT4 uses its own data structure and format while R uses its own data structure and format. A DLL file is required that can convert the data structure from MT4 format to R format and pass the data to R so that it can use it for computations. Then this same DLL file converts that data back into MT4 required format and pass it on to MT4 so that it can use it and display it on it charts. Sounds difficult? Yes it is indeed difficult. We should be thankful to Bernd Kruess for having spend a lot of his valuable time to achieve a seemless integration of MT4 with R. In the beginning there were many challenges. Which programming language to use for writing the DLL file? Bernd Kruess used Pascal instead of C++/C. He justified it by writing a post: Why Pascal?
So instead of using the usual C++/C route that most other developers would have used he adopted Pascal which is still being used widely and successfully connected R with MT4. In the beginning there were some glitches that made MT4 freeze when the connection was made but he was able to solve these daunting technical challenges. The most important thing is he shared his files with us on Forex Factory forum as said above. The R indicator that has been shown in the above screenshot has been shared by Bernd Kruess on Forex Factory Forum. It has been downloaded from there.
The most daunting challenge was that R uses matrices while MT4 uses arrays in storing the data. He was able to convert MT4 price arrays into matrices that are used by R. There 2 files that do that the most important file is the mt4R.mqh. Now another problem arose. After making the posings in 2010 Bernd Kruess vanished. MT4 updated its software and introduced a new build 600. The files posted by him stopped working as they were not compatible with the new build 600. This problem was again solved by another developer from Japan who updated the files posted by Bernd Kruess. You can download the new updated files from this MQL5 post. After uploading the files you should copy and paste them in proper MQL4 folder files. If you face any problem you can leave a comment below and we can help you in that. But we think you will be able to do it as by now you should be familiar on which file to place where. It is not difficult at all. You can take a look at the mt4R.mqh file below and appreciate how much hard work he had put it! mt4R.mqh is the header file for mt4R.dll file.
mt4R.mqh and mt4R.dll Files
#property copyright “ｩ 2010 Bernd Kreuss”
#define MT4R_VERSION_MAJOR 1
#define MT4R_VERSION_MINOR 4
/** @file
* Metatrader-4 -> R Interface Library
*
* This source is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* A commercial license and support is available upon request
*
* mt4R.dll allows the direct usage of the R environment
* (The R Project for Statistical Computing, http//www.r-proect.org)
* from within your mql4 code. You can start an R session (one or more
* for each EA instance), send data to it, call R functions and get
* results back ino your mql4 program. Each R session will run as a
* separate process but the communication of the mql4 program with R
* will appear towards mql4 as synchronuos function calls. Behind the
* scenes it will construct and execute small snippets of R code at
* the REPL, wait for the prompt to return and parse the output, very
* much like you would do it manually in an interactive R session.
*
* mt4R.mqh (this file) is the header file for mt4R.dll. Put the dll
* into experts/libraries and put this file into experts/include.
* Then use it in your EA with
* <PRE>
*
* #include <mt4R.mqh>
*
* </PRE>
* The following two external variables are recommended, but of course
* you might also hardcode them. You must use –no-save but you
* MUST NOT use –slave or otherwise turn off the echo or change
* the default prompt or it will fail.
* <PRE>
*
* extern string R_command = “C:\Programme\R\R-2.11.1\bin\Rterm.exe –no-save”;
* extern int R_debuglevel = 2;
*
* </PRE>
* Debug level can be 0, 1 or 2. During development you should set it
* to 2 which will output every available message. A value of 1 will
* only output warnings and notes and a value of 0 will only output
* fatal errors, this is the recommended setting for production use.
*
* Debug output will go to the system debug monitor. You can use
* the free DebugView.exe tool from microsoft to log it in real time.
*
* The library defines a handful of functions to directly assign and
* read a few data types without the need for manually formatting and
* executing snippets of R code for these purposes. Not all possible
* data types are directly supported, for example there is no function
* to directly assign a matrix of strings since I never saw the need
* to do this. The main emphasis is on vectors and matrices containing
* only floating point values. Once you have transferred the bulk of
* your data as vectors or matrices into the R session you can execute
* snippets of R code to combine them or convert them into something
* more complex if needed.
*
* Also the main emphasis is on getting huge amounts of numeric data
* into the R session quickly but not the other way, since the assumption
* is that you want to feed it with vast amounts of numerical data to
* crunch numbers and only need to get back a single value or a vector
* as a result.
*
* All the vector functions have a size parameter. Be very careful
* that the array you supply has actually the same size (bigger
* won’t hurt but smaller is not allowed). You should always use
* ArraySize() or ArrrayRange() for the size parameter, this will
* make your life much easier.
* <PRE>
* // *** EXAMPLE USAGE ***
*
* #include <mt4R.mqh>
*
* extern string R_command = “C:\Programme\R\R-2.11.1\bin\Rterm.exe –no-save”;
* extern int R_debuglevel = 2;
*
* int rhandle;
*
* int init() {
* rhandle = RInit(R_command, R_debuglevel);
* }
*
* int deinit() {
* RDeinit(rhandle);
* }
*
* int start() {
* int i;
* int k;
* double vecfoo[5];
* double vecbaz[5];
*
* for (i=0; i<5; i++) {
* vecfoo[i] = SomeThingElse(i);
* }
*
* RAssign(rhandle, “foo”, vecfoo, ArraySize(vecfoo));
* RExecute(rhandle, “baz <- foo * 42”);
* k = RGetVector(rhandle, “baz”, vecbaz, ArraySize(vecbaz));
*
* for (i=0; i<k; i++) {
* Print(vecbaz[i]);
* }
* }
*
* double SomeThingElse(int n) {
* […]
* }
*
* </PRE>
*/
#import “mt4R.dll”
/**
* Return the dll version. The upper 16 bit of the return value
* are the major version and the lower 16 bit the minor. This
* is used in RInit() to make sure that this header file and
* the dll fit together.
*/
int RGetDllVersion();
/**
* This is not meant to be called directly, it will be
* called by RInit() after the successful version check.
* You should call RInit() to start a new R session.
*/
int RInit_(string commandline, int debuglevel);
/**
* Teminate the R session. Call this in your deinit() function.
* After this the handle is no longer valid.
*/
void RDeinit(int rhandle);
/**
* return true if the R session belonging to this handle is
* still runing. R will terminate on any fatal error in the
* code you send it. You should check this at the beginning
* of your start function and stop all actions. The last
* command prior to the crash will be found in the log.
* If R is not running anymore this library won’t emit any
* more log messages and will silently ignore all commands.
*/
bool RIsRunning(int rhandle);
/**
* return true if R is still executing a command (resulting
* from a call to RExecuteAsync())
*/
bool RIsBusy(int rhandle);
/**
* execute code and do not wait. Any subsequent call however
* will wait since there can only be one thread executing at
* any given time. Use RIsBusy() to check whether it is finished
*/
void RExecuteAsync(int rhandle, string code);
/**
* execute code and wait until it is finished. This will not
* return anything. You can basically achieve the same with
* the RGet*() functions, evaluating the expression is also
* just executig code, the only difference is that these
* RGet*() functions will additionally try to parse and return
* the output while RExecute() will just execute, wait and
* ignore all output.
*/
void RExecute(int rhandle, string code);
/**
* assign a bool to the variable name. In R this type is called “logical”
*/
void RAssignBool(int rhandle, string variable, bool value);
/**
* assign an integer to the variable name.
*/
void RAssignInteger(int rhandle, string variable, int value);
/**
* assign a double to the variable name.
*/
void RAssignDouble(int rhandle, string variable, double value);
/**
* assign a string to the variable namd. In R this type is called “character”
*/
void RAssignString(int rhandle, string variable, string value);
/**
* assign a vector to the variable name. If the size does not match
* your actual array size then bad things might happen.
*/
void RAssignVector(int rhandle, string variable, double &vector[], int size);
/**
* assign a vector of character (an array of strings) to the variable. If you need
* a factor then you should execute code to convert it after this command. In
* recent versions of R a vector of strings does not need any more memory than
* a factor and it is easier to append new elements to it.
*/
void RAssignStringVector(int rhandle, string variable, string &vector[], int size);
/**
* assign a matrix to the variable name. The matrix must have the row number as the
* first dimension (byrow=TRUE will be used on the raw data). This function is much
* faster than building a huge matrix (hundreds of rows) from scratch by appending
* new rows at the end with RRowBindVector() for every row. This function is optimized
* for huge throughput with a single function call through using file-IO with the
* raw binary data. For very small matrices and vectors with only a handful of elements
* this might be too much overhead and the other functions will be faster. Once you
* have the matrix with possibly thousands of rows transferred to R you should then
* only use RRowBindVector() to further grow it slowly on the arrival of single new
* data vectors instead of always sending a new copy of the entire matrix.
*/
void RAssignMatrix(int rhandle, string variable, double &matrix[], int rows, int cols);
/**
* append a row to a matrix or dataframe. This will exexute
* variable <- rbind(variable, vector)
* if the size does not match the actual array size bad things might happen.
*/
void RAppendMatrixRow(int rhandle, string variable, double &vector[], int size);
/**
* return true if the variable exists, false otherwise.
*/
bool RExists(int rhandle, string variable);
/**
* evaluate expression and return a bool. Expression can be any R code
* that will evaluate to logical. If it is a vector of logical then only
* the first element is returned.
*/
bool RGetBool(int rhandle, string expression);
/**
* evaluate expression and return an integer. Expression can be any R code
* that will evaluate to an integer. If it is a floating point it will be
* rounded, if it is a vector then only the first element will be returned.
*/
int RGetInteger(int rhandle, string expression);
/**
* evaluate expression and return a double. Expression can be any R code
* that will evaluate to a floating point number, if it is a vector then
* only the first element is returned.
*/
double RGetDouble(int rhandle, string expression);
/**
* evaluate expression and return a vector of doubles. Expression can
* be anything that evaluates to a vector of floating point numbers.
* Return value is the number of elements that could be copied into the
* array. It will never be bigger than size but might be smaller.
* warnings are output on debuglevel 1 if the sizes don’t match.
*/
int RGetVector(int rhandle, string expression, double &vector[], int size);
/**
* do a print(expression) for debugging purposes. The outout will be
* sent to the debug monitor on debuglevel 0.
*/
void RPrint(int rhandle, string expression);
#import
/*
* start and initialize a new R session. Call this function in init() and store
* the handle it returns. This will start an R session and all subsequent calls
* to R functions will need this handle to identify the R session. This function
* will check the version of the dll against the version of this header file and
* if there is a mismatch it will report an error and refuse to initialize R.
*/
int RInit(string commandline, int debuglevel){
int dll_version;
int dll_major;
int dll_minor;
string error;
dll_version = RGetDllVersion();
if (dll_version == (MT4R_VERSION_MAJOR << 16) + MT4R_VERSION_MINOR){
return(RInit_(commandline, debuglevel));
}else{
dll_major = dll_version >> 16;
dll_minor = dll_version & 0xffff;
error = “Version mismatch mt4R.dll: ”
+ “expected version ” + IntegerToString(MT4R_VERSION_MAJOR) + “.” + IntegerToString(MT4R_VERSION_MINOR)
+ ” – found dll version ” + IntegerToString(dll_major) + “.” + IntegerToString(dll_minor);
Print(error);
return(0);
}
}
/**
* shorthands for some of the above functions
*/
int hR;
void StartR(string path, int debug=1){
hR = RInit(path, debug);
}
void StopR(){
RDeinit(hR);
}
void Rx(string code){
RExecute(hR, code);
}
void Rs(string var, string s){
RAssignString(hR, var, s);
}
void Ri(string var, int i){
RAssignInteger(hR, var, i);
}
void Rd(string var, double d){
RAssignDouble(hR, var, d);
}
void Rv(string var, double &v[]){
RAssignVector(hR, var, v, ArraySize(v));
}
void Rf(string name, string &factor[]){
RAssignStringVector(hR, name, factor, ArraySize(factor));
Rx(name + ” <- as.factor(” + name + “)”);
}
void Rm(string var, double &matrix[], int rows, int cols){
RAssignMatrix(hR, var, matrix, rows, cols);
}
int Rgi(string var){
return(RGetInteger(hR, var));
}
double Rgd(string var){
return(RGetDouble(hR, var));
}
void Rgv(string var, double &v[]){
RGetVector(hR, var, v, ArraySize(v));
}
void Rp(string expression){
RPrint(hR, expression);
}
The most important challenge is understanding the data structure that is being used by MT4 and the data structure that is being used by R and then learn how to change from one format to another back and forth in real time. You can see how much work has been put into it. As said above this is an important breakthrough. We use utilize it and test it by developing new indicators and EAs using the R. Bernd Kruess has also tried to connect Python with MT4. The work is in progress. Since he stopped posting after 2010 the work has not been updated. A lot of changes has been made in Python also. It’s new version Python 3.5 is much different from the older version 2.6 on which this DLL files were made. So most probably it wont work. But as said above we plan to develop a few machine learning indicators using R. Once we achieve our satisfaction and believe that we have developed what we wanted we will start the development of that indicator and EA in C++/C which is the fastest language. Since R implements most of the machine learning and data science algorithms we don’t need to focus on Python for now. In this post we share a few of our idea on how to develop an artificial intelligence and machine learning based indicator.