Highs and Lows and Time for Candles has turned to Zero

[TesterThomas]

I'm not sure if this is related to Tickstory data or is it MT4, but I just want to see if anyone has had a similar experience.

I have many similar EA's that all return highs and lows, distances and the time of those prices etc.

Just recently all of the highs and lows are not being returned along with the times of the candles, and I mean not even incorrect, it's just zero.

The EA starts at 2019, but as soon as it reaches 26/09/2019 it seems to all be fine, times are there, and the highs are higher than the lows etc...


I first started noticing the problem this morning after I tested something that didn't work too well. I watched a video about using a function to export data directly to Excel from MQL4, once I started the tester I got an unusual freeze and the little spinning circle (I never ever experience that when I run a test), it also took ages to not only test, but to even begin. Ever since I done this, it has been extremely slow to start testing, testing seems slower in general, and I have found these price errors.

I have even tested an older version of the Expert Advisor and the problem is there too.


Have you ever experienced a situation where MT4 itself, or the tick data can become corrupt based on the above-mentioned spreadsheet export test I done?

[tickstory]

Hi Thomas,

Firstly, please make sure you are getting 99% modelling quality on your final report. This will confirm you are correctly using the exported Tickstory tick data. You can also double-check that the ranges of data you expect are in the MT4 History Center.

After that, you may want to refer to the manual reference for iHigh which advises that you can get zero if there is an error. The GetLastError() may give you further information. Of course, the Journal might also be giving some hints of what the problem might be.

As usual with these sorts of problems, putting lots of logging code will often help track down the error such as printing the current price to the Journal. Of course, make sure you are printing it after converting to a double as this is a common gotcha when getting weird values.

Hope this helps.

[TesterThomas]

Thanks

I'm trying this stuff out now. In History Center, for the 5 minute candles, when I scroll to the bottom, the furthest back it goes is to 17/4/2020, even though I exported Tickstory data back to 2015.

I have noticed that the problem is coming from iBarshift, even though TimeCurrent is working fine, the candle shift value isn't being returned.

And here it is, I know what it is, my iBarshift is getting data from the 1 minute candle, that's why everything else is working perfectly, but it can't get data for it because for some reason the History Center data for 1 minute stops at the exact time I mentioned in my earlier post 26/09/2019.

The daily data does go back to 2015, but as soon as I get to the 15 minute chart, that only goes up to mid 2018.

This has not happened before, does it have something to do with the possible corruption idea I spoke about?

And just to clarify, this usually works perfectly fine, I always have lower time frame tools in higher time frame charts.

Can you advise on how I can fill up the history center for smaller time frames please? I know the normal MT4 chart data always has much less on the smaller time frames, but I never experienced this with the strategy tester.

Main thing anyway is at least I know what the problem is, and thank God I'm not delusional...

[tickstory]

Hi Thomas,

It sounds like your data is back-filling from your broker. Could you please log out of your terminal (File->Log into Trade Account->Enter an incorrect password). This will prevent it from happening. After that, you will need to re-export your data range and try again. You shouldn't see any history changes any more and the data in the History Center should reflect the dates you exported with Tickstory.

Hope this helps.

[TesterThomas]

I have been trying the latest suggestions. At first I simply got the data working on another MT4, and I managed to process a small few EA's.

While I was doing that I went through your suggestion, I logged out, I deleted the data, then I re-exported new data from Tickstory, for some reason it didn't work. i totally understand what's happening, but for some reason MT4 just feels like logging back into a demo account again.

I've been trying this back and forth between three MT4's from one broker. Sometimes I simply dragged the data from one broker to the other, even this used to work fine in the past. Many times I simply cut and pasted from one History folder to the other and it worked fine.

I've gotten to a point where very unusually, an EA that operates only on minute charts (and takes nothing from above) is processing the minute highs and times just fine, all the way back to maybe even further than 2019. But even for the 5 minute and all higher, it does absolutely everything else perfectly, but the only one it leaves out is the minute pieces. Is there any explanation for this? Might it be because I copied the data from an MT4 where it wasn't working correctly. Basically transforming the data so that it doesn't grab multiple timeframes at a time?

I must add that just a few hours ago, EA's above the minute worked fine. I have no idea why it stopped working, maybe I was using the wrong MT4.

But I also don't understand why the history center matters, I thought that has no connection to the strategy tester's data?

Also, are there situations where you can screw MT4 up so that even a reinstall in the same location wont work again because there are traces of something?

Right now I'm using a new broker, which I'm currently exporting data to, hopefully that works

[TesterThomas]

I've just tried it on that new broker and it has done the exact same thing, it worked with the 1 Minute EA, but it didn't work on higher timeframes containing 1 Minute. Also, the history center is completely empty.

And I received the error message "data not reached from the least timeframe". I am testing with data going back to 2015, and the test is starting in 2019, so it makes no sense. I done everything correctly, I logged out, messed up the password, created the directory location in Tickstory, exported using the map to symbol name (the only thing I noticed is this broker awkwardly has a # sign in front of the symbol, does that mean we have to enter it?
Then I launched MT4 from tickstory.

Nonetheless it still processed in the strategy tester. However just now I decided to try it again, and now it's saying no history data. Possessed, it is possessed.

I am sure it has something to do with the hash sign, but it was processing a few minutes ago with no changes made


Edit; I can confirm the hash sign has been found in front of newly generated data files. Why do brokers have to be so weird, US30 is the name, leave it alone..... or maybe DOW30, but #US30????????


EDIT again; I'm seesing Get Last Error 4074. I'll do some searching about it, just thought I should mention it here.. This is on the old broker by the way, I have to re-export the data for the new one

[TesterThomas]

Please tell me erorr 4074 means my computer doesn't have enough memory to process the history data, is that correct? It's annoying, but at least it can be solved. I have about 70 GB free out of 475, so there should be enough.

However, my friend told me today that when you use a program to write to a file it can be very intensive, I'm guessing when I tried that write to file function this morning it tore through my memory and wreaked havov somewhere.

I have read before that bad programming or operations can fill up MQL4 memory, but I have no idea how to check this. Any suggestions?

It's definitely something along those lines, it can't be anything else, I've been operating in this exact way for months without any problems

[TesterThomas]

I may be getting somewhere. Since the error message seems to say "not enough memory for history", I decided to export only one years worth of data and now testing on the 1 hour timeframe/EA, it's working. Well, at least I know it is getting data from both time frames.

So I recon the problem relates to using data that goes back to 2015, since that data takes up about 70 Gigabytes. I'm not sure how it works, but maybe if all of that data is accessed at once throughout the test, then it may be too large.

If this is the solution, then that causes a new problem. If I want to use data that goes back to 2018, with large moving averages, such as 200 and even up to 500, that means that before the test even begins, I need to have a maximum of at least 500 days of data in order for a 500 period moving average to be created on a daily chart.

Is there a work around for this? Such as downloading 2 years, then another 2 years and so on? So the data isn't all one chunk? but then to have them combine when I want to use it?

I am getting a larger laptop (2 terabytes of storage) fixed soon, so if it operates accordingly, then I'm sure that will be able to handle far more than my current laptop which has only 475 GB and 80 of that left.

Is that actually how it works? Larger data times need a larger computer?

[tickstory]

Hi Thomas,

What you can do is have several MT4 terminals each with a subset of data (for example, 1st terminal=2015-2018, 2nd terminal 2018-2021, etc).
This will also ensure that you are not over-fitting your strategy.

Given that you are using 500 day moving averages, I am guessing that your trading frequency is quite low. If that is the case, have you tried using the 'Open price' modelling method instead? This should still give you a relatively accurate picture of your strategy's trading performance, especially if you were taking 1 trade every month or so. Using the Open Price method should have much lower memory requirements.

Hope this helps.

[TesterThomas]

Thanks

That's a good idea and I'll see what I can do with it, I'll certainly do things like that in the future, but for now it may not work too well.

Basically all I'm doing right now is getting price interactions with the moving averages, distances from the averages and so on. I haven't actually created a proper strategy with trades yet.
I also use the 1 minute chart on every one of them in order to get the most accurate high and low point, along with its time in the cycle.


So if I split it halfway then there will be a huge gap where cycles were only a small bit complete.

However, first of all, the frequency of cycles of 500 moving averages on a daily timeframe is so low, it may not be as important as the more mid size averages on lower time frames. And second of all, maybe for the much larger timeframes I could use a lower timeframe of 15 or 30 minutes instead of 1 minute in order to get accurate highs and lows.


They're all the main reasons for the data overload, there are simply far too many variables and they are active too often. As soon as I get these distances and start working with them, I'll have much fewer variables, and in use much less often.

Similar to the open price model you mentioned, it would be great if maybe there was some sort of OHLC method, so just the end result of each candle. It wouldn't be perfect, but it would speed things up a bit for some broad view testing.



Anyway, thanks for your help, it's always good to understand more about how all of this works. There's a pretty tough and frustrating learning curve at times.

Just one other thing I'm wondering. If I were to upgrade my computers Ram, would that make a significant difference. I have 8 GB. I imagine 16 would really push performance up. If so, hopefully one of laptops will be capable of such an upgrade

Thanks