Hi
Although i do love this program i now question the accuracy.
Previously i downloaded data from php scripts..processed and converted..Long process. I was useing Birts php scripts.
So i now download with tickstorylite and strangelly my back testing results are greatly different to before.
So i checked the Tick data downloaded both from this program and birts php scripts to find out they differ..
so my question is simple..how do we know whats right here. Both apparently get the data from the same place..So how can they be so different.
50pips
Accuracy of Data
Re: Accuracy of Data
Hi 50pips,
Can you please provide more details.
- You mentioned in another forum that there are file size discrepancies compared to Birt's scripts. Can you please outline what "file size" are you comparing?
Folder size of a particular month? Specific date/time/hour file size? What is the size/contents of the folder? Did you know that Dukascopy changed their file format at the start of the year to a much smaller format? Are you sure you're not comparing files of the old format with the new smaller format? Also note that Tickstory does not download the Dukascopy candle data but instead generates its own candles "on-the-fly".
- What EA are you getting different results for? What options did you use produce the FXT file in the CSV2FXT scripts vs Tickstory? Are you using variable spreads (which Tickstory does not yet support)?
Any additional information that will help identify the issue you're having would be much appreciated.
Thanks.
Can you please provide more details.
- You mentioned in another forum that there are file size discrepancies compared to Birt's scripts. Can you please outline what "file size" are you comparing?
Folder size of a particular month? Specific date/time/hour file size? What is the size/contents of the folder? Did you know that Dukascopy changed their file format at the start of the year to a much smaller format? Are you sure you're not comparing files of the old format with the new smaller format? Also note that Tickstory does not download the Dukascopy candle data but instead generates its own candles "on-the-fly".
- What EA are you getting different results for? What options did you use produce the FXT file in the CSV2FXT scripts vs Tickstory? Are you using variable spreads (which Tickstory does not yet support)?
Any additional information that will help identify the issue you're having would be much appreciated.
Thanks.
Re: Accuracy of Data
Hi admin,
I have data downloaded from dukascopy using jforex from 2007 to july 2012
just getting back into tickdata testing and updating + reusing my existing jforex dukascopy data to build the fxt + hst files (export to mt4 files...)
the logs shows lots of:
"Error occured processing file ...pathTo\jforex\.cache\pair\....bi5. Attempting redownload
Is that normal or what?
Note that all existing data were downloaded back in July 2012, it was valid and i was able to do 99% backtest with it. Why is tickstory now saying "error occured processing file ...bi5 and have to redownload again?
Please advise.
* btw, what is this "on the fly" candle that you mentioned previously?
* you have my email, email me and i can send you some sample *.bi5 file between now and then which are both different, for your comparison
I have data downloaded from dukascopy using jforex from 2007 to july 2012
just getting back into tickdata testing and updating + reusing my existing jforex dukascopy data to build the fxt + hst files (export to mt4 files...)
the logs shows lots of:
"Error occured processing file ...pathTo\jforex\.cache\pair\....bi5. Attempting redownload
Is that normal or what?
Note that all existing data were downloaded back in July 2012, it was valid and i was able to do 99% backtest with it. Why is tickstory now saying "error occured processing file ...bi5 and have to redownload again?
Please advise.
* btw, what is this "on the fly" candle that you mentioned previously?
* you have my email, email me and i can send you some sample *.bi5 file between now and then which are both different, for your comparison
Re: Accuracy of Data
Hi mainw,
Dukascopy have changed their file format at some stage this year and this is why you are seeing this error. Tickstory only supports the latest format which have significantly smaller sizes. When you see this error, it simply means that the older file format is being replaced with the newer. Once all your data has been updated to the latest format, you should not see this error again.
With regards to the candle data, Dukascopy also allow you to download candle information however rather than using and storing this data, Tickstory produces the same candle information by using the tick data. This way, it can support construction of bar information that Dukascopy may not have, not to mention there is no disk space overhead.
Hope this helps.
Dukascopy have changed their file format at some stage this year and this is why you are seeing this error. Tickstory only supports the latest format which have significantly smaller sizes. When you see this error, it simply means that the older file format is being replaced with the newer. Once all your data has been updated to the latest format, you should not see this error again.
With regards to the candle data, Dukascopy also allow you to download candle information however rather than using and storing this data, Tickstory produces the same candle information by using the tick data. This way, it can support construction of bar information that Dukascopy may not have, not to mention there is no disk space overhead.
Hope this helps.
Re: Accuracy of Data
Thanks for the reply, admin.
That's much clearer now.
I have also previously used Birt's csv2fxt and one thing that i like in in that's missing in tickstory is this:
csv2fxt shows an alert telling me where there's a gap in the tickdata.
but since tickstory convert tick data to fxt/hst without needing a csv, i wonder how does that work?
can tickstory generate a report showing if there are any gaps after finishing building fxt/hst?
That's much clearer now.
I have also previously used Birt's csv2fxt and one thing that i like in in that's missing in tickstory is this:
csv2fxt shows an alert telling me where there's a gap in the tickdata.
but since tickstory convert tick data to fxt/hst without needing a csv, i wonder how does that work?
can tickstory generate a report showing if there are any gaps after finishing building fxt/hst?
Re: Accuracy of Data
Hi mainw,
Thanks for the suggestion. At the moment since Tickstory sources its information only from Dukascopy, there is no need to perform integrity checks since the data is assumed to be complete. When Tickstory provides other datasources such as a CSV import, this feature will definitely come in handy. Consider it on the todo list.
Regards.
Thanks for the suggestion. At the moment since Tickstory sources its information only from Dukascopy, there is no need to perform integrity checks since the data is assumed to be complete. When Tickstory provides other datasources such as a CSV import, this feature will definitely come in handy. Consider it on the todo list.
Regards.
Re: Accuracy of Data
Thanks, admin.
birt's approach is
download tickdata from dukascopy > merge data into csv > build fxt/hst using csv2fxt
tickstory approach is
download tickdata from dukascopy > build fxt/hst directly using tickdata
I like the approach that tickstory has taken as it takes out the effort of combining/generating csv from tickdata
BUT, here's the big BUT...
There are still gaps in dukascopy data (not a lot, but it's there. They are still the best though when compared to other data sources)
I'm not actually suggesting for the feature - "provides other datasources such as a CSV import"
But am suggesting that tickstory perform some integrity check when building the mt4 fxt/hst and report if there are any gaps.
Another observation:
The tickstory built fxt are much bigger in file size compared to those i generated back in july 2012 using csv2fxt
That certainly sounds strange since i take it that with smaller bi5 introduced from dukascopy "sometime this year", I would assume that the generated fxt would also be smaller?
birt's approach is
download tickdata from dukascopy > merge data into csv > build fxt/hst using csv2fxt
tickstory approach is
download tickdata from dukascopy > build fxt/hst directly using tickdata
I like the approach that tickstory has taken as it takes out the effort of combining/generating csv from tickdata
BUT, here's the big BUT...
There are still gaps in dukascopy data (not a lot, but it's there. They are still the best though when compared to other data sources)
I'm not actually suggesting for the feature - "provides other datasources such as a CSV import"
But am suggesting that tickstory perform some integrity check when building the mt4 fxt/hst and report if there are any gaps.
Another observation:
The tickstory built fxt are much bigger in file size compared to those i generated back in july 2012 using csv2fxt
That certainly sounds strange since i take it that with smaller bi5 introduced from dukascopy "sometime this year", I would assume that the generated fxt would also be smaller?
Re: Accuracy of Data
Hi mainw,
I can see the reasoning for having this integrity check but interested to know how it would be useful across Dukascopy data? After all, if there is a gap this is essentially the "raw" data that is being used for all Dukascopy & Jforex back-tests. I assume you just want to know for informational purposes?
Regarding the size of the FXT, this should not be dependent on the size of the Dukascopy data files as the FXT format remains the same. The only thing I can think of that would reduce the size of your FXT file is by filtering duplicate ticks (or any other data filter). At the moment, Tickstory does not provide this functionality.
Hope this helps.
I can see the reasoning for having this integrity check but interested to know how it would be useful across Dukascopy data? After all, if there is a gap this is essentially the "raw" data that is being used for all Dukascopy & Jforex back-tests. I assume you just want to know for informational purposes?
Regarding the size of the FXT, this should not be dependent on the size of the Dukascopy data files as the FXT format remains the same. The only thing I can think of that would reduce the size of your FXT file is by filtering duplicate ticks (or any other data filter). At the moment, Tickstory does not provide this functionality.
Hope this helps.
Re: Accuracy of Data
Hi admin,
Yes, it's only for information purpose to know how often gaps happen in the tickdata. For me, it was useful when deciding which data source to use.
For example, I had given truefx tickdata a try and used birt's approach to get the fxt/hst. What I found out was that it too has gaps and it happened more often compared to dukascopy tickdata for the same pair. So in the end, I opt for dukascopy tickdata as it has less gap and happen less often.
Without knowing this "information", users won't know about gap and how accurate their data is...
Yup, birt's csv2fxt has a check for duplicate ticks. Will that be released in the next version of tickstory?
BTW, happy new year 2013!
Yes, it's only for information purpose to know how often gaps happen in the tickdata. For me, it was useful when deciding which data source to use.
For example, I had given truefx tickdata a try and used birt's approach to get the fxt/hst. What I found out was that it too has gaps and it happened more often compared to dukascopy tickdata for the same pair. So in the end, I opt for dukascopy tickdata as it has less gap and happen less often.
Without knowing this "information", users won't know about gap and how accurate their data is...
Yup, birt's csv2fxt has a check for duplicate ticks. Will that be released in the next version of tickstory?
BTW, happy new year 2013!
Re: Accuracy of Data
Hi mainw,
Thanks for the info. We'll put the "gap detection" feature towards when new data sources are available. As for the duplicate tick filter, we'll try to get this out in the next version or the one after.
Hope you have a great 2013 as well - happy trading!
Regards.
Thanks for the info. We'll put the "gap detection" feature towards when new data sources are available. As for the duplicate tick filter, we'll try to get this out in the next version or the one after.
Hope you have a great 2013 as well - happy trading!
Regards.