I'm new here.
I also have a plan for creating my own tick data downloading and maintaining solution, but if you could implement my suggestions below, then I would willingly drop my project and could focus on trading related programming tasks again.
First, eliminating duplicate ticks would be a great improvement. But as I've read, you'll introduce this feature in next version.
Then, my solution would have an "auto-update" feature. This could happen, for example, by calling Tickstory as a scheduled task at end of every day or week (user selectable), and it would download the new data from Dukascopy for all (or the selected) pairs, then it would update the .fxt files automatically (if MT4 is not running, of course). This way we always could have a fresh, up to date set of .fxt files without the need of manual procedure. Of course, we must make an updating setup first (what pairs, when, which TFs, etc.), but then it would be automatic thereafter.
Next, during download, I would decompress the .bi5 files from Dukascopy and would make a new file with 4 byte integers instead of the 8 byte doubles (or longs) of the original. The timestamp could be in yyyy.mm.dd hh:mm:ss MT4 format without the millisecond part, as it would not be an issue at creating .fxt files. Also the prices could be multiplied by 100000 and stored as 4 byte integers, just like the volumes. Later, at creating .fxt or .csv files these price/volume data could be retrieved by dividing the integer by 100000 and normalizing the result to 2/3/4/5 digits (determined by fxt header info). This way the uncompressed data would be 50% of the original size and after compression it would take around 30-40% less space on the storage media. You could even unify the hourly data and store them as days instead, since backtester's smallest time interval is 1 day anyways. This would make even smaller data files after compression and also would reduce number of file accesses later, at generating .fxt's.
Also, I'd implement an .fxt maintainer, which could manage existing .fxt files. I mean we could re-adjust .fxt files' parameters, like spread, leverage, commission, stop level, swap etc. without generating a new file every time.
-----------------------------
There are also some issues that needs to be investigated in my opinion:
- The date fields should be in MT4's format (yyyy.mm.dd) instead of the dd/mm/yyyy, as it is confusing for non-english speaking people. MT4 date would be a generally understood format, anyways.
- Every set of tick data of every pair on Dukascopy has a starting date. For example EURUSD tick data begins on 2007.03.30. At downloading part, if no custom interval was selected yet, these default dates could appear in "from" field to make things easier for those who want to download the whole available interval. These data could be found in Birt's csv2fxt script or I can also provide them, if needed.
- Just noticed, that if I want to convert Dukas data from the very beginning, and this starting date/time is not a "round" number (like 00:00 at beginning of a day), then .fxt ticks will be shifted if timeframe is bigger than M1. For example, on Dukascopy EURUSD tick data is beginning on 2007.03.30 16:01. Tickstory is taking this date as a base and builds bars data shifted by 1 minute. So, if someone wants to make full size tick data in an M5 .fxt, then there will be 16:01, 16:06, 16:11, 16:16, 16:21, 16:26 etc. bars instead of 16:05, 16:10, 16:15, 16:20, 16:25 etc. ones. Of course, if you start at 2007.03.31., then fxt conversion will begin at 00:00 and the 1 minute offset will not present.
- I wanted to compare Tickstory generated fxt files with Birt's csv2fxt output. For this, I have used Tickstory to export downloaded data to csv. Then I have generated the fxt files using the csv2fxt script, but the backtester could not accept them (backtest ended immediately after start, no error message was produced). When I replaced the fxt to the one that was generated by Tickstory, the backtest has been completed correctly. Since I have already used csv2fxt before and it produced correct fxt files, I guess the Tickstory produced csv file can be the reason of the wrong fxt's. What do you think? (those floating point volumes are looking quite interesting, for example)
- I cant make .fxt and .hst files in other than preset MT4 folders. I can select the folder where I want to put the generated files, but Tickstory will yet to put them into the MT4 folder that was set before.
That's all for now.
I hope we can help each other.