The data collection is working perfect and isn’t having any impact on the trading, by which I mean it isn’t slowing it down at all. I use timer to track code execution times.
The issue is file size. At the end of each day’s trading, the data is saved to a new file. These are around 150mb in size. I keep all my trading stuff in cloud storage, which has its limits (capacity). It got me thinking, firstly to store locally, secondly, how to make use of it. This takes me back to where I started with data collection- what to do with it.
It’s all about analysis. If I work out the process of analysis, find the relevant data points I need to monitor, I can perform the calculations live and save the results only, rather than everything.
Having a good few days worth of data will be useful for testing but I don’t want to have endless reams of stuff I might never go back to. So once I’ve got a couple of gigs stored, I’ll stop the collection and start the analysis.
TRADER247 
You should really be logging this data to a database if possible rather than files. Much easier to then query the data. Download SQL Server Express or mySQL. Plenty of tutorials out there.
LikeLike
Hi Scott,
Thanks for the advice, I’ll have a look into it. Anything that helps me improve is welcome, cheers
LikeLike
Of course, when you get onto to coding your own price bot with API-NG then the returned data will be JSON format, which is a database in itself.
I keep all my data in JSON and then wrote a program to convert a subset of the data into CSV if I need to chart it on a spreadsheet or load it into a machine learning program etc. I have no need for a relational database. It would force me to look at data in a relational way. I’d rather be free of such constraints.
BPT – http://www.betfairprotrader.co.uk/
LikeLike
Yes, I guess it’s time to stop messing with VBA and get on with VB.net, just need to get to grips with visual studio then I’m away
LikeLike
VS should be easy to understand if you can program in VBA.
Don’t forget to use VS Community 2015 rather than the old 2013.
BPT – http://www.betfairprotrader.co.uk/
LikeLike