The data collection is working perfect and isn’t having any impact on the trading, by which I mean it isn’t slowing it down at all. I use timer to track code execution times.
The issue is file size. At the end of each day’s trading, the data is saved to a new file. These are around 150mb in size. I keep all my trading stuff in cloud storage, which has its limits (capacity). It got me thinking, firstly to store locally, secondly, how to make use of it. This takes me back to where I started with data collection- what to do with it.
It’s all about analysis. If I work out the process of analysis, find the relevant data points I need to monitor, I can perform the calculations live and save the results only, rather than everything.
Having a good few days worth of data will be useful for testing but I don’t want to have endless reams of stuff I might never go back to. So once I’ve got a couple of gigs stored, I’ll stop the collection and start the analysis.