Tag Archives: Coding

Automated betting

Excel VBA timer

Leave a comment

I use the Timer function in VBA to help identify any problems in my code. After the code is triggered, I set startTime = Timer. My code’s first job is to lift the data that is imported from Gruss to Excel into an array. Once the bot has done its thing, the array is pasted back into excel with any triggers/new data added (so my code only does 1x excel read and 1x excel write). The last thing it does before the paste instruction is to add the time taken to run the code to an unused cell in the array area – stopTime = Timer – startTime. This is just to provide a visual indication while I’m watching, to see what’s happening and if there’s any blockages.

Since I’ve recoded, the time reported is rarely anything other than zero. But zero what? What is zero? What is the resolution of the built in Timer function? I initially read that the Timer counted in 1/64ths of a second. Which is 0.015625 seconds. This made sense, as this is a number I’ve seen flash up on the sheet as the bot was running (I actually saw 0.0156 due to cell format). It would normally bounce between this and zero.

This means my code is running, normally, in less than 0.02 seconds. The shortest interval between price update I’ve observed on Gruss, when on full stream, is around 0.1 seconds (up to 10 refreshes per second). So I know my code is running fast enough not to be missing any data refreshes. That’s good.

An interesting observation from the VPS – I’ve seen, though not often, the reported time as 0.0078. this looks like half of 1/64th, and it is. 1/128th seconds is 0.0078125.

My conclusion is that the Timer function is actually linked to the hardware and/or  the operating system, and therefore may differ depending on your set up.

Automated betting

Building a Better Trading Bot: Lessons from FastTrade to VOLT

Leave a comment

The bot that’s running on the VPS, which is testing my ability to stomach losses, I’ve called FastTrade (now at version 1.3). This is the bare-bones design, and “works” perfectly. I was happy with where I’d got it to but didn’t want to be slowed by years old coding decisions. It’s mad how you can have code that works well, but then when you spend a bit of time actually thinking about it, you come up with another solution. Then you think some more, and change it again.

One example of this was my logging code. In OSCAR (a retired bot) I’d written code for each logging event – this wrote a line to my logging sheet individually as they were called, including an xlUp line (obviously this brings into question how serious I am about efficiency). It did work, though writing to the sheet on each log event is not in line with my quest for speed. So, one goal I had was to store log events to an array and paste the array at market change, where trading isn’t happening so speed is less important. I moved all the logging code to it’s own module, added an array, and put the repeated code in it’s own sub. Each logging event, still having its own sub, added a string element such as “Trading Period Started” and called the sub that added all the other detail at that exact moment. I then refined this further by adding a string argument to the sub for the name of the logging event. My logging code has gone from multiple subs to just 2 – the log-event-with-argument sub, and the log-log-to-log-sheet sub. Splendid.

I’ve been building my new bot from the ground up, including the log stuff. I’m calling this one VOLT (VOLume Trader). I’d tested manually and got to a point where live testing was possible. An important element here – always have a boolean argument that allows trading. Set to false and the entire code can run but the bit that writes the trigger/order is bypassed. Once you’re happy to trade, change it to true. Although this isn’t an emergency stop, it does allow for testing and quick interventions if things aren’t quite right but you want monitoring to continue.

Well I’d tested and tested, debugged and tested, debugged etc. Literally as soon as I went live it failed – subscript out of range. I actually like this fault finding part, it’s solving the puzzle that I enjoy. Fortunately for me it was one puzzle after another. Not checking for previous data entries, so writing the same thing over and over, exceeding the array size. Then not incrementing after adding data, so missing loads of log events. Checking for the wrong cased words (Closing vs CLOSING). The list is long but my time is short (a bit dramatic).

I’ve written this post whilst watching VOLT run through Sunday evening US horse markets on another screen and all is now well. Logging is working, as is market navigation, where I appear to have got rid of the double quick-pick-list refresh that caused no problem but I found mildly irritating.. The trigger code is there but not active yet. Next step is to add some temporary what-if logging, see how it fares. Happy trading, botters.

Automated betting

Some changes

Leave a comment

First a small clarification. I said in the last post that when the currently active bot is triggered, it takes the Back offer. However, it doesn’t do that. It places a back bet at 1 tick lower than the best Lay price. In a tight market that is most likely going to take the Back offer as there won’t be any gap in the spread, but I thought I should be clear. It also explains why I don’t always get matched and the fill or kill element is in the order.

Admin news – I’ve stumped up for a personal WordPress plan to get rid of the banner and ads that came with viewing my blog which should make your experience more pleasurable, or less painful. It also came with a new domain name which looks neat, and my view count has jumped, bonus. There may be some decorative changes to come too, if I get on to it.

Admin news extra – Twitter/X has descended to such a low that WordPress no longer connects to it. But it does connect to Bluesky, so I’ve joined that small village scale community…

To the code – it’s always worth going back to basics. Thinking about it, there are elements of my code that I’ve reused without thinking, as they just work. To that end, in all my time botting and with all the different bots I’ve tried, I’ve always used the same worksheet change event code trigger in Sheet1, which is basically this –

Public Sub Worksheet_Change(ByVal Target As Range)

If Target.Columns.Count <> 16 Then Exit Sub

Application.EnableEvents = False

With Workbooks("MyBot.xlsm").Sheets(Target.Worksheet.Name)

MyCode 'this is in module1

End With

Application.EnableEvents = True

End Sub

Well it turns out that the With reference part – “.Sheets(Target.Worksheets.Name)” is a general reference and uses a look-up to get to the specific sheet from within the Sheets collection. However, for my bot it’s always going to be the same sheet so the reference can be changed to “.Worksheets(“Sheet1″)”. This is apparently better form as specific references should be used where possible. By not looking-up, there should also be a speed advantage (possibly multiple microseconds if I’m lucky). Remember, small wins are still wins. Anyhoo, it now looks like this –

Public Sub Worksheet_Change(ByVal Target As Range)

If Target.Columns.Count <> 16 Then Exit Sub

Application.EnableEvents = False

With Workbooks("MyBot.xlsm").Worksheets("Sheet1")

MyCode 'this is in module1

End With

Application.EnableEvents = True

End Sub

Finally – a shout out to Toby at Punter2Pro who featured my blog in an article last year. I’ve had a few click-throughs, so thanks.

Automated betting

A little P&L, mostly L

Leave a comment

The bot has run flawlessly so far, as in it hasn’t failed to navigate the markets or missed any tick offset or stop loss. I do have a chasing stop loss set, and that’s been activated a few times but worked as it should.

The trade/order I’m placing with this set up when triggered is – Take Back offer, fill or kill 3s, 1 tick offset, 2 tick stop loss, 0.5s chasing stop loss, all set for levelling. I’ve made a few adjustments to the trigger to find the limit of effects of different parameters.

In cleaning the code I have got rid of an annoying problem where I’d get an error every time I linked the sheet to Gruss. I’d select any blank cell on the sheet, press Del, and then F5 the code and away it would go, no further problem. It was the reference to the spreadsheet that was incorrect, I’d not included the file extension .xlsm, which makes me wonder how it ever worked. But it did and now it does better.

Here’s a chart. This is for most of March and April up to today. Stakes are around £2. The early sharper drop was done with quite open settings. After reigning it in the losses slowed and remained relatively consistent.

My new bot is coming on with improved logging and some better tracking going on. I’m hoping to have it on the VPS in May for testing. The volume tracking is working but I think I can improve it, specifically the amount of data it holds, which I think can be reduced, but doing that efficiently without slowing the whole thing is my goal.

Automated betting

Not finding errors

I read this while looking for help –

“There’s only two types of programming languages: the ones people complain about, and the ones no one uses.”

I’d like to think it’s true.

I was searching for help on handling error 9 which pops up when referring to an array that isn’t initialised. I was seeing this error when monitoring the MyBets sheet. My code for logging the changes to MyBets works like this –

  • On change to MyBets – lift the entire data range into an array (Array1)
  • Check this data against previous data lift (Array2) to find any changes
  • Log changes to another array (Array3)
  • Once Array1 is checked, place Array3 in another sheet
  • Save Array1 to Array2 for next time
  • Repeat
  • On market change – erase Array2

This doesn’t work on the first change to MyBets on a new market as Array2 isn’t initialised. It could be initialised if it was a known size but this can change dependant on other data I add to Array1. I could write code to size the array on each market entry, but I was looking for a quick way of saying – if referencing the array gives an error, load it with the current data and move on. I found a way to do this using the Not operator. The code looks like this –

If Not Not Array2 = 0 Then…

Now, instead of an error when referencing Array2, this “Not Not” will return a 1 or 0, so the code flows and continues unhindered.

Reading around various forums it’s said that this method is actually a bug from VB6, which is what VBA is based on. And some say that this method may disappear if the bug is fixed. I also read that VB6 is no longer maintained, so the chance of the bug being fixed is low.

For me, this works for now so I’ll use it where I need to.

Automated betting

What’s going on?

1 Comment

[I started this post last week and now I think I’ve solved the problem, but as I’ve written it, you can read it. solution at bottom.]

Over the past two weeks, Oscar, my UK dogs bot, has had six markets failing to trade out or green up. After the first two, I stopped for two days whilst I got round to checking the code but everything was fine, as it should be as I hadn’t changed anything. I restarted and after another few days of trading ok, it happened again, so it’s now stopped. I can’t see anything particular about the markets it’s happening on, and the connection from my VPS appears good. I’m now thinking of adding some extra code to record more details in hope of catching what’s going on.

I’m less narked than you may think as the result so far is I’m up £40 from the errors. But obviously I can’t let it carry on.

In other news, I’ve been reading. A challenging task for someone who flits from one thing to another. Anyway, this had lead to some really good coding sessions (from my point of view) but nothing live yet, only recording and paper trading.

I put quite a bit of time into code that aims to measure the chance of being matched after a trigger is reached and over what period that match would take place. Recording price movement is pretty easy and amount matched over a short period in one direction is also simple. But tracking price, piq and amount matched if the price moves against and then returns, then goes positive, before returning against, is a job for good code. (I’ve not had it marked yet, so no conclusions.)

And I’ve just put on test some code for showing which runner has won on inplay markets. My aim is to know the winner straight away and not have to have a separate process. I have some old code that would take my bets sheet (paper trades) and check it against the RP results page (by runner) after all the races were finished but it requires random fettling to keep up with the website changes. I haven’t used it for some time. It also only happened once a day. My new code works with obvious winners, reaching and staying at 1.01, but then handles close calls individually, with unknowns highlighted for my attention at the moment (this still requires input from me but I’m not checking through every market). I get a rolling update from this.

Inplay is not something I’ve botted with any interest before. But a post on the Gruss forum caught my attention. It regards a simple strategy shown in a YouTube video. It got my interest in the inplay side of things. Rather than throwing bets in and seeing what happens, I thought I’d approach this with a plan and code my way to results rather than bet my way there. If the testing comes to nothing, no waste, I’ve got some cracking reusable code.

[SOLVED, I think]

As mentioned at the top, I think I’ve found the problem.

It’s a time issue. My server clock was 33.7 seconds behind. You can find this out by going to the website – time.is. (My laptop is 0.8s out.) UK dogs are quite sharp with suspending, 30secs+ could easily miss my greening period.

A simple fix? No. Why would it be? Multiple tries at pointing my desktop clock to various internet ntp clocks resulted in a 0.1 second increase in error (huh). I Googled around and found an article on Dimension 4. Downloaded it. This took a few attempts with different servers and then bingo! Back to time.is for an exact result. Bot restarted. Time will tell (time, it’s a time pun, ha ha).

 

Automated betting

Updated updating and simply greening

Leave a comment

First, Liam replied to July ’17

You can set the conflation during the market subscribe request, defaults to 0 but you can choose how often you want to receive it. It’s odd because it’s a handy feature but none of the trading apps seem to use it.

I know betfair use a piece of open source software called Kafka for managing streaming updates but there is probably something else handling slow consumers (conflation). Sockets are complex but at a high level it is my understanding that once the receivers buffer is full the server is told this which then allows the server to halt sending more data. I believe it is then a case of updating that ‘halted’ data to prevent the consumer receiving ‘stale’ data.

Hopefully that makes more sense.

 

Thanks Liam, it does. I appreciate this is more in-depth stuff and beyond most of us but I am interested in these things. It’s the sort of detail that gets exploited by those in the know before anyone else cottons on. I wonder if the lack of utilisation by trading apps is because setting it for optimal performance would be very specific to an individual socket, the machine’s processor, what else it was doing at the time and the level of market activity?

test_passed_stamp

Now, bench testing complete, I’ve put the new Oscar live on the Aus horses only (staged implementation, sounds good). As mentioned in the last post, there are two areas where significant changes have been made to the code.

I trialled two different methods of navigating the markets. One is quite basic and simple code, the other more complex but I thought it would handle any troublesome delays better. It was over complicated though and as I worked through the errors whilst testing, it became bloated and confusing. Impact wise, the simpler code is only one extra step on an if-then-else statement. And it’s the same amount of steps as the old code. I’ve set it to update after a set time, 1:45am, then select first race in list shortly after. I use the day number, as integer, to check if the process has completed today. Previously the list update was performed within a set time period, which was fine when there were regular updates at the desired refresh rate. But with streaming, if there’s a market loaded that’s a few day’s off, as is sometimes the case for big events, there are no regular refreshes. This can mean that the set time window see’s no updates to run the list refresh code.

That change is almost not needed because of the next change. I’ve coded to turn full streaming off when not within the trading time zone. This forces regular refreshes. The above change will still come into use though if there are any issues causing a delay (can’t think what, but the code is better now, I’m sure).

Finally for this update, I’ve employed the green up trigger to replace my code. The module is less than a quarter of it’s previous size and the trigger takes care of cancelling any unmatched bets, submitting a greening trade and then chasing every second until matched. Splendid.

Automated betting

A comment, an icon, rounding and an offer.

1 Comment
Drive Blind commented –

US Horse Racing Off Times have been a problem for bot developers since the year dot. The US off times are just a guide and are not religiously adhered to like the rest of the world. Initially I got around this by polling the Time to Post stat that gives a guide of when the off time is due. This value can be scraped from a number of different sites. Even this was a little hit and miss. What I eventually landed on was waiting till the overround was less than 105% to indicate that the race was about to go off. Works quite well in sparse US markets where all the money comes in at the end before the off.

Thanks for this comment. I’ve looked into scraping times and have found a few sites. Not made any attempts to integrate yet. The idea of watching the overround is very good and something I hadn’t thought of. This goes on the to-do list, thanks.

 

 

I’ve been coding an app in Visual Studio to replicate a spreadsheet I’d made when I was doing the matched betting thing. I added an icon and thought I’d add it to this site, so if you haven’t noticed, it should be at the top of the address bar and look like this –

favicon3

I like the simple look, which is fortunate. And it should show up as different to WordPress in any bookmarks, or in the blog roll on green-all-over (It’s been quiet over there recently).

 

The app I’ve made calculates the lay stake at given odds. A problem I encountered whilst testing my little app was a difference between what I calculated and what I was seeing on the Betfair site. It’s only small but at large stakes it’s noticeable and any discrepancies are worrying, suggesting an error in the formula used. My error came down to rounding. Initially I was just rounding the result only to 2dp. Each step of the formula requires rounding to 2dp to keep it accurate. At no point can you place a lay stake of £50.083333. Attention to detail is key, my mistake.

 

Talking of attention to detail, I recieved an offer to try out a VPS in Dublin with less than 3ms delay. That’s extremely fast for sports trading and probably as good as it gets without co-location (something I’m sure goes on). I had looked at getting a Dublin based VPS when I was setting up but was not willing to pay the obvious premium rate. The offer I received came via Twitter and the first thing anyone does is look at the website of the people making the offer. I remember this company was looking for people to trial its VPS about a year ago. Their website still has some pages that haven’t been altered from the template settings – the blurb you get when you start a website from a pre-made layout offered by these site builder outfits. There’s a lot of trust when putting your bots onto a VPS and you want to know you’re in safe hands that pay attention to detail. I declined the offer.

Automated betting

Speedy data 2

1 Comment

My Speedy data post generated a few comments and some discussion. I really appreciate people taking the time to get involved and share their knowledge and views.

The first comments came via twitter from TraderBot (here and here) with a link to stackoverflow. This is a site I’ve found to be really useful to get help with many programming issues in multiple languages (That reminds me, I keep meaning to do a list of what I use – apps and sites). The Q&As linked to, although relating to Java, are an interesting read with an answer to the speed question, in summary, of “it depends on what exactly you want to do/measure”.

LiamPauling commented on the post asking where I’m hosted and do I stream? I’m cloud hosted and not streaming. He continues that he thinks bottlenecks are more likely elsewhere, which, after further reference in later comments, seems to be a good point.

Betfair Pro Trader asked why I wanted to use an array. It’s not that I want to use an array more than any other data structure, I was looking at getting the best solution, if such a thing exists (which is becoming less clear).

Tony, via Twitter, suggested running a test code with the different structures used. This could be useful but I was put off from this initially by the confusion I was getting from reading differing opinions based on various implementations of arrays, collections and dictionaries (and later, lists). At this point I was thinking that the optimum stucture is dependant on the specific use and there isn’t an exact answer to my speed question.

Next, a comment from Ken. He points to Lists as it’s something that he uses regularly and he talks of some of the benefits. Again, I’d previously come across articles saying lists were slow but maybe I was too quick to dismiss them. Betfair Pro Trader has also suggested using lists and dictionaries combined. Ken adds that he codes in C# (C sharp) but I think for the purpose of data structures and speed they are similar (they, C# and VB.net compile to the same language and run against the same runtime libraries).

n00bmind added a detailed comment. He makes the point that the advantages of one structure over another are not always so, as mentioned above. Also, he goes on to agree with previous comments that my speed question may be missing the main issues – those being the program/algorithm itself and network latency. Further advice is given about profiling (something, as a specific process, I haven’t come across before) and maybe using a different language, such as Python (I have only a basic understanding of Python from messing with it on my Raspberry Pi).

Finally, Jptrader commented, agreeing mostly with n00bmind, and others, about looking at “handling network latency properly and doing performance profiling”.

Although a simple answer hasn’t been found (because there isn’t one), I’m guided by these comments to focus more on my code, handling serialization and latency, making the algorithm efficient and using the data structures that work for now, whether that’s arrays, collections, dictionaries, lists or a combination of. Moving to another language just isn’t feasible for me at the moment, it’s taken me over a year to get a running bot in VB, with limited hobby time. I am happy to accept that another language may have it’s advantages, so would advise others to look at this for optimising their bots performance (for me the advantage will be seen moving from VBA to VB.net).

The testing I’ve done hasn’t shown any particular advantage of the different structures. From my searches on the web I think this could be due to the relatively small amount of data I’m handling (many articles talk of data lines in the 10s to 100s of thousands when comparing structures). An error on my part also had me making double calls for data with my bot which added to my difficulties and questions initially.

I have plenty to be getting on with for now and will continue looking to improve my bots. Thanks again for all the comments.

Automated betting

A chart, I see

Leave a comment

I’ll return to the speed question soon (as I have an idea for a testing bot, just needs writing), the results of the little testing I’ve done are not really good, as in they don’t really move me forward. Development of the VB bot is still progressing though, well my programming ability is, with the more I learn I add/change what I want to do but it’s all good. I’ve been playing with what data to collect/monitor and how to handle it (to avoid unnecessary bloating of the bot). As I like to visualise things, I’ve been presenting data in different ways. Below is one of the charts I created, just with excel, to show matched volumes. It covers the second favourite for the final two minutes before the off in a middle-of-the-road greyhounds race. The data is at 1 second intervals with a 4% decay added to project but not obscure the matching. The price can be seen to rise as it approaches off time (front of image) with last-price-traded at 4.0. It isn’t much use on its own but I like how it looks. Adding another parameter, or two, gives more meaning.

match_vol_dec_chart