Looks like gradient boosted trees did really well! XGBoost and LightGBM took the top two spots on average error, doing better than the more computationally expensive transformer models.
They did weaker on low-glucose detection, which is pretty bad in this particular case. But it's possible that's fixable by tweaking the loss function to care more about detecting low glucose. As far as I can tell, there's no structural reason the transformer model did better on low glucose, probably just that it was pretrained on a synthetic dataset that had more lows than the author's data.
I love gradient boosted trees because they're fast, easy to train and unreasonably good at tabular and forecasting problems like this one. Of course they can't compete with transformers for complex tasks like writing code, but they're amazing specialists.
It's interesting how "predictive" AI (classical classification and regression tasks) was the rage for a while before "generative" AI came onto the scene. Having said that there isn't a super clear dichotomy since generative AI uses many predictive tools.
89% TIR, well done! Cool to see the research, I’m surprised there aren’t more in-depth n=1 studies out there from statistics/data-savvy diabetics. I built a small dashboard exactly collating carb, insulin, sleep, and movement data and trying to coax out useful correlations to help manage. Some interesting insights, but clearly I should eat less and move more. Keep up the good work, and thanks for sharing!
Ironically, did the same thing and ended up with the same answer.
From wearing a CGM, whoop, using an impedance analyzing scale (I know, I know), blood tests - all the data shows pretty much the same correlation. I mostly did it for fun, but also because I wanted to "figure out how different things impact my body".
While yes, there were some interesting discoveries like:
- Some things raise my blood glucose more than I'd expect, pasta for example - those were the things I usually "knew" and foods I'd avoid eating.
- Sleep quality only had correlation with amount of activity/time I went to sleep, so my food and caffeine choice didn't impact it much.
The results were pretty much:
Eating bad and moving less is bad for you. Eating good and moving more is good. What your intuition tells you is healthy and good for you, usually is. What it says isn't, usually isn't.
Could have saved me some time and money, but at least I had fun.
Yeah I'm thinking of adding more (especially recent) models and compare their published metrics with those of mine to see how they hold up. Then I'll likely post about it here again. See you then!
Incredible work! As a T1D I've been gathering as much data as possible (blood glucose, insulin-on-board, food consumed, heart rate, skin temperature, etc.) for exactly this purpose. Can't wait to dive in!
BG forecasting and insulin response has always seemed to me to be such a personal thing (everyone has such a different response based on the various factors, and there are _so_ many factors) that the one-size-fits-all models in CGMs seemed the lowest common denominator. We live in an age where personal data collection is easier than ever (with wearable BLE hardware) and timeseries forecasting models are better than ever, making this research even more valuable.
Thanks, N=1 training is really the way to go. In my case training on large N >> 1 (i.e. metabonet dataset) actually hurt the performance of the models predicting my own blood sugar.
You don't even need a datacenter GPU. I trained all of my models on a laptop RTX 4070.
Wow this is very impressive work. I think it's the most impressive thing I've seen on HN. I don't understand all the data and what the data all mean. But it's very clear from the plots, tables, and references that you put in a lot of time to understand the science behind all this data; especially since I'm guessing you had to teach yourself all of this to make such a good model. You don't see that level of sophistication very often on HN. Bravo!
Extremely interesting, thanks for sharing!
Have you looked at Timestral at all by any chance?[0]
I'm only dipping my toes into time series models, for somewhat similar reasons.
TSFMs are really really big. My motivation for training smaller models is because I want to run them locally on my smartphone. But yes, I plan to add TSFMs to the github repo soon.
It's so cool. I appreciate the detailed architecture doc. I wonder why the time of day is getting predicted rather than provided in data. Did that perform worse? Or is it to explicitly account for people with uncommon schedules?
When I began this project my model was indeed being provided time-of-day features (they were sinusoidal like in the published literature). Then one day I decided to remove it to see what happens. The accuracy didn't degrade so I thought hmm maybe it's because the model can *infer* time-of-day from past blood sugar alone? So I attached a time-of-day head to test it and it worked!
Interestingly I have noticed that I get sleepy when the model's predicted clock gets close to midnight rather than the actual clock when I travel between time zones.
cool to find this project. I'm implementing literally the same thing recently. I'm using exact same insulins. My setup is a bit different and a bit behind you in terms of model quality.
I run pretrained models on raspberry pi, so raspberry acts as server. I use dexcom, garmin mcp to gather all data. Android phone acts as dumb client. I have create a custom app to log carbs, insulin injections, predict insulin doses from a food photo.
There are also some nighty jobs to recalculate insulin sensitivity, retrain model, generate daily report.
I started logging carbs and insulin doses few weeks ago, so currently there is not enough personal data to train prediction model. I use oref0 algorithm for predictions currently. As a starting point i have created prediction model solely on glucose values. Training on other people data, did not helped me to make predictions accurate.
I also didn't have enough personal data to train on (transformers are very data-hungry), so I created a huge synthetic dataset to help me with that. I use that for pretraining and then plug in a LoRA that got trained using my personal data.
I love this! I'm a T1D as well, and really appreciate N=1 work understanding our own bodies with this (and other) conditions.
Do you mind if I share with an open healthware group that I'm a part of?(https://healthware.oshwa.org/) This is right up their alley
Feel free to share! I am currently testing Mistral's newly released TSFMs (Time-Series Foundation Models) on my blood sugar the results of which will be in the repo in the next couple of days.
Is there anyway way to use xdrip or nightscout or similar to move the data into your (android) app?
I already use the official Freestyle Librelink app plus jugglucoNG (which btw is amazing if you don't use, has a lot of nice stats), and the latter can already export its data streams.
Side note, I'm very happy to see more (diy) work in this field (ever since AAPS I think). #wearenotwaiting !
(Hashtag context for non T1Ds: diy software, unsurprisingly, has been leagues ahead in capabilities than "official" software.)
You can copy the data from the app to nightscout, but I haven't added the reverse direction. My app allows me to read from multiple CGMs simultaneously (I did this because different CGM models have their own algorithms and you don't want your machine learning model to be fitted to one particular algorithm).
I do a lot of reverse engineering to connect my app directly to the CGM sensors, bypassing official vendor solutions (they suck anyway).
Exercise can have a substantial effect on glucose. Have you considered including physical activity as an input feature, e.g. from a smartwatch (heart rate, steps, activity intensity)? Or is that data simply not available?
I absolutely hate exercise. I mean as a machine-learning guy it is the hardest quantifiable feature and yet it causes large swings in blood sugar. I haven't added exercise as additional input yet because I'm still searching for a better way to quantify it.
Is heartrate not a decent first approximation? I wouldn't have guessed it would be the best way, but I would have guessed it would be a decent first attempt and would increase model performance.
Unfortunately no, well at least not in my case. My heartrate gets higher before I wake up, and incidentally this is also the time when there is a higher likelihood of getting high blood sugar.
Regarding exercise, my blood sugar gets higher during anaerobic exercises but lower when doing aerobics.
At the moment I consider heartrate a confounding variable rather than a good predictor of BG. I think it could work for sedentary populations for whom higher heartrate correlates well with some kind of aerobic activity.
Do you have a smart watch, or a Fitbit, or an Apple watch? You can get heart rate data, or steps per minute, which are pretty good proxies for metabolic activity. Garmin has a free personal API you can use, Fitbit routes through Google Health, and Apple Watch of course goes through Apple Health. Even beyond "exercise" in the traditional sense, my guess is that walking (steps per min), or even just lying/sitting/standing indicators (present in some watches), would improve prediction a lot.
I have done a lot of work on health wearables, and "predict the time series" does a lot better when you can cross-reference it against the biological drivers of the function you're looking at. The other obvious one here would be meals - hard to think of how to flag that properly, lots of options.
Hmm maybe I should build my own wearable using an ESP32 + heart rate sensor? I'm already pretty good at embedded stuff and it would be a nice weekend project.
I would highly recommend a Garmin Smartwatch with 1+ week of battery life. They have built-in models that are very good at measuring activity from heart rate + accelerometer data. I use it for calorie tracking among other things and it works very well.
You can directly export the timeseries over USB (though I just download from Garmin's portal which syncs via bluetooth/wifi.) It also shouldn't be hard to write your own bluetooth/wifi sync.
I spent a long time trying to build predictive models that use complete and accurate carbs/protein/fat macronutrient data as well as insulin delivery. I found that there were enough non-linearities and other unmodelable factors at work in the BG data that led to very poor predictive ability. Considering protein & fat had a nil-to-negative effect on the models.
It's possible that accounting for heart rate as well as carbs & insulin might have better results, with enough data.
Yeah I was using the macro-nutrient approach as well, but then switched to simply providing GI (glycemic index) alongside the carbs in grams and found out that there was practically no difference between the two.
During training I artificially wobble the carb amounts so they are more like guesstimates than true carbs. This also partially fixed the quantile bands (before that they were too narrow).
Helping hands are always needed. You can test the models on your own data locally and tell me how they work for you. I have 2 kinds of models in the repo: those trained on the metabonet dataset which consists mostly of AID, and mine which is MDI.
Have you ever thought about fixing the underlying issue instead? Maybe it makes this whole project obsolete. I know T1D can't be fixed, but using ketogenic diet should have much smaller peaks/lows & more stable blood glucose in all cases.
Removing carbs will remove most (not all) need for insulin.
Doing keto diets with T1D can be extremely risky because it’s easy to fool yourself that you’re getting enough insulin. Diabetic ketoacidosis (how untreated T1D kills) stems from generally a lack of sufficient insulin, is characterized by a buildup of ketones in the bloodstream (among other things), and requires a multi-day hospital stay to treat it, assuming you’re lucky to catch it early enough and have access to medical help.
I have heard of T1D’s who have done this successfully but afaik it’s not a widely encouraged practice. Also, as I’m sure you’re aware, it makes social eating way more complicated than most people can comfortably tolerate.
A dream of mine but sadly we are still quite a ways off…
Have you read up on any of the Faustman lab work on using the BCG vaccine to treat T1D? I’m cautiously hopefully but not sure if it can or should be believed…
I have done keto for 4+ years, and have done ~2 years epilepsy keto, where the measurement was to have blood ketones 2+ & glucose 4 (mmol/l) , 1 hour after wakeup.
Epilepsy-keto is in the same realm as this, but even harder, because you need high ketones & low glucose. You're literally eating 10-20g carbs a day. The maximum that I ever ate while maintaining epilepsy keto was 50g carbs/day, and it was only possible by doing heavy exercise (120 squats with your body weight in the bar).
I don't have experience of epilepsy so wouldn't assume about its challenges, but do have T1D. I've done some long periods of strict keto, and it does have some big positives, but I don't find it a great long term solution.
They did weaker on low-glucose detection, which is pretty bad in this particular case. But it's possible that's fixable by tweaking the loss function to care more about detecting low glucose. As far as I can tell, there's no structural reason the transformer model did better on low glucose, probably just that it was pretrained on a synthetic dataset that had more lows than the author's data.
I love gradient boosted trees because they're fast, easy to train and unreasonably good at tabular and forecasting problems like this one. Of course they can't compete with transformers for complex tasks like writing code, but they're amazing specialists.
From wearing a CGM, whoop, using an impedance analyzing scale (I know, I know), blood tests - all the data shows pretty much the same correlation. I mostly did it for fun, but also because I wanted to "figure out how different things impact my body".
While yes, there were some interesting discoveries like: - Some things raise my blood glucose more than I'd expect, pasta for example - those were the things I usually "knew" and foods I'd avoid eating.
- Sleep quality only had correlation with amount of activity/time I went to sleep, so my food and caffeine choice didn't impact it much.
The results were pretty much: Eating bad and moving less is bad for you. Eating good and moving more is good. What your intuition tells you is healthy and good for you, usually is. What it says isn't, usually isn't.
Could have saved me some time and money, but at least I had fun.
BG forecasting and insulin response has always seemed to me to be such a personal thing (everyone has such a different response based on the various factors, and there are _so_ many factors) that the one-size-fits-all models in CGMs seemed the lowest common denominator. We live in an age where personal data collection is easier than ever (with wearable BLE hardware) and timeseries forecasting models are better than ever, making this research even more valuable.
Thanks so much for publishing your work!
You don't even need a datacenter GPU. I trained all of my models on a laptop RTX 4070.
[0]: https://mistral.ai/news/timestral/
Interestingly I have noticed that I get sleepy when the model's predicted clock gets close to midnight rather than the actual clock when I travel between time zones.
Do you have a GitHub repo or something btw?
Is there anyway way to use xdrip or nightscout or similar to move the data into your (android) app?
I already use the official Freestyle Librelink app plus jugglucoNG (which btw is amazing if you don't use, has a lot of nice stats), and the latter can already export its data streams.
Side note, I'm very happy to see more (diy) work in this field (ever since AAPS I think). #wearenotwaiting !
(Hashtag context for non T1Ds: diy software, unsurprisingly, has been leagues ahead in capabilities than "official" software.)
You can copy the data from the app to nightscout, but I haven't added the reverse direction. My app allows me to read from multiple CGMs simultaneously (I did this because different CGM models have their own algorithms and you don't want your machine learning model to be fitted to one particular algorithm).
I do a lot of reverse engineering to connect my app directly to the CGM sensors, bypassing official vendor solutions (they suck anyway).
Regarding exercise, my blood sugar gets higher during anaerobic exercises but lower when doing aerobics.
At the moment I consider heartrate a confounding variable rather than a good predictor of BG. I think it could work for sedentary populations for whom higher heartrate correlates well with some kind of aerobic activity.
I have done a lot of work on health wearables, and "predict the time series" does a lot better when you can cross-reference it against the biological drivers of the function you're looking at. The other obvious one here would be meals - hard to think of how to flag that properly, lots of options.
You can directly export the timeseries over USB (though I just download from Garmin's portal which syncs via bluetooth/wifi.) It also shouldn't be hard to write your own bluetooth/wifi sync.
It's possible that accounting for heart rate as well as carbs & insulin might have better results, with enough data.
0xdeadf1sh: good luck with your quest!
During training I artificially wobble the carb amounts so they are more like guesstimates than true carbs. This also partially fixed the quantile bands (before that they were too narrow).
JK, as the spouse of someone with T1D I really appreciate seeing this!
Removing carbs will remove most (not all) need for insulin.
I have heard of T1D’s who have done this successfully but afaik it’s not a widely encouraged practice. Also, as I’m sure you’re aware, it makes social eating way more complicated than most people can comfortably tolerate.
Have you read up on any of the Faustman lab work on using the BCG vaccine to treat T1D? I’m cautiously hopefully but not sure if it can or should be believed…
Epilepsy-keto is in the same realm as this, but even harder, because you need high ketones & low glucose. You're literally eating 10-20g carbs a day. The maximum that I ever ate while maintaining epilepsy keto was 50g carbs/day, and it was only possible by doing heavy exercise (120 squats with your body weight in the bar).
What are the issues for long term?