Writing a Slogger plugin to import MyFitnessPal data
May 22, 2016I’ve been using MyFitnessPal (MFP) to log exercise and food for a couple of weeks (before that I tried SparkPeople but the app was buggy and the food database wasn’t great), and I really like knowing exactly how many more calories I can eat, as well as keep track of exercise and weight.
Now, it’s great to be able to log stuff using an app with a barcode scanner, and then view stats and info online, but I don’t like the idea of my data all being locked away and belonging to someone else, and what it I want to use a different service? Basically, I want to feel like I own this data.
My go to logging tool is Brett Terpstra’s Slogger, which scrapes various online services and then spits the results into Day One, but there’s no plugin for MFP. So, after a quick peek at some other plugins I decided to write one.
Data
If you can cope with the idea of your diary data being public, you can get a simple HTML view of your daily food intake and exercise by visiting a URL like this:
http://www.myfitnesspal.com/reports/printable_diary/mattpfoster877
This page has a fairly large amount of markup for what’s basically two tables,
but the tables are just that, and they’ve got handy id attributes so should
be nice and easy to find using a scraping library.
My aim is to take the usable data from this page, convert it to Markdown, and then save it to Day One, using Slogger. I’m going to do this by writing a plugin.
As an aside, I’m happy with this sort of data being visible to anyone, because frankly, who is going to care? If I wasn’t, I’d have to work out how MFP’s session management worked, and probably use something like Mechanize, to login using a POST request, ensuring that whatever anti-CSRF mechanism MFP uses is obeyed, and then save a session cookie or two for subsequent requests. This is all very doable, but extra effort!
Scraping and Munging
For this project, I’m using Ruby, because that’s what Slogger is written in. I still love Ruby, but have more experience writing perl these days, so forgive me if my Ruby code looks a bit perly!
I decided to use the Nokogiri library to parse the HTML and extract the data I want.
The two tables look like this:
<table class="table0" id="food">
and
<table class="table0" id="excercise">
So, it’s pretty easy to extract what we want using CSS selectors (note the misspelling of ‘excercise’!), and code like this will do it:
doc = Nokogiri::HTML(open(url))
food = doc.css('table#food').to_s
exercise = doc.css('table#excercise').to_s
This will save the tables, as HTML, into the food and exercise variables.
Next, I need to turn these tables into Markdown. Now, I could probably find a
gem to do this for me, or implement something myself, but I decided that
since I’m using one of Brett’s tools I may as well also use another, and went
with the online converter Marky.
Basically, you pass its API an urlencoded string and it spits back markdown for you. So you can do this with a method like this:
def to_md(str)
str = URI.escape(str)
to_md_url = 'http://fuckyeahmarkdown.com/go/'
conv = open(to_md_url + '?html=' + str)
conv.read
end
The open function is part of
open-uri,
which is in the standard library. It would be good practice to check the
response code here, but I’m being lazy! Also, the docs for the library are
really sparse, so if you’re doing any Ruby programming do yourself a huge
favour and buy
Programming Ruby.
Now, chaining these two together with a bit of other glue code gives me the following:
#! /usr/bin/env ruby
require 'open-uri'
require 'nokogiri'
def build_report_url(
username,
baseurl = 'http://www.myfitnesspal.com/reports/printable_diary/',
date = Time.now.strftime("%Y-%m-%d")
)
baseurl + username + '?from=' + date + '&to=' + date
end
def grab(url)
doc = Nokogiri::HTML(open(url))
if doc.to_s.include?('This Username is Invalid')
abort "No data, check username is correct and diary is public"
end
log = {}
log[:food] = doc.css('table#food').to_s
# Note: the div name is spelt wrong!
log[:exercise] = doc.css('table#excercise').to_s
log
end
def to_md(str)
str = URI.escape(str)
to_md_url = 'http://fuckyeahmarkdown.com/go/'
conv = open(to_md_url + '?html=' + str)
conv.read
end
username = 'mattpfoster877'
url = build_report_url(username)
logs = grab(url)
puts '# MyFitnessPal daily report'
puts
puts "User: [#{username}](http://www.myfitnesspal.com/food/diary/#{username})"
logs.each do |name, table|
puts
puts '## ' + name.to_s.capitalize
puts to_md(table)
end
Which prints something like this when run:
## MyFitnessPal daily report
User: [mattpfoster877](http://www.myfitnesspal.com/food/diary/mattpfoster877)
### Food
| Foods | Calories | Carbs | Fat | Protein | Cholest | Sodium | Sugars | Fiber |
| --------------------------------------------------------------------- | -------- | ----- | --- | ------- | ------- | ------- | ------ | ----- |
| Breakfast | | | | | | | | |
| Hovis - Hovis Wholemeal Seed Sensation Seven Seeds Toast, 2 slice 44g | 218 | 28g | 5g | 10g | 0mg | 322mg | 3g | 11g |
| Butter With Salt - Butter, 15 g | 63 | 0g | 17g | 0g | 45mg | 135mg | 0g | 0g |
| Eggs - Scrambled (whole egg), 1.5 large | 152 | 2g | 11g | 10g | 322mg | 256mg | 2g | 0g |
| Lunch | | | | | | | | |
| mission - deli wrap multigrain, 1 wrap | 183 | 31g | 4g | 5g | 0mg | 0mg | 0g | 2g |
| Aconbury - Organic Mixed Bean Sprouts, 50 g | 70 | 11g | 1g | 4g | 0mg | 40mg | 0g | 1g |
| Carrots, baby, raw, 1 large | 5 | 1g | 0g | 0g | 0mg | 12mg | 1g | 0g |
| Quorn - Meat Free Chicken Style Pieces, 75 g | 72 | 1g | 2g | 10g | 0mg | 180mg | 1g | 5g |
| Dinner | | | | | | | | |
| Chicken Breast - Roasted - Chicken Breast, 200 g | 394 | 0g | 16g | 60g | 168mg | 142mg | 0g | 0g |
| Potato - Mashed Homemade, 270 g | 260 | 52g | 8g | 10g | 10mg | 200mg | 2g | 4g |
| Snacks | | | | | | | | |
| Tesco Everyday Value - Lightly Salted Tortillas Jaffa, 40 g | 189 | 26g | 8g | 3g | 0mg | 320mg | 1g | 2g |
| Naked - Cocoa Crunch Bar, 30 g | 106 | 14g | 3g | 6g | 0mg | 0mg | 13g | 2g |
| Ryvita - Thins Sweet Chilli Flatbreads, 1 thin, 8g | 32 | 6g | 1g | 1g | 0mg | 1mg | 1g | 0g |
| TOTAL: | 1,744 | 172g | 76g | 119g | 545mg | 1,608mg | 24g | 27g |
### Exercise
| Exercises | Calories | Minutes | Sets | Reps | Weight |
| -------------------------- | -------- | ------- | ---- | ---- | ------ |
| Cardiovascular | | | | | |
| MFP iOS calorie adjustment | 150 | 1 | |
| Strength Training | | | |
| Beast maker | | 5 | 5 | |
| TOTALS: | 150 | 1 | 5 | 5 | 0 |
A Plugin
This is exactly what I’m looking for, so now I just need to plug my proof of concept into a Slogger plugin. To do that, I copied the plugin template, and populated the comments and config sections. Then I added all my helper functions and instead of printing the markdown to stdout I pushed it all into a string which I then used to create a diary entry. The guts of the plugin essentially boil down to this:
url = build_report_url(username, today.strftime('%Y-%m-%d'))
logs = grab(url)
entry = "## MyFitnessPal daily report\n\n"
entry +="User: [#{username}](http://www.myfitnesspal.com/food/diary/#{username})\n"
logs.each do |name, table|
entry += "\n### " + name.to_s.capitalize
entry += to_md(table)
end
entry += "\n\n#{tags}"
DayOne.new.to_dayone({ 'content' => entry })
Which is pretty compact, and works well too! It’s almost exactly as above, but
the today variable is set by slogger.
For now, this plugin is only available in my fork of
Slogger. If you want to use it, grab
mfplogger.rb,
put it in your slogger plugins directory, and run ./slogger to have it
generate the config stubs you need. Then edit the slogger_config to set your
username.
My config looks like this:
MFPLogger:
description:
- MyFitnessPal Logger
- Your diary must be public. Set at http://www.myfitnesspal.com/account/diary_settings
MFPLogger_last_run: Sat May 21 23:50:13 2016
service_username: mattpfoster877
tags: "#social #fitness"
I plan to write a couple more plugins, but once I’ve done that I’ll submit a github pull request and hopefully get this merged into the main repo!