Howto: Track your practice with WordPress and gnuplot

graphs

Here’s how to set up a system with pretty graphs to track your practice time and display it at your WordPress site!  I’m using mine to track the time I spend each day on four different categories of practice, and also show my progress toward the goal of 10,000 hours of total practice.  Every day I add my practice times to a log file and the system generates updated graphs nightly.  When you visit my blog site, the running tally is given in the header (or anywhere else I want it on the website… like here! (5045.95).  And the graphs at my “10,000 hours of taiko” blog post are also automatically updated every night.

Requirements

The following things should be installed on your system. You should have root access to your server.

  • WordPress
  • Gnuplot
  • standard gnu/linux utilities (awk, cron, etc)

Create a directory on your server where all the data files, scripts, and gnuplot commands will live. I created /var/www/practiceGraph/.

Overview

Every day I add my practice hours to a file called prac_data.dat on the wordpress server. I’m tracking how many hours I spend doing four different types of practice. I want the system to help me see how I’m using my practice time between these categories, and also a grand total of all the hours I practice. A few days in prac_data.dat looks like this:

090921 0 1.6 0 0
090922 .4 0 0 1
090923 1.1 0 0 0

At 2am every night, two scripts run automatically:

  • generate_practice_times.sh
  • generate_cumulative_practice_data.sh

The generate_practice_times.sh script tallies the total practice hours and puts them in a file called times.php. The times.php file assigns these tallies to “short-codes” so I can access them from within WordPress. The other scipt, generate_cumulative_practice_data.sh, preps the data in prac_data.dat so it is easy to create a cumulative graph of total practice progress. After these scripts have run, the gnuplot program creates up-to-date graphs, over-writing the old versions.

Scripts / files / data

If you’re fairly comfortable with gnuplot, WordPress, and bash scripts, you can probably figure out how the system is running just from the scripts themselves. Here are all the files used to cobble together the system. The two bash scripts (.sh files) and the one php file should have permissions that allow your regular user to run them (“chmod 755 generate_cumulative_practice.sh generate_practice_times.sh times.php”).

generate_practice_times.sh
generate_cumulative_practice_data.sh
times.php (right-click and “Save As”)
1_hour_day_line.dat
2_hour_day_line.dat
practice_graph_progress.gpi
practice_graph_daily.gpi
prac_data.dat
prac_accumulated.dat

Recording your hours

As I mentioned previously, I’m tracking the time spent doing four different types of practice. Of course, you could track anything, miles jogged, books read, etc, and can track any number of categories you wish.

The data is stored in a format that gnuplot can read to generate the graphs, in a file called prac_data.dat, on the wordpress server in /var/www/practiceGraph/. Each line of prac_data.dat represents one day’s numbers, with the date at the beginning of the line in the format yymmdd. There can be any amount of white space between the numbers within a row. Here are a few lines from my file.

091016 3.1 0 0 0
091017 4.3 0 0 0
091018 .2 0 0 .1 # Err perf so-so
091019 .5 0 0 0
091020 0 0 0 0 # recent focus on practice graphing system, irony!

The hash mark can be used for adding comments to the file that gnuplot will ignore. I’ve started using these comments as a sort of practice diary, though I haven’t figured out what I’ll do with all these notes yet.

Note: When you input your practice times, add a carriage return at the end of the line so there’s a single newline at the end of the file.  For some reason the generate_practice_times.sh script will not read the last line if it is also the end of the file.  (I think this is because the EOF triggers the “while read” command to stop before it gets a chance to add that line’s numbers.)  This is not too big a deal to me… if the carriage return is forgotten it simply means the tally and graphs are behind by one day.

I use the incredible text editor emacs (no offense, I like vim too!) and tramp to edit the file remotely from my workstation on a daily basis.

Tallying the numbers

The generate_practice_times.sh file is a basic shell script that reads through prac_data.dat and adds up each category as well as the total of all categories. It then writes these numbers out to a file called times.php. More on times.php in a moment. Every night, this script runs first, so that the tallied numbers are up-to-date when gnuplot when it generates the graphs. Here is the generate_practice_times.sh script.

#!/bin/bash
time_inputs=”/var/www/practiceGraph/prac_data.dat”
time_file=”/var/www/practiceGraph/times.php”

line_num=0
personal_practice=0
onensemble_practice=0
group_practice=0
live_practice=0

while read day
do
day_times=( ${day[@]} )

if [[ ${day_times[1]} = *[0-9]* ]]; then
personal_practice=$(bc <<< “${personal_practice} + ${day_times[1]}”)
fi
if [[ ${day_times[2]} = *[0-9]* ]]; then
onensemble_practice=$(bc <<< “$onensemble_practice + ${day_times[2]}”)
fi
if [[ ${day_times[3]} = *[0-9]* ]]; then
group_practice=$(bc <<< “$group_practice + ${day_times[3]}”)
fi
if [[ ${day_times[4]} = *[0-9]* ]]; then
live_practice=$(bc <<< “$live_practice + ${day_times[4]}”)
fi
done < <(cat “$time_inputs”)

total_practice_time=$(bc <<< “$personal_practice + $onensemble_practice + $group_practice + $live_practice + 4218″)

echo ‘<?php’ > $time_file
echo ‘$personal_practice_time='”$personal_practice;” >> $time_file
echo ‘$onensemble_practice_time='”$onensemble_practice;” >> $time_file
echo ‘$group_practice_time='”$group_practice;” >> $time_file
echo ‘$live_practice_time='”$live_practice;” >> $time_file
echo ‘$total_practice_time='”$total_practice_time;” >> $time_file
echo ‘?>’ >> $time_file

The echo commands of the above script write the tallies to the times.php file. The result looks like this.

<?php
$personal_practice_time=92.69;
$onensemble_practice_time=55.45;
$group_practice_time=26.20;
$live_practice_time=15.30;
$total_practice_time=4407.64;
?>

Generating cumulative data

One more script processes the data before gnuplot can create the up-to-date graphs. In order to generate a graph that shows the total number of hours accumulating over time, we need a data file that contains the total hours by date. In other words, we need our data to show that on Sept 17, 2009, we had logged a total of 4338.64 hours to date… like this:

090917 4338.64
090918 4342.14
090919 4344.34
090920 4349.09
090921 4350.09

To generate this data, a one-line awk command reads through prac_data.dat one line at a time, and replaces the four numbers on each line with a cumulative total for that day. In my case, I had a certain number of hours prior to implementing this system, so awk adds this number to each row’s total. These new cumulative totals get written to the prac_accumulated.dat file. This data will be fed to gnuplot in the practice_graph_progress.gpi file to generate the cumulative practice graph. Here is the generate_cumulative_practice_data.sh script.

#!/bin/bash

cp /var/www/practiceGraph/prac_data.dat /tmp/
mv /var/www/practiceGraph/prac_accumulated.dat /var/www/practiceGraph/prac_accumulated.old

awk ‘{x=x+$2+$3+$4+$5; print $1,x+4218}’ /tmp/prac_data.dat > /var/www/practiceGraph/prac_accumulated.dat

Generating the total progress graph

If you’re new to gnuplot, start by reading this excellent tutorial. Gnuplot is an amazing program but is tricky to get the hang of. Thankfully, there is an active and helpful community of users in the gnuplot forums.

The practice_graph_progress.gpi is a gnuplot command to create the 10,000 hours progress graph. It took a lot of tweaking to get it working properly but the initial elements of the command should be self-explanatory. I don’t exactly understand every detail of the “plot” function, but feel free to ask if you have questions. Here are the contents of practice_graph_progress.gpi.

set terminal jpeg medium
set output “/var/www/wp/wp-content/uploads/2009/10/practice_graph_progress.jpg”

set xlabel “Date”
set ylabel “Hours”

set xdata time
set timefmt “%y%m%d”
set yrange [3000:10000]
set xrange [“090622″:”270101”]
set xtics rotate

plot ‘/var/www/practiceGraph/prac_accumulated.dat’ using 1:2 with lines lt 1 lw 3 title “Actual practice progress”, ‘/var/www/practiceGraph/1_hour_day_line.dat’ using 1:2 with lines lt 8 lw 1 title “Predicted progress at .91 hours/day”, ‘/var/www/practiceGraph/2_hour_day_line.dat’ using 1:2 with lines lt 5 lw 1 title “Predicted progress at 2.05 hours/day”

In addition to plotting my actual practice hours, I wanted two separate lines predicting my progress based on plausible daily practice averages. On my own I figured out how many days it would take to cross the 10,000 hour threshold at 1 hour a day and created a very simple data set called 1_hour_day_line.dat to represent that progress.

091020 4393.54
260814 10000

I then calculated how long it would take me to reach 10,000 practicing 2 hours per day and created a similar data set called 2_hour_day_line.dat.

091020 4393.54
170408 10000

Generating the practice categories graph

The practice categories graph shows a daily tally of each category so I can see where my practice time is focused. It is generated by the practice_graph_daily.gpi script. I had to get help from the gnuplot community for this one. I got confused trying to create the graph with the “histogram rowstacked” style but it turns out that “boxes” mode works just fine. Here are the contents of practice_graph_daily.gpi.

set term jpeg large
set output “/var/www/wp/wp-content/uploads/2009/08/practice_graph.jpg”

set ylabel “Hours”
set xlabel “Date”
set style fill solid 1 border -1
set xdata time
set timefmt “%y%m%d”
set xtics rotate

plot ‘/var/www/practiceGraph/prac_data.dat’ using 1:($2+$3+$4+$5) title “Live performance” with boxes, ” using 1:($2+$3+$4) title “Other group practice” with boxes, ” using 1:($2+$3)title “On Ensemble practice” with boxes, ” using 1:2 title “Personal practice” with boxes

Automating the process

We use the wonderful “cron” to schedule the running of the scripts on a nightly basis. The generate_cumulative_practice_data.sh script must run first, so I scheduled it for 1:55am, and the remainder of the scripts for 2am.

55 1 * * * /var/www/practiceGraph/generate_cumulative_practice_data.sh
0 2 * * * /var/www/practiceGraph/generate_practice_times.sh
0 2 * * * /usr/bin/gnuplot /var/www/practiceGraph/practice_graph_daily.gpi
0 2 * * * /usr/bin/gnuplot /var/www/practiceGraph/practice_graph_progress.gpi

To edit your cron entries, run “crontab -e” as your regular user and copy and paste the four lines above (assuming the directory locations are correct on your system as well). Double-check that your user has permission to run the two bash scripts and write access to the location where the graphs will be put. This is set in the .gpi files (and in my case is /var/www/wp/wp-content/uploads/2009/08/practice_graph.jpg and /var/www/wp/wp-content/uploads/2009/10/practice_graph_progress.jpg)

Using the tallies and graphs within WordPress

To access the tallies in times.php and to insert the graphs you have generated with gnuplot, we will use WordPress “shortcodes”. Here is a useful overview of shortcodes.

At the top of my /var/www/wp/wp-content/themes/default/functions.php file, we’ve added the following.

function total_practice_time() {
include “/var/www/practiceGraph/times.php”;
return “$total_practice_time”;
}
function personal_practice_time() {
include “/var/www/practiceGraph/times.php”;
return “$personal_practice_time”;
}
function onensemble_practice_time() {
include “/var/www/practiceGraph/times.php”;
return “$onensemble_practice_time”;
}
function group_practice_time() {
include “/var/www/practiceGraph/times.php”;
return “$group_practice_time”;
}
function live_practice_time() {
include “/var/www/practiceGraph/times.php”;
return “$live_practice_time”;
}
function practice_graph() {
$img_mod_date=system(“date -r /var/www/wp/wp-content/uploads/2009/08/practice_graph.jpg”);
$img_html='<img src=”http://onensemble.org/wp-content/uploads/2009/08/practice_graph.jpg”>’;
return “$img_html”.”<br />Graph generated: $img_mod_date”;
}
function practice_graph_progress() {
$img_mod_date=system(“date -r /var/www/wp/wp-content/uploads/2009/10/practice_graph_progress.jpg
$img_html='<img src=”http://onensemble.org/wp-content/uploads/2009/10/practice_graph_progress.jpg”>’;
return “$img_html”.”<br />Graph generated: $img_mod_date”;
}

add_shortcode(‘personal_practice_time’,’personal_practice_time’);
add_shortcode(‘onensemble_practice_time’,’onensemble_practice_time’);
add_shortcode(‘group_practice_time’,’group_practice_time’);
add_shortcode(‘live_practice_time’,’live_practice_time’);
add_shortcode(‘total_practice_time’,’total_practice_time’);
add_shortcode(‘practice_graph’,’practice_graph’);
add_shortcode(‘practice_graph_progress’,’practice_graph_progress’);

With this in place, you can refer to the tallies or the graphs anywhere in WordPress using the following shortcodes. I’ve added a space between the “[” and the next character to keep them from becoming valid shortcodes. When you use them, there should be no spaces.

[ personal_practice_time]
[ onensemble_practice_time]
[ group_practice_time]
[ live_practice_time]
[ total_practice_time]
[ practice_graph]
[ practice_graph_progress]

I also wanted to have my tally in the header of my personal blog site, so we edited /var/www/wp/wp-content/themes/default/category-13.php to add the following.

<p class=”totalPracticeTime”><?php echo total_practice_time(); ?> hours and counting!</p>
<p class=”totalPracticeTimeURL”><a href=”http://onensemble.org/2009/10/10000-hours-of-taiko/”>Read more about Kris’ 10,000 hours of practice</a></p>

In order to make the text large, red, and right justified, we created a few new CSS classes at the bottom of /var/www/wp/wp-content/themes/default/style.css .

.totalPracticeTime {
color: #FF0000;
text-align: right;
font-size: 20px !important;
margin: 0;
padding: 0;
}
.totalPracticeTimeLeft {
color: #FF0000;
font-size: 26px !important;
margin: 10;
}
.totalPracticeTimeURL {
color: #0000FF;
text-align: right;
font-size: 12px !important;
margin: 0;
padding: 0;
}

Conclusion

Although I’ve only used it for a week or so, I am very pleased with the system. The daily input of the data through Emacs+Tramp is quick and easy, and it’s surprisingly satisfying to see the numbers climb. I have the sense that the total tally and the graphs will provide continued inspiration. I’m much more wary of “0 hours” practice days and my practice feels a bit more validated as the numbers tick upward. I have not yet been tempted to practice simply for the sake of the numbers, a danger of tracking one’s time. With the end-goal so far away, I worried that the system might actually be discouraging. I’m relieved this has not been the case.

Happy practicing to you too!

Thanks to…

My sincere thanks to Matt Gallizzi of Notix Technical Solutions for helping me implement the trickier parts of this project. If you’re looking for technical support or an all-new website, Matt is a great resource!

Thank you to all the supportive taiko players who have encouraged me in this challenge!

My thanks also to all the helpful friends in the #lalugs IRC channel at irc.freenode.net. My nickname is “xuxa” there… feel free to pop in and chat!

Tags: ,

3 Responses to “Howto: Track your practice with WordPress and gnuplot”

  1. Brad says:

    How do you find time to do all this, Chris? Thanks for the howto!

  2. kris says:

    Cool! linuxtoday.org published a link to this howto. The Free Software community is so great…

  3. Matt says:

    Awesome article Kris! Please let me know if you have any other creative technical projects to tackle. =) See ya around. (PS. Kris is an awesome Taiko player, with the rest of On Ensemble. I’ve got his autograph! :D)

Leave a Reply

On Ensemble is proudly powered by WordPress
Entries (RSS) and Comments (RSS).