Lessons learned from my first attempt at making a footbag database website

Back in 2006 I had just started really getting into the sport of footbag. Being that I didn't have an innate natural talent for the sport it took me quite a long time to learn the basic techniques. One thing that I frequently found myself doing was trying to look online for good technique advice whenever I was trying to learn something new. There were a few online forums where you could go and ask people about advice for tricks. The sport is uncommon in the way in which some of the best players were highly accessible and would answer questions that newer players had.

Despite this many players had trouble finding good information about techniques. The problem wasn't the quality of the information, it was finding the info. Some of the best info was buried in forum posts and was really hard to find. And you couldn't easily find anything at all if you didn't already know the nicknames of moves.

So I realized that a website with a catalogue of moves along with associated trick tips would be a good resource. Just making it easier to find the relevant information would have been a huge help to me when I was starting out and would no doubt be a huge help to new players now.

I made a website that contained a catalogue of the moves in freestyle footbag along with some tips/advice around 2006-2007. However the site never really got to be as useful as I would have hoped. Sure there there were some technical issues but even if I had the technical proficiency I had now I doubt it would have been a huge success. There were conceptual issues with the approach I took that would have got in the way of a really useful end product even if the implementation of that approach was perfect.

Now that I've had a few years to learn things and think about a better approach to making this work I've decided to try to get this project going again. Given that I'm trying to learn more about Python web frameworks it seemed like trying to do this footbag project again using Django would be a win. I bought some domain names and I decided to put the code up on GitHub here: https://github.com/shuttle1987/footbag-db

In order to actually get it right this time around it's really important to look back and reflect on what happened the first time I tried this. By learning from the mistakes of the past there's a chance of not repeating them.

Conceptual issues

The conceptual issues that the old site had mostly fell into 2 categories:

Content creation issues:

The big issue with the first site was the UI/UX was not particularly good. Adding content to the site and performing administrative tasks wasn't anywhere near easy enough. With any collaborative project like this a crucial goal has to be making it as easy as possibly for your collaborators to collaborate and add content. The site really fell short in this regard. The interface to add new information was pitiful. There was no way to keep track of revisions and undo changes. If you changed something then that change was final. If you accidentally deleted a page it was gone and you had absolutely no recourse. Some of the reasons for this were of a technical nature that I address in the technical issues section below. Regardless of the technical reasons the result was a bad UX as it made it way too easy for content to get damaged either accidentally or deliberately.

The other thing was that severely lacking were administrative/moderation helpers which made maintenance much harder that it needed to be. For example if a link was dead or other parts of a page were in some bad state there was no internal mechanism to flag it. The only thing you could do was write an email or get in contact with someone via some other means to alert them to the issue.

Usage issues:

A few things really got in the way of the site being "user friendly". Given that the site was designed with the express purpose of trying to make it as easy as possible to find relevant information about techniques you want that part of the site to be as easy to use as possible. A recurring issue was the difficulty in effectively searching the database. Unfortunately you could only search based on one string of text and some modifiers in some checkboxes. The interface was very clumsy and wouldn't return any results if your search query didn't exactly match a name already in the database. So if your local community spelled something in a slightly different way then your search just wouldn't find anything at all. Unfortunately this meant that you still needed to know the names of things you were searching for in order to get access to the information. The irony of this was not lost on me at the time but I hadn't come up with a better solution yet.

You could mitigate the issue of people not knowing the right keywords by creating additional tutorial materials to help people get started. Next time I attempt something like this I will be sure to expend a lot more effort describing the component movements from which everything else is built. When people know the basics it's much easier to bootstrap this knowledge. However this doesn't solve the problems created by requiring people to know the names ahead of time. After all footbag is just a sequence of movements you make with your body and fundamentally the only requirement you really need is to be able to describe those movements. Being able to search for information based on the movements and not necessarily the names of the movements will make the whole project a lot more useful.

I feel as though this should be reflected in the UI/UX decisions. So something that will feature much more clearly in the UI/UX will be emphasizing that any move is made up of constituent components. When viewing any move you can easily look up the information about the components via a link. The other thing that's important is going to a lot of effort to describe the base movements as these are the building blocks for everything else.

In order to make this huge pile of information easy to use really requires careful organization. One of the big realizations I had was that there is a pile of related metadata and I just wasn't using it effectively in the old site. In any sort of project like this using the metadata effectively is very powerful. Given that we are dealing with information here it should be no surprise that metadata is important. For example it gives you the ability to do fuzzy searches, create suggestions for other moves and helps you locate relevant information. Essentially I was just using the move nickname with Job's notation and storing each component in a SQL database. There was a lot of other information but it either wasn't stored in the database or wasn't easily accessible.

There's a lot of possibilities if you can let people have logins to the site. This goes far beyond content entry and maintenance of the information contained in the site. There's a lot of scope for personalized information about footbag. People can save tricks they are working on and keep track of their progress in the sport. You can offer personalized suggestions to people for technique pages that might be relevant to them. There's a bunch of other uses of this information that could potentially help people learn better.

Technical issues

While I had hacked around with some code in 2006 I didn't have a good understanding of software engineering and the code had all the hallmarks of someone who was an inexperienced programmer. In all honesty it was the classic PHP noob site, basically it was a total mess. As a software professional I would be extremely embarrassed to have such a low quality piece of software attributed to me these days.

Since 2006 I have learned a bunch of very important things about software engineering and web development. I'm proactively thinking about maintenance now and the full life cycle of the website. Code readability is vastly better, everything is documented and I'm using the PEP8 style guide for the Python code. Due to better software engineering practices the chances of anything catastrophic derailing the project is much much lower. Here's a short list of specific problems and shortcomings from last time that I aim to solve this time:

  1. I wasn't using version control software and had an ad hoc way of keeping the source in tarballs. This time I'm using Git for everything and I'm regularly keeping backups of the databases. These database backups are automated too.
  2. No thought was put into the deployment of the site. It was just running on a cheap web hosting provider and I FTPed files directly to the live site hosting directory when I needed.
    • I'm using pythons virtual environments to try to keep a stable snapshot of the packages that the environment requires.
    • Development and the live site are in different locations and I'm using a one step deployment with Git/Fabric now (no more oops I made a small mistake and brought down the live site again)
  3. I didn't automatically test the code. I didn't even know what a unit test was back then let alone how to effectively use them. Integration testing involved loading the page in the browser and randomly clicking links for a few minutes. This time I'm thinking that I might even set it up so that it automatically aborts the deployment if the certain tests fail.
  4. Last time a lot of operations that involved updating the database could only be done via writing direct SQL queries. A lot of people who are interested in contributing to this project won't know the first thing about SQL and really they shouldn't have to. This time I'm using Django and making a user interface for adding content that requires no programming skills. The entire focus this time around is on making it as easy as possible for the community to participate, if people have to be programmers to participate that's way too much of a barrier to entry.
  5. Last time I handled the SQL part really badly:
    • The trick tips was stored in the DB as this huge varchar that just contained HTML. Because this needed to fit the largest page this was tremendously inefficient with regards to space. Next time I think the idea will be to use something more like a wiki approach for the tips/advice section and have the database just point to that page.
    • The interface to update the database was completely open to SQL injection attacks. I didn't use prepared statements and I definitely didn't sanitize the input well enough. Ultimately no beginner ever sanitizes the input well enough and I being a beginner was no different. The search might have been vulnerable to attack too, chances are I didn't sanitize that input well enough because even though parts of the search used prepared statements there may have been string concatenation somewhere in those pages.
    • I didn't use transactions for the SQL queries and the PHP side just didn't handle SQL errors properly. Whenever something went wrong it would mostly just die or enter into inconsistent states.
  6. Videos had to be uploaded in whole somewhere and you had to link to them with an external link. This time around I will give users the option to add videos via video API's such as youtube. There's a few reasons I think this is a big win:
    • This allows you to timestamp videos and just pull out the relevant part from other videos. When you are trying to make a catalogue from scratch that contains video demonstrations of a large number of techniques it helps a lot to be able to look through existing video footage, even if it wasn't made specifically for this purpose, and pull out any clips that are appropriate. Many of these video API's allow you to link to a beginning and ending timestamp and only that portion of the video is actually downloaded. This saves a lot on bandwidth usage.
    • Videos were not displayed on the same page as the other information about a techniques, you had to follow a link. While it's not a total deal breaker I think the UI/UX is better if you can have all the information at once. So this time I will use a video API to display the videos on the same page if that's possible.
  7. Support for readable URLs was poor to say the least. I did think about it and I implemented something but it just didn't work particularly well. This time I'm really wanting better URLs so you can just go to /move/clipper and have confidence that it will work instead of needing to send people links like /move/move.php?id=1f17g9f01g1f.
  8. The security of the data was, by professional standards, very poor. I at least knew that you shouldn't store passwords in plaintext [1] but I just had a bad encryption algorithm. Now I know more about that and logins will be hashed+salted and not encrypted. This way the even if the database gets compromised people's passwords hopefully won't get exposed.
  9. I'm just a lot better at programming. I've learned some really hard lessons in the last few years about how to do things properly. I'm hoping that the quality of code I write this time around will be much higher. If the project ends up getting some collaborators I'm hoping that I can provide some mentoring to people and pass on some of my knowledge in this area.
  10. I wasn't using an issues tracker. For example all of the issues I have listed here would have gone into the issues tracker if that website still existed. This time I'm using the GitHub issues tracker extensively.
[1]I find it completely boggling how many people, including so called "professionals", don't get that concept. The number of sites on plaintextoffenders.com is quite scary.