Dusting the Cobwebs out of Singularity: Mistakes and Lessons

I'm pretty tired today, so I'm going to talk a little about my strategy for finishing up this Singularity update and some of the issues I see in my old code.

Homecoming

Getting back to an old codebase is usually a wince-inducing experience, but this instance was worse than usual. Singularity was always a learning process for me, as I taught myself software development over the past few years. Since putting the project down, I've had close to 9 months of software development experience. Naturally, my skills have changed drastically and my old code looks terrible now.

Since I left off mid-refactor, the code is in a bit of a half-finished state. I don't want to go all the way back to the last update, so I'm trying to fix and re-purpose what I've already got to avoid some poor decisions from the past.

Static Shock

For a long time, I regarded globally-accessible data as a very bad thing. As a result, I avoided static functions and variables like the plague in most of my software. You don't want to overuse these types of things, but it's important to remember that there's pretty much nothing in programming that should always be avoided.

In this case, the most obvious use case for statics is Singularity's settings classes. That's right, there were two of them. Depending on how loose your definition of "settings" is, you could even say that there were three (One to act as an interface to the application preferences, one to provide command-line settings, and one to provide the resolved path to the database). None of them were static, so if I needed them in a class I had to make that class hold a reference to the one(s) I needed. Some of those references even had different names!

I have since made the contents of both "main" settings classes static, and nested the command-line settings into the global settings. While I was at it, I also folded the path resolver into the command-line settings (since that's what decides the db path most of the time anyway). The result absolutely violates SOLID, but that honestly doesn't matter because it also greatly simplifies the affected code and makes it much more understandable.

Lesson learned: Trying to follow a particular coding paradigm perfectly at the expense of your actual design is, naturally, a terrible idea. At the end of the day, there's a time and place for everything.

Shameful SQL

I don't think I've ever really enjoyed working with SQL, nor do I expect this to be a particularly unpopular opinion. Within Singularity's SQLite database, there's a table containing all of the feed entries that it has saved. As it turns out, these entries have no unique keys and are subject to change. That makes swapping them out in a clean fashion damn near impossible.

So where did this terrible idea come from, anyway? The answer is GUIDs. RSS/Atom feed entries require a unique ID. However, we can't use that as our key in the database because keys could overlap between two feeds. My current plan is to combine something from the owning feed with the entry's GUID, and then hash the result. Simple and reasonable, right? Well, my OLD solution for handling updates was this:

1. Create a new temporary table and fill it with entries that match our feed.
2. Create a brand new unique index on the GUID (since there's only one feed, it should be unique now).
3. Save the items to this temporary table
4. Copy the entire thing back to its original location
5. Drop the temporary table 
6. Repeat for every individual feed (about 700 times)
7. Put the kettle on the stove, because this is going to take a while

Sometimes, (exactly 50% of the time) it crashes the entire program because a GUID managed to show up twice and database errors are treated as critical. Looking back, all I can do is cry.

Lesson Learned: Just because you just learned a couple fancy SQL tricks doesn't mean you should actually use them. Instead, consider if there's a better solution that doesn't overwork your database for no reason. Or at least use a better database, like PostgreSQL or Lucene.

Think Big! No, Smaller!

Often, it's a good idea to consider how well a solution might scale. After all, projects grow in scope over time and you don't want to leave yourself with a subpar solution down the line. On the other hand, it's also important to consider the current scale of the project you're working on.

Case in point, I made the rather poor decision to load as little as possible into memory. This seems smart, since it saves on RAM, but I don't need to do it! Seriously. Let's look at the numbers:

• I'm subscribed to ~700 active feeds
• My current Singularity database holds ~1 year of feed entries
• My current Singularity database is just over 100Mb in size

At this rate, singularity might end up using 1Gb of RAM...in a decade. Most modern web browsers regularly eat up that much right now! By the time the database gets big enough to fill my current machine's 16 gigs of RAM, I'll be several decades underground already. Unless I decide tomorrow to make Singularity into a big public online service, there's no point to optimizing for space.

More than just saving on RAM, I've repeatedly butted heads with a messy "clean up" system to auto-delete entries after a certain length of time passes. This is even more misguided, considering how cheap and plentiful hard drive space is. On top of adding extra bugs and code, this "feature" is much less useful than, say, a simple archival system that hides old entries while still letting you search for them. Unless a user was tracking a ludicrous number of feeds, I really don't see the point in trying any fancy space-saving tricks that slow things down and introduce bugs.

Lesson Learned: Base your projections on real use-cases, not crazy what-ifs and guesswork.

Conclusion

There are plenty of other problems left, but these are the big ones that I'm working on right now. Besides this I've still got a few other projects that I'm toying around with. Still, this one has my interest right now so you can probably count on more updates as development continues.

DFGame: Glade-ing Along to Victory

I planned to take a break from DFGame, but I happened to be in the mood to play around with the editor module. I ran into some issues while working on a new demo, and ended up having to do a few hours of research to get things working. Then I thought to myself: "Gee, someone else is bound to have this problem. Maybe I should write about it!"

What's New?

Sometime last year, I learned how about this cool feature in Vala, and started using the Glade UI editor with my software projects. However, it was only recently that I learned about a way to add custom widgets (such as DFGame's editor widgets) to Glade for graphical editing. Naturally, I had to give this a shot for the new and improved DFGame!

Let's discuss what went wrong, and how I solved these problems.

Problem 1: Nothing Happens?

It took quite a while for me to actually get any new UI elements to appear in Glade. There were a couple issues that caused this, primarily thanks to the documentation being rather vague about about how templates work... at least until you notice the tiny "next" button. Let's look at the example widget catalog that the documentation shows:

<glade-catalog name="foo" library="foo" depends="gtk+">
  <init-function>my_catalog_init</init-function>

  <glade-widget-classes>
    <glade-widget-class name="FooFrobnicator" generic-name="frobnicator" title="Frobnicator"/>

    ... widget classes go here
  </glade-widget-classes>

  <glade-widget-group name="foo" title="Foo">
    <glade-widget-class-ref name="FooFrobnicator"/>
    ... widget class references go here
  </glade-widget-group>

  ... widget groups go here
</glade-catalog>

(Borrowed from this page)

The difference between "name", "generic-name", and "title" isn't immediately obvious. If you look at a later page, there is an explanation:

• "name" is the name of the widget's class in code. It also seems to be used for class references in widget groups later in the file
• "generic-name" is used internally, I guess? Still not 100% sure on that one
• "title" is what the user will actually see in Glade

Unfortunately, this still wasn't enough because there remained an underlying problem to address: I write my Gtk code in Vala, but Glade only cares about GObject C. For the unacquainted, Vala acts as a wrapper language of sorts, and gets converted to GObject C for compilation. This means that there are two more things to adjust:

First of all, the class names are different. C lacks namespaces, so in order to provide them Vala adds each namespace onto every class that you create. In my case, this meant changing "Viewport" to DFGameViewport.

Next, there's a less obvious issue. Vala generates something referred to as a "get type function" which does something? I assume it either returns something that represents the object's type, or it returns an instance, but I don't have enough experience with GObject to know for certain. Glade overrides this function to create placeholders, but it apparently guesses the function name based on the "name" value that's passed in. In my case, I had to pass in the function name explicitly in order to make the catalog work. This is probably just a quirk in Vala's naming conventions, I suppose.

Once that was out of the way, Glade finally displayed my new viewport widget for DFGame and all was well, right?

Problem 2: Nope, That Doesn't Exist

As any seasoned programmer will tell you, life is never simple.
The fixes above were enough to let me create the UI for a new demo, but once the time came to actually test it out the UI wouldn't load! Instead, I was left with a cryptic message:

"(demo_editor:24355): Gtk-CRITICAL **: Error building template class 'MainWindow' for an instance of type 'MainWindow': .:3:4733 Invalid type function 'df_game_viewport_get_type'"

...???????!!?!??!??!?!!!??!!?!
The class was there. I could see it in the code. The library was definitely getting linked. So why then did it fail? Given a good couple hours, I was unable to answer this question. Luckily, the internet came through for me. All I had to do was slip in a single line of code at the start of the program

typeof(Viewport).ensure();

...and that was enough.

Takeaways

 This little experience further cements my thoughts regarding Gtk and many GNOME technologies, and I thought I'd discuss that briefly before as this post wraps up.

I like working with Gtk, Vala, and so on, but I think many of these technologies share the same problem. At the same time as GNOME works to streamline the design of its software and simplify things for normal users and new devs, there's this weird dropoff point where most docs past the beginner tutorials assume that you understand the technology stack and just need a quick reference.

The result is (presumably) "intermediate" devs like myself getting lost and confused after their gentle start suddenly drops out from under them. Thankfully, I ran into most of those issues a couple years ago, but occurrences like this one remind me that they still exist.

So I guess today's lesson is...maybe I should try writing a couple mid-level Vala/Gtk tutorials? Maybe GNOME needs more acknowledgement of their newer users and non-C APIs? Beats me.

Roundup, September 2017 - It's done!

It took longer than planned, but we're finally there:

Additional Comments

As I said in the video, DFGame v1.0 is now out! After almost 2 years of effort, I'm glad to see this project more-or-less done. There will be more, but for now it's time to wrap things up and move on.

I had an idea for another DFGame demo, as I mentioned last time. However, I wasn't very happy after messing around with it. After giving it some consideration, I decided not to go forward with it.

Next Up

Now that I'm done with this project, I'm going back to my other projects to continue them. I haven't had much time to work since I started working on DFGame again, so I'm going to go back and work on some of my software projects. In particular, Singularity has been sitting around mid-rewrite for a long time, and I'd love to have a new release out later this year (or early next year).

Update: Thinking About the Future, and Looking to the Past

Thanks to my shoulder, I've had a lot of time to think lately. I'm not very happy with the work that I've produced over the years, and I think that something fundamental needs to change. If you don't care too much about the specifics, you can scroll to the last heading for a quick summary of what I plan to do.

...Still here? Alright. Let's look back over the past 5 years.

---

In the beginning...

In late 2012, I was an eager and bright-eyed lad who was ready to try out this whole blogging thing. I was primarily inspired by the work of Shamus Young, whose blog I'd been following for a few years. To some degree, I think it shows:

"Shamus wrote some really great D&D stories, I should document the D&D campaign that I'm playing now!"
"Shamus wrote about some cool programming projects. I should do cool programming projects and write about them!"
"Shamus has a webcomic. I should do art stuff!"

He wasn't the only influence, but certainly the strongest. I also remember seeing video creators on Youtube and webcomic artists putting out work several times a week. For that reason, I created this ridiculous daily posting schedule. One thing that I failed to consider was that these individuals did nothing but make what they were posting. Indeed, for many of them this was their job, and I've heard that many such creators work well over 40 hours a week to keep such schedules. This plan of mine was doomed from the start, because I chose to try and mimic people who were quite different from me.

Alas, looking at work made by someone else and thinking that it's cool doesn't always help your own work. I couldn't simply copy Shamus's work, so the blog spent a long time being updated intermittently, with occasional apologies and failed promises.

---

Good projects, and AMAZE-ingly bad decisions

Around early to mid 2013, I started one of the projects that would last throughout this blog's life: Singularity. I started it with no plan, merely creating it out of necessity. Despite that, I could argue that it's also my most successful project: I created something just for myself, with no expectations, and ended up with something nice. Keep these italicized sentences in mind, they'll be important later.

Singularity is not all that I started then, either. During the summer, I also started AMAZE and the "Games I Play" series. The latter was about as doomed as my first schedule, and here's why:

I still remember how the concept of "Games I Play" was conceived. I had the problem of having too many untouched games in my Steam library, and no time to actually play them. Why was there no time? Well, I had to work on projects and post about them online! In other words...

1. The Problem: No time to play games, because I had to work and blog.
2. The Solution: Make games part of the work and blogging.

Seems pretty damn stupid in hindsight, huh? But hey, "Shamus Young writes about the games he plays, I should do that!" Here we have a time commitment issue, and a response that worsens the true problem in an effort to be someone else.

Now, let's talk about my darling AMAZE. Despite how much I hate the final product, AMAZE was ok as a subject. I posted screenshots, and wrote regularly about the things I was interested in! However, there's an interesting trend hidden in the AMAZE posts. As I became less interested and more "serious" to hit deadlines that I'd set, my blog posts went from this to this. Excitement and screenshots gave way to changelogs and walls of text. Once again, these two projects demonstrate a simple fact: I produce the best content when I'm excited and engaged with it, not when I feel obliged to make it. Obvious, I know. But let's move on, there's more to cover.

---

Total overkill

I'm skipping Space Douchebag because the same conclusions can be drawn from it as from AMAZE. Instead, we now fast-forward to 2014, DFEngine, and Akradion. Technically, DFEngine began with AMAZE, but this point is when I really got serious about it. As I've elaborated before, the concept of DFEngine was flawed. Trying to make a full-blown game engine because I'm too slow to iterate is like trying to build the LHC because there's a mosquito in your room: Not only is it a glorious waste of time, it also makes absolutely no sense. I still believe that DFGame will be a better bet in the long run, as it's mostly just a way to skip writing boilerplate code.

So, what about Akradion and Cloudy Climb? Besides being the first projects made after a DFGame version, they have one other thing in common. Rather than being games that I dreamed of and was excited for, they were instead an attempt to prove something. Ultimately, both games suffered dearly because of that. My line of reasoning when making these games went a little like this:

1. If I'm making a game engine, it obviously needs a physics engine. That's a thing that game engines have.
2. I need to make a game to prove that this game engine idea works
3. My game engine has physics, so I need to make a game with physics to prove that the physics engine works.

 That set of thoughts above has many problems. Let's dissect them a little:

1. My games have always been simple and 2D. Obviously I didn't need a physics engine, I had never used one before!
2. As it turns out, making a full-length game just to be sure that some tech works is a huge waste of time. I keep doing it too, it's a bad habit. This is why I forced myself to keep the demos I'm making for dfgame as small and simple as possible.
3. Neither Akradion nor Cloudy Climb are actually good choices for testing a physics engine! Seriously! Akradion is a game that relies on an object that retains constant velocity at all times, a worst-case scenario for physics! Cloudy Climb only really makes use of gravity, so a big physics engine is totally unnecessary.

In other words, a project that started as a supposed time-saving measure wandered off course in an attempt to make a fancy game engine. It then split into more projects, which were doomed from the start because I'm a bad planner. Ultimately, DFEngine was about a one and a half year-long diversion from what I cared about. I'm actually ok with most of that, since trying out things that interest me is fun. However, looking at the posts you can see the usual changelog post structure from AMAZE take over after a while.

---

Why teach?

Next, let's discuss tutorials. The roguelike tutorial series that has been in progress for years has never really reached a point where I was happy with it. So why did I start it? For that matter, why am I still writing it? Let's discuss.

I spent many years learning to program and make games. I actually started all of this back in middle school with Game Maker, only transitioning to more "professional" tools once I started college. Throughout my life (even now, in fact), I've regularly referred to guides and tutorials in order to learn various tricks and techniques. For much of that time, I felt a desire to give back with tutorials of my own once I got good at my craft.

Unlike the many of the blog-related decisions I've made, the decision to write tutorials in general had decent reasoning behind it. Furthermore, I think it'll work out eventually. So, why was the roguelike tutorial a failure for me?

At the time, I wanted to make an ASCII strategy game of sorts. When I returned to the project, I had a cool roguelike idea that I wanted to make. I don't think that's a coincidence. Rather than simply make the games that I wanted to, I found an excuse to try and work on something similar, via tutorials. Rather than just doing what I wanted, I invented a new series of posts to 'legitimize' my desire as an extension of my blog. It's no wonder then that the tutorial was never well thought-out or exciting to me. I wanted to do something else the whole time! Compare the forced and lackluster entries that I've been putting out lately to something like this post, and you'll see a world of differences.

---

Unfortunate truths

Ok, I think that's enough tiptoeing around the subject. What do all of these events point to? What's my point writing this? Well, it's really quite simple:

By forcing myself to only do "blog-worthy" things, and acting beholden to a theoretical reading audience, I'm harming both my blog's quality and my own life.

That's a conclusion that's taken about 2 months to reach. The one thread that ties many of my threads together is an attempt to emulate the content creators that I like. I envision myself as them, and imagine that I have an audience the same size that they do. Often, I make decisions based off of this. It's why I feel an urge to apologize at every delay and schedule change. It's the true reason why I just can't leave a series that I don't enjoy alone. To some degree, it also stops me from making progress. Every month, I spend:

• ~1 week preparing, recording, and editing an update video
• ~1-2 weeks writing a new segment of the roguelike tutorial
• 2-4 days writing some other post

That leaves me with only a week or two to actually work on DFGame. It also keeps me from trying out new ideas and iterating. I've had a lot of really cool project ideas since I started work, and I've had no choice but to put them off until "after DFGame".

Except, that's not true at all. I have a choice to stop and do what I want, whenever I want. The only thing that stops me is my schedule. The only reason I have a schedule is to guarantee regular content on my blog. And why do I need to guarantee that? See Above.

---

The almighty power of boredom

Let me tell you a little story:

About a year ago, I had a cool game idea. The game in question would involve some fairly heavy adult content, so it wasn't "blog-worthy" at all. At the time, I was still adjusting to life in Montreal and figuring out post-immigration stuff. I also had a few issues in my life that put me in a bad position for a while, and I wasn't sure if I could keep writing the blog due to my job.

Since I didn't want to work on my blog, I worked on the game. In just a few scant months, I wrote an entire engine from scratch, complete with a data-driven dynamic UI system. I started writing down ideas, building roadmaps for development, drawing first-draft animated sprites for characters, etc. Then, I got the greenlight to blog again and I haven't touched the project since!

Flowcharts tracking development progress.
The bottom red circle was a public demo release.
So close, yet so far...

You see, I've written in the past about how blogging with a schedule keeps me productive and how vital it is, but at the end of the day you can lock me in a room for 3 months with no blog and I'll still write yet another game engine. Development is what I do. It comes naturally, and trying to keep to a rigid schedule or only focus on one thing for a long time doesn't help.

---

If you're scrolling past the wall of text, you can stop now

Ok! That's it! No more long-winded storytelling! It's time to talk about how I'm going to fix this problem. I'm going to keep this short and sweet, you just read the whole story so you don't need much more:

• Effective immediately, I am cancelling the roguelike series. No hiatus. It's over, it was a bad idea, and I can always revisit the subject when I'm older and wiser. I'll leave the text around for now.
• Starting next year, no more schedules and no more promises. I'll write when I feel the urge, and I'll develop whatever I damn well please. There will be fewer posts. I guarantee it. But every single one of them will be carefully written and edited for maximum quality.
• When that happens, I'll finally redirect the huguesross.net domain to my site's landing page. That page and the about me page have been redesigned, and I'm working on the tutorials section. That section will hold both longer tutorial series and any informative posts that I think are useful and good enough to share.

If I change my mind about all of this, I may still keep the schedule. It's possible that I'm just in a rut, or mildly burnt-out. I'm giving myself 3 months to reconsider this decision, to ensure that it's really the right way to go. At the end of the day, I don't want to drop the blog. I think keeping a blog that shows your process of working and shares your knowledge is awesome, but I need to remember that blog posts should not be my biggest source of work.

This is a blog about programming, not time management. (except here, I guess)

Let's Make a Roguelike - Stats, Items and, Messages

Well, here's a new tutorial segment!

This one was pretty awful to write, mostly because I chose too broad of a topic. As a result, it's also quite a bit longer than most of the others (~14 pages, with a previous record of ~11). On the upside, I'm very nearly done here. Currently, I'm planning the remaining 3 segments to cover:

1. AI and combat
2. Generating artificial spaces (e.g. mazes, dungeons, buildings, etc.)
3. Various remaining loose ends, like visibility and overall game structure

I didn't quite do item systems justice here, unfortunately. I seem to have missed dropping/equipping items in particular, although these actions should be fairly trivial to implement with the knowledge from the tutorial.

Since it looks like things are mostly on schedule despite last month's delays, I'm currently pegging the completion date as December 17th, 2017. This will be one full year after I restarted the series, minus a couple days to make it land on a Sunday.

Speaking of dates, this series will be 3 years old in a couple weeks. All I can say at this point is that I'm glad it's coming to a close. It's going to be a while before my next series, I have a few ideas but I want to put a lot of consideration into any future endeavors. However, I do have a one-off informative...thing that I'll be putting up on the tutorials page not too long after this series.