How Tarn Adams upgraded and optimized Dwarf Fortress for its official Steam release

Tarn Adams, co-creator and until recently sole programmer of Dwarf Fortress, might be the most interesting person in game development right now. Recently, citing a need to support themselves, he and his brother released a paid version of Dwarf Fortress on Steam (alongside the free version which is still available), but with graphics and an easier-to-use interface. In light of the release's massive success, we reached out to Adams, who took some moments from his work to answer our questions about Dwarf Fortress's past and present, discussing everything from optimization, the balance between designing for expert and novice players, and how to stay interested in a single project after so many years in development.

The Continued Popularity of Dwarf Fortress

Dwarf Fortress is older than it feels. The game first went public on August 8, 2006, over 16 years ago. That it's still being worked on after all this time is an accomplishment in itself. Before the Steam release, while it wasn't as talked about as it was in the days of Something Awful's Let's Plays, it was still the subject of multiple YouTube channels. The hype of its Steam release plus the added accessibility of its graphics and interface are introducing a new generation to the joys of leading hapless dwarves through their horrifying world. How do you keep yourself interested in the project after all this time?

It's easy to stay interested since there's so much to work on, and that's only more true now that there are so many other people involved. There are more possibilities than ever.

As for the journey to this point, well, it's twenty years to try to wrap my head around, which is a tall order! It's been mostly unexpected, and we just try to take things in stride now as they continue to happen, and try to keep walking toward wherever it is we want to go. Which is still the myth and magic release pretty much [laughs].

How has the experience working with a publisher and releasing through Steam treated you? Do you enjoy the new system? Has it added any overhead to your process?

It's been fine. Getting the builds up and stuff like that has all been super low friction, and Kitfox has been great. [We have] lots of talented people around now, and things seem to be working fine. I've got Discord channels to check and version control stuff to use now, so the process is definitely a little more involved. And we haven't tried yet to do a real feature arc like the ones we used to do, since the Steam release was so recent. So I think there are still some issues that'll need to be worked out, or more like, just the general balance of scheduling and making sure everybody is able to do things smoothly and not all at once and that kind of thing. This seems to be possible. I haven't had a chance to rest yet or ponder it at all, but I'm optimistic we'll be able to figure it out.

Looking back on the days of perilous wildlife

The days of iconic deadly bugs like murderous elephants and killer carp seem to be over. They made the game harder, but they also made for entertaining stories. Do you ever feel the urge to just slip in some absurd danger, like in the old days, just to keep players on their steel-shod toes?

We did, haha. Wild areas are harder than ever, the underground is currently too ridiculous if a bit rote and waiting for the siege stuff. And there are some new deeper hazards. There'll always be Fun around, as people call it. It's part of the game.

Remembering Flatland

I remember back when Dwarf Fortress was a strictly 2D game, with only one "Z level," and digging into the mountain meant tunneling toward the right side of the map. That was the time when Dwarf Fortress first became really popular. How did you transition away from that style of the game to the current 3D environment?

That was back in 2008-9 or something, so I'll probably be misremembering things at this point, but my recollection is that the core issue was fluids. It was just getting too muddled with bridges over magma [and] water on bridges and then you can hook them to levers and it's just not possible to have that all make sense without getting some really convoluted logic and data structures. So we made a clean break to the 3D slices.

There are a lot of problems with this, with the tightness of the game flow, that we're still dealing with to this day. You want to be able to smash all the procedural bits together in close proximity, without dead zones, so that people can move from story moment to story moment as they explore, and it's possible to do this of course in a 3D environment. But it's harder because you basically can't find points or lines in 3D space, everything has to be sheets or big blocks, etc., which is restrictive, or you need to have enough varied content that you can jam a lot of small things together, or you need to signpost your interesting stuff somehow even if it's small. And we can't do any of that without rewriting a ton of map code. Which is now called the Map Rewrite in our notes and in the community, haha.

It's the next big lump, [we're thinking of] a kind of repair or iteration on our first 3D stuff to make it actually work, and for it to be properly extensible and modular, unlike the current cavern systems and underground sites and features. It sits as a barrier to what we what to do with the myths as well since those will involve locations that also aren't supported in the current system. We'd rather not have another multi-year release cycle on our hands. We're still pondering a way around that. If version control continues to work out, we can make a branch for it of course, and push it forward while continuing to patch or even add substantial features, but the merge conflicts will be fairly epic. I expect, so we have to be careful with that and I'm not experienced there.

This one comes from Dr. Power Nap on Mastodon. He remembers that the switch from 2D to 3D was one of the biggest refactors of the code in Dwarf Fortress's history, and wonders if there are any other features lurking in the future that you've been putting off or avoiding because it'd require a similar level of rewrite?

Hahaha, well, I just mentioned the Map Rewrite which is essentially Part II of the 2D to 3D refactor. But looking elsewhere, yeah, the entity rewrite to make civilizations and cultures more interesting will also be pretty huge. The economy stuff. These have all been kicked down the road for years. Other types of map changes as well—supporting more kinds of fluids for instance, we've wanted to do that for a long time and it'll be a pretty large project. There are huge projects everywhere, haha. The graphics and UI stuff for the Steam release count too, I expect, as a recently accomplished extensive rewrite that had been put off for a long time. Half the project works in pushes like this, maybe. So many mechanics touch so many other mechanics that incremental changes just don't work after a point, and large things must be done.

Satisfying experts at the expense of newbies

Something I wonder about with long-lived, still-popular games like NetHack and Dwarf Fortress is in their designs chasing the knowledgable player. Like, NetHack's quest seems like it's built atop a long sequence of responding to players' exploits, which makes the game more complete, but also makes it harder to get into. The days of abusing Elbereth, infinite nurse dancing, abusing Constitution loss and gain to get maximum HP, and black pudding farming are in large part over, which makes it more interesting for experienced players but removes footholds that new players used to get into the game. Do you worry about this, about designing the game around the experience of advanced players?

We try not to do this, and it's pretty well-entrenched game design wisdom at this point. "Do not design for your experienced players!" etc. Some strategy games, some fighting games, some sports games, some RPGs, all genres probably, fall into this a lot, and they can be almost entirely impenetrable to new people after a while. Which isn't awful if you don't want too many new players and want something more like a club. But despite appearances, up to this point, we've always wanted a lot of people to be able to tell stories with our game, haha.

At this specific point, though, I feel like the experienced players aren't generally asking for things that'll make Dwarf Fortress worse for new players. It's not always like that. Mostly these days, I just see reasonable usability stuff, like they are all in the same boat, and experienced players can have good insight in this direction of course, even if they know a lot about the game. You just have to keep it in the back of your mind (or the front, wherever), the New Player Experience. Certainly it's the main thing we've been thinking about for the last 2-3 years.

On moving to Steam, and maintaining separate versions

These days, you're maintaining two codebases, one with graphics and the new UI, and the Dwarf Fortress of old. Who was responsible for the new UI design? Were there any challenges to splitting up the code like that, or did you manage to separate out the code from the interface fairly cleanly? Are there any worries concerning allowing other people to work on code that only you have touched before?

The new UI [was] mostly Zach and I and to some extent also Mike and Meph, as we were working through art stuff early on and menus were a constant topic. And there were also contributions from everybody else as we went. We also had an actual UX person take a look later on (later than we should have, not a mistake we're planning to make with adventure mode), which is why we have our nice building menu and several other edges smoothed off on the main screen.

There aren't really two codebases at all, haha. The switch that lets you go between Classic and graphical is just swapping out some glyphs. The grid structure underneath is the same everywhere, which leads to some restrictions, but this has made it really easy to maintain the free version.

We've brought on Putnam, the new programmer, as of this year, and there haven't been any troubles yet. Certainly there are some baked-in assumptions which rarely require some tweaks to her additions and fixes, but that's gonna be true of most projects I expect, especially ones like this that weren't designed for multiple people from the beginning. I think it's working out great so far.

Creating generative history that feels like what could occur in the game

Something I've wondered about Dwarf Fortress. Famously, before you start playing, the game spends some time generating a pre-history for your world, events that serve as the basis of the playground for your own dwarves' efforts. Something I have wondered is, once that generation ends and the player begins their own explorations and settlements, does this generation end? Can those events occur after gameplay starts? I seem to remember once that I encountered the beginning of an age during the run of a fortress I was playing. If further history generation does take place, is there any functional difference between it and what occurred before the bright red line of player involvement? Can players influence this generation?

So there was this release back in 2014 or so we called World Activation. This was our first real effort to push a lot of what was happening in world generation to play, throughout the world. There are a lot of differences and incomplete portions though. Like, they'll have athletic competitions in world gen, and you don't see those at all in play, not in any mode or offscreen or anywhere. And there's a lot more detail after worldgen; stuff like armies and invasions and all that are much more complicated after worldgen is complete, so it has taken a lot of work to make them look/function at all similar. It's just a huge ongoing process.

The players can impact all of it pretty much, and it keeps track of what they do in Legends mode, the same as worldgen. All the data structures for historical events are the same between all the modes, though some modes have more details than others.

You can play with this if you like, to some extent—the world generation is based on a seed, so you could run one world out to 110 years, say, and the same world out to 100 and play the last ten years yourself instead. You'd end up with two 110-year worlds with near identical histories, but the last ten would diverge. Legends mode lets you compare what has happened, to random people or locations or whatever. Somebody that might have lived out their life peacefully in the non-player 110-year world might have been conscripted to die in one of your saw traps in the year 105 in the player-influenced world. Or something happier.

On not using hit points

Some aspects of the legacy of Dungeons & Dragons lurk in the background of CRPGs, invisible but foundational. One of the most obvious is the ubiquity of hit points as a measure of creature health. I remember you saying before that you disliked hit points as a concept and I find there are a lot of unexamined assumptions around that whole idea. Dwarf Fortress eschews them, replacing them with a wound status for each of an entity's major body parts. The result, it seems, is creatures that degrade in a meaningful way as they are wounded, and the possibility of even a strong monster getting hit in the head by a lucky blow. Does this line up with your experience? How else does this aspect of the sim affect the game?

Yeah, that's right. And of course, once you have body parts (and tissues, and...), they can and will be used anywhere and everywhere else. For instance, our butchery system has to look at the carcass of the animal (or partial carcasses), and its current state (from old combat wounds or otherwise), and then carve it up. This isn't something you're necessarily signing on for when you add body parts and wounds! And of course, you can avoid it if you want, but you start to get inconsistencies in stories quickly, you might end up with two skulls for one creature or something, just a lot of little things go wrong if you don't respect your mechanics everywhere. We were able to use the body part system all over the place though. The poison/syndrome stuff turned out to be more interesting because of it for sure.

At the same time, hybrid systems are also interesting—even a straightforward HP + wounds system covers the degradation and lucky shots stuff you mentioned, without getting into some of the other complications. [There are] lots of things to do with damage systems (and spell systems improving on MP etc).

You just know some venture capitalist out there is taking notes

The internet has been abuzz lately over recent advances in AI-based content generation, accomplished by feeding a vast corpus of works into a machine learning algorithm and having it then generate work using that corpus as a basis. Dwarf Fortress, by contrast, produces its generated content the old-fashioned way, procedurally. Do you see a future for this new system in Dwarf Fortress, or maybe other games like it?

Caves of Qud already uses lots of generation algorithms along these lines, I think? Stuff that doesn't have as-controlled outputs, like waveform collapse or Markov chain books. And probably various other ones. There's certainly a lot of room for it! For me, I feel like I always need to be able to extend the things I'm dealing with, and going over to black box systems like this makes that harder to do, so I don't use them. Though you could consider our map generation a system like this, somewhat, so it really just depends on the specifics of how you want to use and understand what you are making, and what you want to be able to do with it later.

Simulationism vs Gamism

Dwarf Fortress has always straddled the line between being a game and being a sim. Which side do you think matters more to Dwarf Fortress? Do you worry about being fair to the player or creating inescapable situations?

I don't think I have to choose here—an emergent narrative engine requires both. The player needs to be engaged and interested and investigating, and the simulation needs to be giving back in consistent yet surprising ways.

Fairness is something more maybe like good exposition in this case. A player should have somewhat reasonable expectations about the effects of their actions/inaction and etc. And we do have these things to deal with—for instance, in the myth release, if we had the game decide a particular world was going to be wiped out by a meteor in the year 250, we have to make that interesting, rather than just ending the whole world suddenly (tastes can vary here.)

Similarly, if we have a sim that has random earthquakes and tsunamis, say, it also feels a bit much to completely destroy a seaside fortress at random, just because the dice weren't favorable. At the same time, that kind of feels like maybe it should be up to the player—certain levels of disaster are great, it's all a matter of frequency and prevention and all that. I'm sure some players would be fine just getting randomly wiped after 25 years of play, as long as they can look at the legends and see what happened, and then continue building on the world. But my instinct for default settings would be to have situations like this only arise through hubris or folly, say, to prevent too much player frustration. I don't think there's one answer, even for a single person, which is why settings are great.

World generation

Dwarf Fortress's world generation is renowned for the interesting situations it produces. How does Dwarf Fortress take the general abstract plans and events of worldgen and actualize the into gameplay events that the player can interact with, even affect?

There are a few different levels where this is going on. At its most basic, worldgen produces a lot of situations as sites spread out and war and diplomacy happen, and the player is faced with the raw outcome of this, which just continues along, sometimes with the mechanics still acting (perhaps differently), sometimes just with the situation in general without the forces still at play (because they haven't been implemented.)

Then there are all the historical figures. These include dwarves who might come as migrants, as well as goblins or monsters who might attack, and a host of visitors and others. Worldgen establishes the cast and their relationships in an ongoing fashion—at any point where the player might stop world generation, the existing historical figures are there to fill their roles, with the history sitting there to make them feel realistic in whatever sense.

So taking the civilizations and sites, then down to the historical characters, there's a lot to work with. Specific events lead to specific existing diplomatic relationships and tensions, though this isn't surfaced very well currently, and sometimes it only partially matters. Very roughly speaking, you can kind of think of it as having a zero-player strategy game with only AI opponents operating, and then jumping in and adding a player-controlled side in the middle of the game, but it's an imperfect analogy. But thinking about it like this lets you see at least how you could take a world generation process and then have a player continue history.

And to the question, with the ongoing villain's work, we now also have actual plans that survive worldgen and continue onward for the first time. These have detailed data structures which are the same in all modes, so plots can continue to unfold, and so the player will be able to create their own plots which can survive after their control (for instance when they retire a fort or adventurer).

You've generated a lot of worlds. One thing I wonder about is, how to balance between the different civilizations in a world. Do you try to do something about, say, necromancers who just become stronger over time, and inevitably take over the whole world? What means do you use (if any) to keep worlds stocked with an interesting amount of conflict?

We try not to put any fingers on the scale, since those fingers will just be chopped off or broken or become too fat when mechanics change. It's best if things work because of the underlying mechanics, obviously for the simulation side, but also for the game, if you want the player to feel like their actions have ongoing and proper-feeling consequences. Where mechanics are incomplete, as with sieges in fort mode, we still do just ramp things up in a "gamey" way, but we're working to remove that behavior as much as we can (but this requires adding a whole set of new features where the player has more control over armies in the world, etc).

For world generation, taking your problem of necromancers, yeah, we added alliances, for instance, to try to slow them down. We're planning on adding even more mechanics, like orders hunting them even during peacetime, etc., to get them even more under control. Ultimately, if their powers themselves have to be nerfed, that's also fine, but we'd like to keep them strong and just have the humans/dwarves/elves/etc. respond more robustly. Once a realistic-feeling, robust reply is in, then we can tune the necromancer powers a bit to make the world feel balanced (or just have settings). It's an ongoing process. Right now, it's still just too far over to the necromancer's side.

Generally, for conflict, we just supported some typical reasons people have disputes, and it took care of itself. The underlying systems, especially the economic stuff, need to be more robust before it feels at all right though.

The relative power of experience and equipment

Dwarves improve in their actions through both experience, that is, doing something and getting better at it through repetition, and equipment, or rather, the effectiveness of the tools that they use. Experience is like the dwarf's personal history being brought to bear; equipment is like the power of the fortress itself aimed at the enemy. Do you think this is a useful distinction? Which is more important to a fighter in your world?

They're both important. I'm not sure where the balance is right now—certainly some weapon/armor mismatches are very hard to overcome, regardless of skill, but legendary dodgers can also just keep dodging for a long time, haha. And of course, all the decisions made here map over partially to adventure mode as well, where all the materials work the same way and the player also has to become better at their skills by using them.

Ultimately you want to reward the player for both utilizing their material resources well, and for taking care of their soldiers, giving them time to train, and keeping them happy and healthy. And you want to reward an adventurer both for using their skills and training and also for making good finds/purchases. And we wanted the extremes should feel extreme—both our adamantine weapons and legendary skills are pretty far out there.

Generally, in terms of stories and world-building, we wanted people to be able to be heroic, which means that skill can overcome a bit more equipment and numbers than it might be able to in reality, though maybe not so much as in a game like Dynasty Warriors where one hero can easily dispatch 300 people almost at once. Once we have more settings, though, and a bit more flexibility with the myth/magic stuff, I imagine that scenario will also be possible. And we should allow for 'grittier' worlds at that time where individual skill doesn't count quite as much—once you have enough mechanics, you can start to see how this might just be mapped to a series of parameters for the player, but we'll see how it actually plays out when we get there.

Miscellaneous matters

A fortress can get quite complex as it grows, encompassing a lot of space and territory to simulate. What kinds of tricks does the simulation use to keep things moving at a playable rate? Are there any optimization tricks you might pass along?

There's all kinds of stuff in there, yeah, though I'm self-taught so it's all crap I've come up with or picked up at random.

Recently, for instance, somebody told me about a project that kept the closed node list for pathfinding as a data object, storing each of the nodes. This is how the algorithms are described a lot of times in examples, but for a large map that's the kind of thing that just gets untenable, since it needs to be maintained, and that can get unwieldy or slow depending on how you do it, and it starts to hog resources. If you can just set a flag on a tile or otherwise store whatever values you need in existing structures, it might be faster and leaner. Might! You have to check, since a flag per tile is also expensive in its way.

Generally, all sorts of things can be skipped, delayed, or otherwise scheduled. You have to look at where your actual slow points are, but you can avoid them in advance by taking a little care sometimes. People sometimes admonish against "premature optimization" but it's more complicated than that in reality, of course. You run into trouble very quickly if you just throw in some nested loops that are creating/destroying things every frame. Think about how often you actually need to do something and try to do it only that much.

You can also keep some buffers 'dirty', and only clear them out every so often—we do this with some pathing stuff for instance, but elsewhere as well. The base value might be 0 the first time, as an algorithm describes, but 500 the next, all the way until it's about to blow out a 64-bit integer, so you essentially never need to clear your existing buffers, but this means knowing that the values it sets are between the base and the base+499. This assumes the clearing process is both necessary and expensive of course (and that the buffer needs to exist at all, etc). In any case, there are various little tricks along these lines. I learned this one back in the '90s when I was doing depth buffers and manually drawing little UV texture triangles haha, but it has lots of applications in sims.

I've recently found it quite useful to add flags to objects like, say, an item flag called "might contain items"—the vast majority of items don't contain other items, and even the ones with this flag might not contain anything, but it lets me cut out a lot of checks and calls that might otherwise trigger and it has the benefit that I don't have to verify the state perfectly (which can be very expensive). If anything is put in an item, the flag gets set, but the flag doesn't have to be (expensively) checked when items are removed. The flag can be updated if some subsequent function finds the item doesn't in fact have anything inside, but it doesn't have to. Generally, this type of flag is resilient against a future programmer's lack of attention, which is very handy, since that's often me, haha. It does cost a single bit on all those items that never have things inside, but for the savings in this example, it's well worth it, since the game has so many contents-related checks. You can of course organize your items so that certain never-container items (e.g. boulders) are excluded from checks in advance, and we do a lot of this too, but the flag still helps a lot, and we have others like it.

One place where I'm not good with optimization is modern CPU architecture. They prefer things to be done in ways that didn't matter as much when I was starting out. So I'd really recommend people look into that as well. How to structure data to minimize cache misses, how to structure loops and conditions within them that might cause unpredictable branching, etc.

The dragons in Dwarf Fortress have formidable flame attacks, but aren't quite as fearsome, compared to other monsters, as one might think. Is it possible for a monster to exist that absolutely could not be defeated by any means?

Haha, the giant sponge accomplished this for a while if I remember, since it had no meaningful organs and nothing about it was modeled.

Dwarf Fortress makes use of that old, but reliable, system for representing a game world, the rectangular grid. It's simple, efficient, and easy to map to a text-based display on a one-for-one basis of tile to character. Are there any drawbacks, in your experience, to using it?

Diagonal movement is always gonna be a little weird. Hexes win hands down, but of course, hexes lose most everything else you mentioned, haha. And of course, compared to representations with finer resolution, any grid has a lot of drawbacks, in terms of pathing, how many creatures can surround another creature, visual representation and underlying complexity of the map, etc. But the grid is computer-friendly and easy to work with.

This isn’t really a question, but my favorite of Dwarf Fortress's many dangers are the dwarven ones. With forgotten beasts, bronze colossuses, flying dragons, invading armies, necromancers and more out there, it doesn't lack for dangers from without, but the dangers from within all feel like you're wrestling with dwarvishness itself: tantrum spirals, fell moods, demanding nobles.

The more internal dynamics we add the better, since they'll come up in pretty much every story and are easy to relate to. As long as they relate to other mechanics and can be seen by the player, modeling character behavior and making it more interesting has been great. And we were fortunate that dwarves in fantasy are well-suited to this kind of play, where even a badly-tuned simulation can be accepted, at least for a time, since people expect dwarves to react strongly to events. That's probably true of elves and humans as well, really, in this kind of setting, to some extent. Tolkien's elves certainly weren't circumspect all the time. But dwarves may have the edge.

The future

How far along do you think you are in Dwarf Fortress's development? Do you worry about feature creep? What is next up on your list of features to implement?

We're just going to keep working. I think the feature creep horse has escaped the barn and does not even remember the barn. It is happy out in the wilds, with the rest of the feature creep herd. Still, it's important to keep the game playable, and that's obviously something we've been focused on for the last few years. So in that sense, we're trying to keep it under control.

We just put the arena back in, so the next step is to do some quality-of-life stuff and then get adventure mode updated to the next standard, while also getting the Linux/Mac ports done and continuing to bring back classic functionality like keyboard support. Then it's on to (finishing) villains, army/siege improvements, and the myth/magic work.