I love watching Stardew Valley challenges. How fast can you unlock Ginger Island; most gold you can make without leaving the farm!

I occasionally get the urge to try it out myself. But I can’t sink that much time into the game and, to be honest, I wouldn’t have the patience for it either. But still the urge persists.

So I decided to upskill myself and delegate the work to an AI. This isn’t a ChatGPT plays Stardew experiment: no, I will be coding up my own traditional game AI system to execute my vision.

I’ve been meaning to learn more about these approaches and it turns out that writing your own in a existing game is quite illuminating.

The Basic Architecture

The first hurdle we have to overcome is interface with the actual game. We need some way of extracting information from the game and, once we’ve decided what to do with that information, some way of executing actions within the game.

Computer vision is one approach but it’s a whole ordeal in itself. Fortunately for us there’s an alternative: the Stardew Modding API SMAPI. This exposes the game engine itself which we can use to provide the relevant information.

So I code up a simple wrapper mod which reads information from the game engine and provides it as a JSON payload over TCP to our planner bot. The planner bot then responds with a limited set of options which our mod then enacts.

On the SMAPI side this is a little complicated; but it at least lets us simply things on the Julia side. We initialize with

using StardewBot

conn  = connect_bot()
state = request_info(conn)

The mod then sends back a snapshot of the game state containing

  • the time of day
  • the player’s gold
  • the player’s current location
  • the player’s stamina
  • the player’s toolbar and inventory
  • the current map including item locations and warp tiles

As we add more features this will continue to grow: for instance we’ll need to add some fishing minigame specific information when we tackle that.

On the Julia side we need to process that information and respond with one of the following commands:

  • request_info essentially waits and gets a new snapshot
  • navigate x y walk to a tile
  • navigate_to_warp location walk to the warp tile leading to a location
  • select_tool index equips a toolbar slot — equip a toolbar slot (0-based)
  • use_tool x y
  • walk_up / walk_down / walk_left / walk_right move one tick in the appropriate direction

We’ll add some actions as well as we expand.

Upon completion the mod sends the state back and the loop continues.

Challenge 1: Clearing out the Farm

Decision Trees

We’ll use a decision tree to put some intelligence here:

are we inside -> go outside or continue is it past 11pm -> go to bed or continue do we have energy -> clean closest debris or go to bed

This simple algorithm can iterate blindly and in the course of a couple days will clear the farm. You can imagine we make this more and more complex, but the cost of that complexity is going to overwhelm us. Consider us trying to add watering crops: the depth of the tree is going to be insane. We need a better way of representing this tree (graph actually, you probably want shared subtrees). One potential way is to introduce state.

do we have chores done -> do chores and set chores=done or do the rest.

This makes things simpler in some respect but now we need to consider state as well.

State Machines

It’s easier to represent this behavior as a State machine. We have states: a particular behavior that repeated and transition between states. We can represent our state easily:

We run into a problem. We have a list of redundant checks for night time. Hierarchical state machines let us simplify this by pulling state machine into a hierarchy.

We are already effectively doing this within the sleep code. Conceptually these states are bed and then sleep.

using StardewBot
include("julia/src/FarmFSM.jl")
using .FarmFSM

conn = connect_bot()
run_bot(conn, "Bot")

function get_obstacles(state)
    obstacles = Obstacle[]
    for obj in state.objects
        if obj.name == "Stone"
            push!(obstacles, Obstacle(obj.x, obj.y, Stone))
        elseif obj.name == "Weeds"
            push!(obstacles, Obstacle(obj.x, obj.y, Weeds))
        elseif obj.name == "Twig"
            push!(obstacles, Obstacle(obj.x, obj.y, Twig))
        end
    end
    for t in state.terrain
        t.type == "Tree" && push!(obstacles, Obstacle(t.x, t.y, Tree))
    end
    return obstacles
end

@enum FarmBotState InFarmHouse ClearingDebris OutOfEnergy Sleeping

const MIN_STAMINA = 5  # go to sleep when stamina falls to this level

function step(conn, state, ::Val{ClearingDebris}, skipped)
    String(state.location) != "Farm" && return state, InFarmHouse
    state.player.stamina <= MIN_STAMINA && return state, OutOfEnergy

    obstacles = filter(o -> (o.x, o.y) ∉ skipped, get_obstacles(state))
    isempty(obstacles) && return state, OutOfEnergy

    obs = argmin(o -> tile_dist(state, o.x, o.y), obstacles)
    new_state = clear_obstacle(conn, state, obs)

    obstacle_present(new_state, obs.x, obs.y) && push!(skipped, (obs.x, obs.y))
    (new_state, ClearingDebris)
end

function run_bot(conn::BotConnection, save_name::AbstractString)
    state = load_save(conn, save_name)
    fsm = InFarmHouse
    skipped = Set{Tuple{Int,Int}}()
    while true
        println("[$fsm]")
        state, fsm = step(conn, state, Val(fsm), skipped)
    end
end