Writing a SIGGRAPH paper from scratch.
40 minute read
In September 2024, my final year of college, I began working on a side project called “Terrain Diffusion.” The goal was to create a learned procedural terrain generator that could drop into almost any game engine with minimal friction. I had nothing but one RTX 3090 Ti, which I bought with money from an internship at Walmart that prior summer. As I worked on the project, I developed InfiniteDiffusion, a new algorithm for infinite generation with diffusion models [5]. I continued this work throughout my final year of college, a move to California, and a full time SWE job at Walmart. In December 2025, I released the first preprint to ArXiv, and in January, I submitted it to SIGGRAPH 2026, where the paper was later accepted.
To my delight, the paper was very well received! I converted Terrain Diffusion to a Minecraft mod, which was showcased by AsianHalfSquat to over a million viewers. The paper’s launch on Twitter went about as viral as I could hope, also nearing a million views. And shortly after, the paper was showcased by Two Minute Papers. The preprint also reached the front page of HackerNews.
This paper was the result of about a decade of self-study in machine learning, though publishing did not become a goal of mine until much later. I don’t intend for this to be a tutorial for publishing at SIGGRAPH (or any venue for that matter), since I don’t consider myself to be an expert on that topic. Instead, I’ll share my own experiences so that you can have a better idea of what to expect if you decide to take the same route I did. I’ll also share how I learned everything I know, because for that I do have a process, and it’s proven to be quite effective.
Baby Steps
I began programming when I was 13, and over the course of about a year developed a modest portfolio of mostly simple Java and C# applications. None of these were particularly impressive, but I found the entire process of learning and building to be quite fulfilling. As I became more skilled it became more appealing to specialize, and build things that I could be genuinely proud of not just for the process, but for the result. As someone who enjoys math, I was particularly interested in projects that would allow me to mix programming with mathematics, and use both skills at once. Graphics was one such area, and I spent much of the following year learning C++ and various graphics libraries.
In an alternate universe I might have gone all-in on rendering, but around this time StyleGAN came out [8], generating photorealistic images from nothing, and it blew my mind. Skimming the paper, I found much of it surprisingly intuitive, though there were many terms I had never heard of, and so the full picture remained out of reach. Regardless, I had countless ideas for how this technology could be used (including as a terrain generator!), and so I decided to learn how to use it.
I initially tried following TensorFlow tutorials, but this quickly proved futile. I could get through them by following instructions, but as soon as it was time to go out on my own and start applying the knowledge, I had no idea what to do (in fairness, this may have just been the unreasonably confusing design of TensorFlow v1). I figured I was suffering from prerequisite debt – the tutorials taught me how to put layers together, but without knowing what those layers actually represented, that knowledge was mostly useless. So I put together a study plan, starting with all of the math I thought might be useful, and ending with “A Super Harsh Guide to Machine Learning”, which seemed pretty comprehensive. That led to this final list:
- High School Math, Up to Precalculus through Khan Academy
- Note: I found Khan Academy to be so effective for learning math that I ended up repeating a lot of the courses I had already taken in school.
- “A Transition to Advanced Mathematics” by Smith et al.
- “Calculus” by Gilbert Strang
- “Introduction to Linear Algebra” by Gilbert Strang
- “All of Statistics” by Larry A. Wasserman
- “Elements of Statistical Learning” by Hastie et al. (Chapters 1-4 & 7-8; ideally all of it)
- Andrew Ng’s Coursera Course
- “Deep Learning” by Goodfellow et al.
I dedicated about 1-2 hours a day to finishing this plan, aiming for about 10 pages a day. I was able to get through Calculus and Linear Algebra in ~1.5 years, and finished the remainder of the plan after another year. For the math textbooks, I mostly didn’t bother doing the exercises in full, and instead would read them and think through the solutions in my head. I found it was usually pretty obvious when I did or didn’t understand something this way, if I was being honest with myself.
After finishing the textbooks, I spent 1-2 years reading various papers from ArXiv, mainly focusing on computer vision. At this point, I found the most effective strategy was to find the state of the art for whatever topic I was interested in, and from there recursively go through the references to find work that was either a direct prerequisite or otherwise useful. I found the related works sections to be particularly helpful, since they typically cover the prior SOTA, often categorized by technique, giving you a convenient chain to follow. Any methods that were crucial to the implementation, or that I’d seen pop up frequently in other papers, I would also go read about in more detail. Simultaneously, I’d begun learning about the code side of things, and once I understood the textbooks, learning the libraries became almost trivial – the learning curve was mostly adapting to Python as someone most familiar with statically-typed languages like Java and C++.
I have nearly no regrets with the plan I followed, and for those interested in learning the fundamentals today, I don’t think much modification is required. Even as the field of ML has grown exponentially, the fundamentals have remained largely unchanged. That being said, you can largely skip Part III of the deep learning book, as that time is probably better spent learning about the things that the book was too early for, like transformers and diffusion models. Diffusion models in particular have had quite a complicated history, but thankfully there is a great (free) textbook [10] on the topic, which I would highly recommend if it is a topic that interests you. Transformers on the other hand, at their core, have remained largely the same since Attention Is All You Need [18], except for a handful of advancements you could catch up on in a few days1.
In summary, if I were starting over, I’d follow the same plan except for these changes:
- Skip Part III of the Deep Learning Book.
- Catch up on transformers after. Mainly Attention Is All You Need [18], RoPE [17], and GPT [19] (for language) or ViT [3] (for images). Karpathy’s nanochat is a great reference as well for what a modern transformer looks like.
- If you’re interested in Diffusion Models, add Chieh-Hsin Lai’s textbook [10] to the end.
- Read more summaries instead of just singular papers.
While the fundamentals haven’t changed much, each specialty has a substantial literature of its own. What to read next depends on what you want to work on, so beyond this foundation, I’d follow the same approach: start with surveys and recent papers, then work backward through their references.
- This applies to transformers as they apply across vision and language. Tech trees get much deeper as you specialize.
An Aside on Books
I believe books are the best way to learn practically any technical topic. This is an opinion based on my experience, but there is reasoning for it that generalizes. I think if you are serious about learning fast, your first priority should be to avoid bottlenecks, or anything that prevents you from learning as fast as your brain will allow you to. This includes people. If you need to wait on someone else to teach you something, that person becomes a bottleneck. Of course, if you have access to a smart person and questions your materials don’t cover, then ask them. Just don’t wait for them.
Online courses and videos don’t suffer from this problem, but books still have many advantages. They tend to be both more detailed and comprehensive, which helps with understanding and generally learning more. With text, you can read at your own pace. With video, you must follow the pace of the lecturer, which I would argue is rarely in-sync with your own ability to process information. With video, you must watch contiguous sections of video, whereas with text you can dart between precisely the words you need to understand the topic at hand. Textbooks tend to carry far more exercises as well, and are typically easier to search through if you need to backtrack.
Also, do not dare use AI for anything more than individual questions you have while learning from another source. AI makes a lot of stupid decisions, hallucinates (yes, still), and generally doesn’t have great intuitions about many topics, especially ML in my experience. If everything you learn comes from an AI, you’ll never be able to detect when it’s being stupid. Anyways, onto the paper.
Technical Backgrounds
If you’ve ever played any game with procedural terrain, most famously Minecraft, you were looking at procedural noise. Procedural noise has been around for about 40 years now, and is used to algorithmically produce infinite textures. Infinite procedural worlds typically convert this into 3D by reading this texture as a heightmap, so white corresponds to high elevations, and black to low ones. An alternative, more complex method (not used here), is to generate 3D noise and read it as a volume, so white corresponds to filled space and black to empty space. The vast majority of terrain generators rely primarily on Perlin noise [13] (or Simplex, though they look very similar), which is stacked and scaled to produce multiple levels of detail, and then combined with hand-written post-processors to make the infinite worlds you are probably familiar with. This process of stacking noise is illustrated below.
Procedural noise is particularly useful because they typically possess many convenient properties. While not all procedural noises have the same properties, there are a handful that most terrain generators require:
- It is infinite and nonperiodic.
- It is deterministic. Provided a seed and location, procedural noise always generates the exact same output.
- It supports random access. Any point on the infinite grid can be evaluated independently and in constant time.
The first property is obviously important for generating infinite, diverse worlds. The second means you can share worlds with friends, and you don’t need to store every output indefinitely. The third is important for parallelization, since it allows you to compute many points in parallel on the GPU, but it also enables teleportation and multiplayer, since two distant locations can be computed independently.
In the 2010s, GANs and diffusion models were invented. GANs generated realistic images [4] by training a generator model to create images, and a discriminator model to tell apart real images from fake ones. The generator would learn to fool the discriminator, and after a while would generate images that looked very similar to real ones. Diffusion models came a bit later [16], demonstrating that it was possible to generate data by instead starting with a noisy image, and iteratively removing noise until there was no noise left, leaving you with a clean fake image. Diffusion models quickly became the standard for image generations, proving to be far easier to train and scale than GANs, which were notorious for being unstable, especially when scaled. The main tradeoff was that diffusion models were considerably slower, since they required dozens of steps (thousands when they were first released) to generate an image, as opposed to GANs which took only one. Later, distillation enabled diffusion models to generate images in as few as one step, though this required additional training stages and until recently required a considerable sacrifice to quality.
Both of these methods were quickly applied to terrain generation. By learning from real-world datasets, they were able to generate incredibly realistic terrain. However, they were typically too slow to use interactively, and due to factors I’ll discuss later, the quality of AI terrain generators was bottlenecked, so it didn’t improve at quite the same pace as general image generation. Additionally, learning to generate a single patch of terrain is a much simpler problem than generating an infinite world. Simply placing generated images next to each other would leave mismatched edges, and even overlapping patches could predict different elevations at the same location.
Forming A Thesis
My goal was to create a terrain generator that could replicate all the important properties of procedural noise, but otherwise use the best technology available for image generation. To solve this problem, I would need to generate each region independently, while ensuring it remained consistent with the terrain around it. This was the core challenge – quality and performance could (hopefully) borrow all the same techniques that general image generation uses. To make independent tiling work, ideally we would generate the outputs at every pixel from the same input, regardless of which patch they lie in. Then outputs are aligned by construction. With convolutions, which many models are built on, this actually is possible. For a 3x3 convolution, the usual size, each pixel in the output is produced from a 3x3 neighborhood around it. At the edge of an image, some of that context is missing. Most models solve this by filling the missing space with fake values, typically zeros. This technique is called padding, and it is what creates a mismatch at the boundary – a pixel in one patch has full context, while a pixel in another patch does not. So the solution for convolution-based models is simple – stop using padding. Without padding, pixels at the edges do not have adequate context and cannot be computed, so every convolution produces an output slightly smaller than its input. In practice, this means if you want to maintain the same output size, you must instead include that extra context in the input.
With padding
With extra input context
InfinityGAN uses precisely this technique [11] to generate infinite images with GANs. With a padding-free model, they are able to exchange a bit of extra context in the input for exactly matching overlapping patches. Unfortunately, training a diffusion model the same way is much easier said than done. One problem is that it requires every part of your network to be built on local operations like convolutions, which filters out many modern architectures (i.e. anything with attention). The bigger problem is computational cost. A padding-free model was relatively manageable for GAN generators because they build images by progressively upsampling a coarse grid, so the number of input pixels discarded by each convolution decreases deeper in the model. But diffusion UNets [15] first perform many downsampling operations while processing the input. After three downsampling steps, a pixel in a hidden layer represents eight pixels in the original input. To use a padding-free model, you’d need an input size 10x larger than in the original model, massively increasing compute requirements. It was also unclear whether a padding-free model would even work in theory, since diffusion is an iterative process (more on this later).
In order to allow myself to use all the bells and whistles of a modern diffusion model, I opted to forget about making it padding-free in the first place, and instead focused on replicating the outcome – translation-equivariance: the same inputs produce the same outputs, regardless of where they appear within a patch. This is rarely true in the general case. Most images have position-dependent features. For example, pictures of faces usually have the face in the center, not half-cut-off at the edge. I figured I could at least partially resolve this by explicitly training positional features out of the model. That is, by training the model to predict random crops of a larger image. In this case, the entire planet. Then location-dependent decisions could only ever harm the model.
If I could train a model to be approximately translation-equivariant, I could potentially get away with a simple tiling algorithm; though it could never be as straightforward as it is with GANs. GANs go straight from latent to output, whereas diffusion models generate dozens of intermediate outputs. A tile’s next denoising step depends on neighboring predictions from the previous step, which themselves require surrounding context. Working backward through many steps expands the region that must be computed. For many steps, this again leads to very expensive generations. My hope was that since I intended to distill my models into two steps, this method might work with only two blending steps, and so the harm from those expansions are at least limited.
Training the First Models
I spent the first month or so researching the SOTA in efficient diffusion models. With fast training and inference being top priorities, I was led to EDM2 [7]. It offered ~10x speedups over competitors without sacrificing quality, putting it in a league of its own. Due to my hardware constraints, it’s likely that without EDM2 this project would not have succeeded, or at least not produced a satisfying result.
While I eventually wanted to develop a high resolution, high quality model, my initial goal was to just get some terrain generator working so I could begin experimenting with approaches to infinite tiling. The first step was thus gathering data. After a bit of research, I found Google’s Earth Engine to be the best source for DEMs (heightmaps) that cover the entire earth, and I spent the first few weeks downloading this data and processing it. Early on, I wasted a lot of time on data processing. I would often spend a day downloading hundreds of gigabytes of data, just to find that there was a small error in the pipeline that made it unusable. I was often too lazy to visualize the data, which often led to me not noticing errors for far too long and losing quite a bit of progress. This was my favorite mistake to make, but I eventually learned my lesson a few times. I now always start with small subsets and visualize everything extensively. Look at your data.
Once I had a decent data pipeline, I began training the first prototypes of a terrain generator. I started with 64x64 pixel diffusion models, which were very low resolution but good enough for simple experiments. With the smallest EDM2 models, I could get a decent checkpoint after about a day of training.
Around this time, sCM [12] was released, promising to improve the performance of EDM2 by 30x with minimal quality loss. Since performance was a priority for me, I decided to implement this paper. I spent days painfully debugging my implementation. I plotted practically every step of the training process, monitoring gradients, outputs, and losses for every conceivable input. I read and reread the paper so many times that I felt I could have written it myself. Eventually, I gave up on debugging, and decided to search for errors in the paper itself. I spent a day rederiving every formula in the paper, which led me to finally discover the minor typo I’d spent the past week unknowingly chasing:
-v_t = torch.cos(t) * torch.sin(t) * sigma_data
+v_t = torch.cos(t) * torch.sin(t)
After that, everything worked, so I banged my head against a wall for a few minutes, and then finished training my first distilled model, which didn’t take long since distillation is pretty fast relative to training.
The Cheap Experiment
With a first prototype for a two-step terrain generator, it was time to make an infinite generator. Initially, I wasn’t sure exactly what this algorithm or my final model would look like. All I knew is that I needed some way to tile operations on an infinite canvas. I figured I would have to iterate on the exact algorithm quite a bit, so I spent some time working on a generalizable framework, dubbed infinite-tensor, which as the name implies would allow me to tile arbitrary operations on practically infinite tensors.
The first version of infinite-tensor took about a week, but I found that as my ideas developed and it became clear what features would and wouldn’t be helpful, the framework continued evolving, going through several paradigm shifts between its first prototype and the final working version that is publicly available today. My first attempts were overengineered in many ways, something that improved over time as I discovered what worked, what didn’t, and what added complexity where it wasn’t needed.
Nonetheless, the first version of the framework was good enough for my needs. I hooked up my first 64x64 two-step terrain model to infinite-tensor, and amazingly, it worked! I saw no blurry edges where the tiles overlapped, and in fact I couldn’t even tell where they overlapped at all. There was only one notable problem: global relief was suppressed. At every step, adjacent tiles were blended together, so if one predicted a large mountain range, and the adjacent tiles predicted flat plains, the results would average out into a small hill. For something like a large mountain range to survive, all neighboring tiles had to predict a large mountain range. Since pixels were generally produced by about 4 weakly-correlated overlapping windows, this was a quite rare event, and most major features were flattened, leaving satisfying local details, but little overall variation.
I initially tried to solve this problem analytically, seeing if there might be some way to force the model to generate the full range of elevations even with only local visibility. I eventually concluded that even if that were possible, it probably wouldn’t lead to satisfying results, as it wouldn’t resolve the fundamental problem of little correlation between patches. Instead, I opted to make future versions of the pipeline use a multi-stage approach, where I could have one model generate the broad features, and have additional stages refine them to generate high-quality local details. With that solution in mind, the infinite generation problem appeared solvable.
Anticipating Failure
My research into prior terrain generation approaches revealed a common failure mode where diffusion models and GANs produced noisy outputs, which made these terrain generators look pretty choppy when viewed interactively. I’d guessed that this was caused by an amplification of errors during denormalization. If a training set contained heightmaps spanning 10,000 meters across the set, normalization would compress those values ~2000x, and denormalization would expand errors the same amount, making any error quite obvious while walking around. I found some prior work that resolved this by normalizing inputs per-crop or removing data above a certain magnitude.
Unfortunately neither of these solutions would work for me. The former would make it impossible to identify absolute elevations, and the latter would limit diversity. Therefore, before training anything big, I decided to come up with a new input format that would permit the model to work in standard value ranges without losing information. After a bit of thinking, I found that the simplest solution would probably be to generate Laplacian pyramids [2], which retained the smoothness of typical images (unlike, for example, the fourier transform) while separating frequencies into different bands. The high-frequency bands took a far smaller range of values than the dataset as a whole, which meant the high-frequency band was automatically relieved from the exploding errors problem. The low-frequency band’s variance was unchanged, so they required another solution. My first idea was to blur the low-frequency band directly after generation, but this turned out not to work – even real encoded heightmaps wouldn’t be the same after denoising them. After a bit of thinking, I realized I could instead combine the two bands into a noisy heightmap, and blur that to produce a cleaner low-frequency band. Then I could combine this new band with the unchanged high-frequency band, preserving the high-frequency details while removing the noise. This led to the technique “Laplacian denoising.” I verified this technique on real heightmaps first, and to my surprise, even with a ton of synthetic noise, this technique almost entirely eliminated errors and nearly perfectly reproduced the original heightmap. From then on, all inputs to this model used this dual-frequency format.
Low-frequency + noise
High-frequency residual
Provisional heightmap
Denoised heightmap
High-frequency residual
Denoised low-frequency
The Expensive Experiments
By now (~3 months in) all the pieces were in place to build the full pipeline: a coarse model would generate the coarsest features, like low-resolution elevation and climate (Climate was only available at low resolution, so the latent stage conditioned on it but didn’t generate it). A latent diffusion model would condition on that data to generate the high-resolution output in a dual-frequency space, and laplacian denoising would be used as a post-processing step to ensure the results were never noisy.
While the core technical challenges had been solved, the bulk of my time went to iterating on the smaller details: optimizing hyperparameters, deciding the optimal tradeoff between quantity and quality of outputs (do we predict *all* climate variables or just a few?). I also tried some failed ideas, like using InfinityGAN for the coarse model but not the high-resolution one; with a GAN the best results I could get were roughly on par with perlin noise (possibly a skill issue), which is why I ended up seeding the model with perlin. In the end I decided to do the simplest thing that looked cool – 4 climate variables in the coarse model only, and a latent model that predicted only elevation.
The first step was training the autoencoder. Most autoencoders for latent diffusion models are GANs, which work fine, but I opted to train a diffusion decoder. This was partially because I already had all the code I needed to train one and distill it into a one or two-step model, and partially because training GANs is painful and slow. My approach here was to train an autoencoder with only an MSE loss, and then use those latents to train a diffusion model. I made some attempts at training the latents and diffusion decoder together, but this was prone to training instability and didn’t prove to be necessary, so I abandoned that approach. These days, methods like Unified Latents [6] have made joint training much easier. I also could have used an off-the-shelf VAE like Stable Diffusion [14], which has been shown to work on things like depth prediction [9], but that was ugly and I would rather die.
I spent a good bit of time iterating on the autoencoder. The biggest issue I noticed is that the model seemed to struggle with small details like shallow rivers. I traced this issue to crops that had small within-crop variance, and suspected the root cause of the problem was the noise distribution. The model was trained on noise sampled from a log-normal distribution, so very high and low noise levels were rarely sampled, but the smallest features were only learned at the lowest noise levels. I tried fixing this initially with a direct fix – instead of sampling noise from a log-normal distribution, I would sample the signal-to-noise-ratio directly, so the noise would scale with the data. This solved the problem, but it seemed a bit overengineered. In search of a simpler solution, I discovered that low-variance crops typically had low absolute elevation. This led to the much simpler (and less impressive-sounding) solution of transforming the data with a signed-sqrt transform before training, which proved to work just as well.
I had reasonable results after about a month or two of iterating on this pipeline, at which point I tested InfiniteDiffusion on the full model, and discovered that it was equally as effective even at high resolution! Laplacian Denoising worked great too. In practice, no generations had visible artifacts, except occasionally for rivers that ended randomly or flowed uphill, but even then the hydrology was far better than I expected. For a while, that was the end of the journey. I had no plans to publish, no desire to, no need to, and there was nothing that severely needed improvements. I thought it would be cool to integrate it into a game, but at this point my mission was complete, so I’d put it off, got into Valorant for a few months, finished college, accepted my full-time job offer at Walmart, and moved to California.
Deciding to (Try to) Publish
When I was an intern at Walmart I worked on an ML team and got to work on some projects that I found reasonably enjoyable, but upon returning for the full time role my team was changed, and I found myself working on Java backends that didn’t interest me nearly as much. It didn’t take long before I decided the best move was to leave, but at this point I had little proof of my abilities with ML, so I decided to leverage Terrain Diffusion as best I could to change that. Initially the goal was just to make it a solid open-source project with a nice presentation for interviews. That did get me a couple interviews that led to offers, but none that were practical over my existing role, mainly due to a $40k clawback I’d have to pay if I left before year one.
I spent a bit more time applying and sent a number of cold DMs/emails, but these mostly led nowhere. At a certain point I decided to stop relying on other people, and to instead take the most merit-based route I could think of: publishing at a double-blind conference, and open-sourcing everything while I’m at it. I identified SIGGRAPH as the most appropriate venue for a paper targeting infinite terrain generation, and with the submission deadline approaching in a few months, I dedicated myself entirely to making the absolute best submission I could pull off in that time frame.
Turning Code Into Research
By the time I decided to take the research route, I already had a basically working solution. InfiniteDiffusion wasn’t formalized yet, but I’d proven the algorithm to work in every scenario I cared about. I’d already proven I could train a model that runs efficiently and looks fantastic. Combining Laplacian Denoising and my open-source infinite-tensor framework, I had a handful of additional contributions that could contribute to a solid paper. Thus the final stretch of writing the paper mostly consisted of polishing what I had already built, making evaluations and comparisons, and finally creating compelling demos for people to play with.
My first priority was building evaluations and comparisons. Typically it’s a good idea to compare directly to previous work, but I found that to be particularly difficult in my case. Most previous work used proprietary datasets, or didn’t have open weights, or targeted very different domains, such as focusing on a small region of the world instead of global terrain. My method’s primary contribution was also offering features that no other methods could, so a direct comparison would necessarily focus on the wrong things. Therefore, I decided to avoid direct external comparisons, and instead focused on beginning with a basic, largely standard baseline, and performing ablations for significant new components. Since the model was primarily based on the EDM2 architecture with no major modifications, I figured that was a reasonable baseline to start with. This intentionally shifted the focus of the paper from proving that my architecture/design was superior to previous work, and towards proving that, with an appropriately designed architecture, InfiniteDiffusion enabled new applications without negatively affecting quality.
There were three quality-contributions I wanted to emphasize: InfiniteDiffusion, Laplacian Denoising, and the signed-sqrt transform. Since I couldn’t afford to perform a complete training run removing each contribution, I focused on training-free comparisons for each. Laplacian Denoising technically consisted of both a new input parameterization, as well as a post-hoc denoising pass after generation. Ideally I would have an ablation for both, but I figured that the new input parameterization was similar enough to training in a new latent space, which I was already doing, and thus didn’t require an explicit comparison. On the other hand, the post-hoc denoising algorithm, which is where the gains really come from, was more straightforward to ablate. It could simply be turned on/off after generation, so this was one of the first tests I ran. I expected it to result in an improvement, but I was ultimately surprised to see gains as drastic as they were – the final model’s FID was reduced from 75.2 to 12.7, an over 80% decrease. This was likely in part due to FID being more sensitive to noise than humans are, so it wasn’t as obvious from the images as you might expect, but the numbers did accurately reflect a major practical improvement, so I didn’t consider them misleading.
For InfiniteDiffusion, there were two metrics that I cared about, and which I figured others might care about: performance and quality. Since InfiniteDiffusion was the paper’s biggest contribution, it would be the focus of the results section alongside qualitative outputs. The results then become comparisons of the base model, with and without distillation, against the distilled model with InfiniteDiffusion. Since InfiniteDiffusion couldn’t perform better than the base model, the goal here was only to show that the quality degradation was minimal. Meanwhile, the performance section aimed to show that performance was practical for real-time applications. I was honestly surprised when I first measured this. In the first version of the paper, startup took 7.6 seconds, and subsequent tiles took 2.4. I liked to compare this to real–world vehicles, to see what vehicle in a game could potentially outrun the terrain generator itself (i.e. reach an unloaded chunk). I calculated that a fighter jet would take about 84 seconds to cross a tile, meaning the generator itself was about 35x faster than a fighter jet.
The signed-sqrt transform was a bit harder to make comparisons for. Unfortunately I didn’t save any sort of before and after, so when it came time to write the paper my only option for a proper comparison was to train the model again. In my head, the signed-sqrt transform felt like a meaningful contribution, as I had spent quite a lot of time discovering it. But ultimately, it didn’t seem like a particularly meaningful discovery alongside everything else. So I opted to deprioritize that contribution, and allow InfiniteDiffusion and Laplacian Denoising to do the heavy lifting for the paper, while the infinite-tensor framework and signed-sqrt transform could serve more of a supporting role.
Beyond comparisons and evaluation, I also put a good bit of effort into formalizing my contributions. The InfiniteDiffusion algorithm was initially written purely in code, and didn’t have a convenient closed-form, so a significant portion of the writing process was spent breaking apart the infinite generation code into the ~10-line algorithm that the paper now leads with. One of the most time consuming aspects of this process was simply finding a way to explain the algorithm in a way that was intuitive and easy to implement without sacrificing its extensibility and generalizability. I also took the time to formally prove all the theoretical guarantees I laid out in the paper. I think the paper makes this part look a bit harder than it really was, as intuitively these properties are very easy to sketch out, but they served a valuable role in the paper by making my claims bulletproof.
Throughout the writing process I used AI to help me draft and proofread. I am typically not a fan of AI writing, particularly in creative/personal writing. For instance in this blog I don’t let AI write anything (though I do ask it for notes). But I think for very technical writing it can be a very helpful tool for getting feedback and drafting on a shorter timeline. I still put in a lot of effort into writing the paper though, and most sections went through many revisions before submission.
Releasing the preprint & submitting to SIGGRAPH
The paper’s first draft was completed and released on ArXiv on December 9th. As submitting to ArXiv requires an endorsement from someone with at least 3 papers in an adjacent category, I reached out to a few people to see if they might be willing to endorse. I was glad to get a response from a professor at UMD who I once took a class with. Alongside the paper, I also released a working Minecraft mod that allowed you to use the models in the game, though initially this mod required a Python client running in the background to serve the terrain tiles, which prevented the mod from being used outside of some very dedicated early adopters.
After much internal debate, I opted to title the paper “Terrain Diffusion: A Diffusion-Based Successor to Perlin Noise in Real-Time Terrain Generation”. This reflected the achievement I was most proud of – a learned terrain generator that retained nearly all the properties of procedural noise. It was a pretty bold title, but as an independent researcher with no PR platform and no social media following, I needed the title to do most of the work, and I figured a bold title was better than a safe one that was lost in the noise.
The morning after the ArXiv release, I figured I would post about it on some boards where my lack of following wouldn’t be a hindrance. The first place I went was HackerNews, but I was delighted to see it was already on the front page! I also posted a short video of it working in Minecraft on r/proceduralgeneration, where it also hit the front page, as well as on my Twitter, but at that time I had like 10 followers and the post died after ~15 likes.
The initial HackerNews traffic led to the GitHub repo reaching about 100 stars. It also gave me valuable feedback regarding the paper. I was surprised to see that most of the HackerNews comments were complaints about speed, and particularly about how it wasn’t practical in real applications. Based on that feedback, I spent the remaining time before the final SIGGRAPH submission optimizing performance. The biggest change here was modifying the tiling algorithm to run everything in memory with an LRU cache, instead of saving to disk, which resulted in a ~4x performance improvement.
Once the ArXiv paper was complete and I finished making the final adjustments, submitting to SIGGRAPH didn’t take much more work. My biggest concern was accidentally violating the double-blind rules by mistakenly including a link to a public GitHub repo somewhere, or forgetting to remove my name from some file in the code. I was also worried I might mess up the formatting in a way that would get me desk-rejected for some reason. The second concern was largely me being paranoid, as SIGGRAPH does not say if formatting issues could lead to desk-rejection beyond going over the page limit, so my brain assumed the worst and I spent quite a while just verifying formatting – which I still messed up slightly, but thankfully this did not cause any problems and I was able to fix them before the camera-ready submission.
SIGGRAPH also requires that you do not promote the paper around the review period, so shortly after the ArXiv release I was very careful not to meaningfully update the GitHub or post about the paper anywhere else. A few weeks after submission I received an email indicating that the paper was not desk rejected, and would be reviewed.
After submitting the paper, I had a couple months before initial reviews came in, so I spent that time running some more experiments with InfiniteDiffusion. The original paper was pretty much entirely focused on Terrain Diffusion, and with the paper exactly hitting the 7 page limit, I didn’t have much space to add anything more. Yet there were still open questions surrounding generalization, and a few other directions you could take InfiniteDiffusion for new capabilities. The most obvious thing to test was to what extent InfiniteDiffusion would generalize to new domains. Terrain Diffusion was designed under the assumption that this algorithm, due to the number of blending steps being heavily truncated, would operate most effectively on models that operate locally – i.e. those that are heavily conditioned, or produce repeating textures. I designed Terrain Diffusion’s main architecture around this assumption, which allowed InfiniteDiffusion to extend generations infinitely with almost no loss in quality compared to the same model without tiling.
The question that remained was how well InfiniteDiffusion performed on standard models, where that assumption is not made? It turns out that InfiniteDiffusion almost perfectly matches the quality of MultiDiffusion [1] when using Stable Diffusion. With basically every prompt I could think of, the outputs were nearly identical, and the FID was a near match – 5.83 vs 5.75. There was still a gap to the base model, which Terrain Diffusion was able to largely avoid by being trained to operate locally, but it did mean that you paid little cost for using InfiniteDiffusion instead of MultiDiffusion, which is a pretty valuable result.
The Rebuttal
In March the first set of reviews came in. I was pretty terrified to see the results, since before this point there was absolutely no indication of whether anything positive or negative had happened behind the scenes. The stated release time of 12pm had passed with nothing happening, so I ended up pacing and refreshing my phone for an hour until something finally showed up. The reviews were right at the top of the page: a 3, a 1, a -1, and a -3. A perfect 0 on a -5 to 5 scale. Not bad. I felt pretty confident in the paper and reasonably confident in my ability to write a rebuttal, so this definitely seemed winnable.
The harshest review took me a while to process, since it brought up a good point I had only partially considered. The review questioned whether InfiniteDiffusion’s random access was directly comparable to Perlin noise. While the concern was justified with some math which was subtly wrong (and I spent a while deconstructing), the underlying concern was valid: evaluating a single point was comparatively inefficient with InfiniteDiffusion compared to most procedural noises. With procedural noise, efficient point-wise access is particularly important as it enables parallelization on the GPU. Thankfully, neural networks can skirt this issue entirely by being basically automatically parallelizable on GPUs. The remaining limitation is then inefficiency in the rare cases where you genuinely needed to access a tiny region. This does happen, for instance in Minecraft the game sometimes does scattered point queries to find biomes, but it’s rare enough that I didn’t think it would sink the paper. I opted to call it out myself in the rebuttal and offer to list it explicitly in the limitations section.
The other concern that most reviewers had was a lack of validation. The paper was almost entirely focused on terrain generation, and while it did that job really well, it didn’t prove that the general algorithm worked universally. The paper could have plausibly been accepted on the terrain generation advancements alone, but the paper also didn’t compare against other terrain methods. While that was largely out of necessity and I provided other valid comparisons, it did make it harder for the reviewers to recommend acceptance. Thankfully, I had spent those last two months running experiments with InfiniteDiffusion alone, so when this turned out to be the largest concern, I opted to scrap my ideas for a second paper (which weren’t really going anywhere) and instead use the results to make my rebuttal as bulletproof as possible. The remaining requests were to discuss certain papers in the related works, minor reformatting of the paper, some minor clarifications, and an overhaul of Figure 3. With these I was able to just promise to make the requested changes in the final version, so I did.
Alongside the reviews for acceptance, which primarily judges novelty and technical soundness, SIGGRAPH also has a separate track of reviews for deciding if a paper will be a conference paper or a journal paper, provided it is accepted. SIGGRAPH’s official guidance is that journal papers and conference papers have the same set of standards for novelty and advancement of the state of the art, but journal papers are expected to be more comprehensive and “extensively tried and tested.” My reviews for journal acceptance were -2 across the board, meaning every reviewer moderately recommended conference track only, if accepted. Every reviewer also provided conditions under which they would increase their rating or recommend journal acceptance, and I was able to pretty comfortably address all of these requests (though I wasn’t too concerned about the outcome here; I was satisfied just to be accepted at all).
Acceptance and Revision
A couple months passed waiting for the final decision on the paper. Eventually, an email landed in my notifications. The final list of conditionally accepted papers was released. I scrolled through the list of anonymized paper ID’s, searching for my own, and was ecstatic to see it had been accepted! I was with some of my extended family at the time so we went to dinner to celebrate. The reviews were not yet available, so I had to wait a bit longer to see the actual scores and the final conditions for acceptance.
Shortly after I received instructions on how to view the final reviews for the paper, and instructions for revising before the camera-ready submission. For the most part, the final reviews were unsurprising given the paper was accepted. The reviewers were satisfied by the rebuttal and all increased their score, except for one reviewer who seemed to have dropped out mid-process and did not write a final review at all. That reviewer also gave the strongest initial score of 3, so it didn’t affect the results much. My biggest critic, who initially gave a score of -3, revised it upward to 1. The other two reviewers both increased their score by 2.
Two things surprised me about these reviews. First, the journal scores remained unchanged at -2 across the board, and no reviewer mentioned them in their final review. Given that journal scores were not mentioned in the final reviews at all, even though all reviewers stated conditions for journal acceptance (which I believe I satisfied), I assume that the journal recommendations were fixed at the first set of reviews. Second, the required changes involved changing the paper title. This objection would not have surprised me had the original reviews brought it up, but the only mention was one review stating that singling out “Perlin noise” was off-topic, since that was just one of many techniques applied to terrain generation. I ultimately didn’t respond to this in the rebuttal, since it was the only comment in any of the reviews that was explicitly marked as “minor,” and the rebuttal space was limited to 1000 words, which got used up surprisingly quickly. Nonetheless this gave me the opportunity to change the title to focus more on InfiniteDiffusion, which with the added results made a lot more sense, and it much better reflected the core contribution of the paper.
The remaining requests were mainly the improvements I had already promised, which I spent the next couple weeks working on. This part took a bit of effort, especially the change to Fig. 3, though that was a hugely positive change and well-worth the effort – certainly something I should have taken the time to do before submission.
Reception & The Future
Acceptance to SIGGRAPH alone didn’t make much of a difference for visibility. I posted about it on HackerNews and it got like two upvotes. After that, I spent some time working on the Minecraft mod to get it in more people’s hands. By now AI coding had improved enough where I could practically port the entire thing into Java in an evening. After a little bit the mod was playable without manually downloading anything externally. I initially posted the mod to two Minecraft subreddits. On the r/minecraft sub it died, but on r/feedthebeast (a sub for Minecraft mods) it did quite well. Within about a week the mod was picked up by AsianHalfSquat, a Minecraft modding YouTuber, and reached about a million views. The Minecraft mod’s github repo became a lot more active, but not much else happened.
For a while I put off posting anything on X / Twitter. I thought that might be where an independent paper would be best received, but I initially had only about 30 followers, and when I had first posted the preprint the post died quite quickly. I put it off for a few more months while I farmed more followers (20), and then put in the effort to make an actually good “teaser” video for X (and SIGGRAPH). It turns out that the better presentation (probably?) helped quite a bit, because this time I was news. About a week after, Two Minute Papers also made a video on the paper.
After this I had a lot of people reach out, mostly on X. Many were just curious about my story or wanted advice (hence this post, sorry about the wait). There was also a lot of GitHub activity, mainly around the Minecraft mod, which many loved. About half of the X messages I received and most emails were people wanting to hire or interview me. I had the opportunity to meet a great deal of incredibly talented people working on a whole bunch of awesome things, mostly in graphics. Some of these conversations turned into offers, others didn’t, but I came away with good options and a lot to be grateful for. Ultimately, I joined a stealth startup (in fact the first company to reach out to me; probably not a coincidence) to work primarily on world models and 3D generation, especially for gaming.
I’m especially grateful for all the conversations I was able to have during this time. I met so many truly incredible researchers, and had so many fantastic conversations about the present and future of this field and how AI will shape the future of entertainment. Until this paper, I’d spent years learning and experimenting entirely on my own, without any real opportunities to discuss my interests in depth (independent research, after all, is not particularly collaborative). The opportunity to finally do so with some of the most enthusiastic and intelligent people in the field was certainly the most memorable part of this experience.
If there’s one thing from this experience I’d encourage others to try, it’s making a habit of learning before you know what you’ll use it for. Even when you’re really in the weeds, force yourself to take some time to learn, because those results compound faster than you might expect. What I learned from three years of reading ten pages a day is genuinely absurd. It seems like a long time at the starting line, but looking back, I feel like it should have taken way longer.
For those attempting ambitious projects on modest hardware (good!), I’d encourage you to spend a lot of time reading adjacent work before committing to a particular approach. For Terrain Diffusion, the discovery of EDM2 largely enabled much of the project, and allowed me to train models that were research-grade on consumer hardware. Other models probably could have achieved acceptable results, but not impressive ones. Since then, I’ve approached other problems, and there’s typically been a couple tools or techniques buried in the literature that have made my life a whole lot easier. I also found a lot of value in making my expectations for experiments explicit. What would have to be true for this approach to work? How do my requirements differ from those of existing approaches, and what did others do that I can’t? What can I do that others couldn’t? These questions often helped me spot problems before I ever ran an experiment, and pointed me towards solutions, or a targeted experiment.
For those interested in pursuing independent research, I would say to do it, but choose a project you’d find worthwhile even if it never became a paper. I started Terrain Diffusion as a passion project in college, and by the time I considered SIGGRAPH, I already had a project I was proud of. That made rejection a much smaller risk than if I’d spent nine months working towards a publication that determined whether the whole thing felt worthwhile. There’s a whole lot of luck involved in publishing, and even if you succeed, there’s no guarantee anything of use will come from it. I am fortunate to say that I am quite satisfied with how my situation has turned out, but my experience is not universal, and I can’t say how much of my experience was just luck, or maybe the circumstances of the job market, or whatever other effects being in the right place at the right time might have. As Andreas Madsen put it: “Many like to think, that success, especially earned success, is followed by more success. But success is followed by reality and reality can be cruel.”
My current goal is to make generative models more useful for longer-running, interactive experiences. I’m still unsatisfied with how diffusion models (video, 3d generators, etc) scale spatially and temporally. Generating a short clip or a single object/asset has become relatively easy, but an infinite world that someone can explore forever, is largely out of reach. I think Terrain Diffusion is a big step in this direction. It’s not perfect and could use another couple iterations, but it scales well, looks good, and is genuinely practical. I’d like similar tools to exist for every other aspect of worldbuilding too. I loved games like Skyrim and other RPGs as a kid, and always wanted to make a similar game myself, in a world of my own creation. This had always been completely infeasible as a single developer, but I don’t think that has to be the case. I think there’s a future where a single person can make a completely interactive world of that scale, and I’d like to help build that future.
At 14, I probably would have described almost exactly this as my dream job, without any expectation that it would ever be a real option. I would have been happy to work on ML in almost any capacity, and ecstatic just to be training diffusion models as part of my job (really, any model). Getting to work toward this particular goal, just as technology is beginning to make it possible (maybe), is something I’m incredibly grateful for.
References
- [1] Bar-Tal, O., Yariv, L., Lipman, Y., and Dekel, T. 2023. MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation. http://arxiv.org/abs/2302.08113.
- [2] Burt, P. and Adelson, E. 1983. The Laplacian Pyramid as a Compact Image Code. IEEE Transactions on Communications 31, 4, 532–540. https://doi.org/10.1109/TCOM.1983.1095851.
- [3] Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. http://arxiv.org/abs/2010.11929.
- [4] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., et al. 2014. Generative Adversarial Networks. http://arxiv.org/abs/1406.2661.
- [5] Goslin, A. 2026. InfiniteDiffusion: Bridging Learned Fidelity and Procedural Utility for Open-World Terrain Generation. Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, ACM, 1–10. http://arxiv.org/abs/2512.08309.
- [6] Heek, J., Hoogeboom, E., Mensink, T., and Salimans, T. 2026. Unified Latents (UL): How to train your latents. http://arxiv.org/abs/2602.17270.
- [7] Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. 2024. Analyzing and Improving the Training Dynamics of Diffusion Models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 24174–24184. http://arxiv.org/abs/2312.02696.
- [8] Karras, T., Laine, S., and Aila, T. 2019. A Style-Based Generator Architecture for Generative Adversarial Networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 4396–4405. http://arxiv.org/abs/1812.04948.
- [9] Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R.C., and Schindler, K. 2024. Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 9492–9502. http://arxiv.org/abs/2312.02145.
- [10] Lai, C.-H., Song, Y., Kim, D., Mitsufuji, Y., and Ermon, S. 2026. The Principles of Diffusion Models. http://arxiv.org/abs/2510.21890.
- [11] Lin, C.H., Lee, H.-Y., Cheng, Y.-C., Tulyakov, S., and Yang, M.-H. 2022. InfinityGAN: Towards Infinite-Pixel Image Synthesis. http://arxiv.org/abs/2104.03963.
- [12] Lu, C. and Song, Y. 2025. Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models. http://arxiv.org/abs/2410.11081.
- [13] Perlin, K. 1985. An image synthesizer. Proceedings of the 12th annual conference on Computer graphics and interactive techniques - SIGGRAPH ’85, ACM Press, 287–296. https://doi.org/10.1145/325334.325247.
- [14] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 10674–10685. http://arxiv.org/abs/2112.10752.
- [15] Ronneberger, O., Fischer, P., and Brox, T. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In: N. Navab, J. Hornegger, W.M. Wells and A.F. Frangi, eds., Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. Springer International Publishing, Cham, 234–241. http://arxiv.org/abs/1505.04597.
- [16] Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., and Ganguli, S. 2015. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. http://arxiv.org/abs/1503.03585.
- [17] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. 2024. RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomputing 568, 127063. http://arxiv.org/abs/2104.09864.
- [18] Vaswani, A., Shazeer, N., Parmar, N., et al. 2023. Attention Is All You Need. http://arxiv.org/abs/1706.03762.
- [19] Radford, A., Narasimhan, K., Salimans, T., et al. 2018. Improving Language Understanding by Generative Pre-Training. OpenAI Blog Post. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf