How we migrated our community from Spectrum to Discourse

They say that 70% of Data Science is moving data around and cleaning it. Unfortunately they’re wrong: 70% of Data Science and engineering is moving data around and cleaning it. We recently decided to move Retool’s community from Spectrum over to Discourse, and with hundreds of members and thousands of posts, there was a lot of moving to do.

You can jump ahead to the repo here.

Communities are all the rage in devtools these days (see Docker, Hugo, Atom, Rust, etc.). A lot of that is because developer products tend to be flexible: there are so many things that you can do with them that it’s impossible to support everything from scratch. Communities let users solve each other’s problems and share what they’re working on, which eases the support load and creates a more engaging product experience.

The most popular software powering these communities is called Discourse, and it’s from the founder of Stack Overflow. Most devtools “communities” are really forums where users can post about problems they’re running in to with the product or tasks they’re trying to accomplish. Other users (or staff) can jump in and reply. There’s basic markdown formatting, moderation features like badges and trust levels, as well as categories and tags for organizing posts.

Discourse is the most popular open source forum software out there, but it’s not the only one; the Retool community started out on an open source competitor called Spectrum. Spectrum is a little lighter-weight than Discourse out of the box, but over the past year it wasn’t really supporting our community’s needs as the volume of users and posts grew. In particular, we needed better search, tagging, user profiles and information, and gamification features.

The biggest challenge with any data migration is ergonomics: the schema of where you’re taking data from almost never matches that of where you’re sending data to. This project was no exception, as the Spectrum schema for storing posts was quite a trip to work with. The single biggest problem? Spectrum didn’t document anything about how data is stored or provide export utilities, so this entire project was pretty hacky.

Hierarchy and organization

In Spectrum, there are threads, messages, and channels (again, I had to figure this out by querying the underlying RethinkDB, not because Spectrum documented it):

Each message has an associated threadId that it belongs to, and each thread has a channelId that it belongs to.

Schemas

To write and store text, Spectrum uses a text editor framework built for React called DraftJS, and stores the data in a RethinkDB instance. The way that DraftJS actually stores the posts that you write in it is really difficult to work with programmatically. Here’s an example of how a thread gets stored – all examples in this post are from actual data that we used for the migration.

Each message occupies its own JSON-like object, and any formatting (bold, italics, code) is attached to parts of the object as a descriptor with metadata about where it applies to. If we wrote a sentence with bold text in the middle, DraftJS would give you the sentence, and then where the bold appears, you’d see an inlineStyles parameter:

That format only applies to inline text decoration though; for images and links, DraftJS uses a completely different convention called entityRanges that get appended to the end of the whole post’s JSON. If your sentence has a hyperlink or an image in it, it gets marked with a placeholder and ID, which get referenced by the entities at the end of the object. If we wrote a sentence with a hyperlink in it, here’s how it would show up in the database:

Both of these formatting descriptors use the same convention for identifying where the bold goes or where the images go: an offset value for where the styling starts, and a length value for how long it goes on for. Each unit is a character in the post string, so an offset of 18 means that the bold-ness starts 18 characters into the message.

This is all fine and dandy, except that Discourse isn’t built in React (it’s not even in Javascript!) and doesn’t use DraftJS. It uses Markdown.

Discourse’s schema is a bit simpler to parse: the API is clearly documented and, overall, returns useful responses with helpful information. Handling users was pretty easy, but topics and posts were where things got interesting.

Building a DraftJS → Markdown parser

Posting content through the Discourse API requires a raw payload, which corresponds to the content’s body, formatted in Markdown. We needed to translate the DraftJS data into Markdown, so we built a basic parser that iterated through all of the “blocks” in the DraftJS data structure and concatenated it into something that looks right in Discourse.

There are still some edge cases that this doesn’t account for: for example, if an entire message is a screenshot, the way that the RethinkDB stores the data doesn’t allow you to actually get the image URL, so the parser needed to just skip it.

The general workflow that we followed was to (1) export data from Spectrum, and then (2) iterate through it and post it all to Discourse via the Discourse API.

→ Users

Creating users: Users in the Spectrum RethinkDB were stored pretty intuitively, with properties like email and username. Discourse provides an endpoint for creating users, and we passed in the email, username, and name from Spectrum. For password we created a temporary one and asked users to change it when they signed in for the first time.

Post attribution: To make sure the migrated data looked smooth, we wanted all of the authors of topics and posts to be accurate: if user #1 created a topic in Spectrum, we wanted that topic to be attributed to the same user when it was moved into Discourse. The Discourse API endpoints for creating topics and posts don’t support this directly, but you can get around it via HTTP headers. Every time we created a new post or topic, we dynamically changed the HTTP headers of that request to use the username of whoever originally wrote the content.

→ Threads / Topics

Once we built the Markdown parser, all that was left to do was actually post the parsed threads / topics via the Discourse API. Discourse has a singular endpoint for both topics (“first posts”) and posts (“replies”): if you include a topic_id the backend reads it as a post (as a reply to that topic_id), and if you don’t, the backend reads it as a topic.

Because we set up our channels in Discourse differently than in Spectrum, we also needed to create a mapping layer that figured out which Spectrum channel a thread was originally part of, and which new Discourse channel that mapped to. Discourse provides a categories endpoint that lists your Discourse server’s categories (channels) and their associated IDs; we used that endpoint every time we posted a topic to find the appropriate category ID, and passed that category ID to the topics / posts endpoint to tell it where to post the topic.

→ Messages / Posts

The hardest part of the migration was handling posts (“replies”). To create a post via the Discourse API, you need to pass a topic_id that identifies which topic that post is replying to. That topic_id is created when you post the original topic and is a Discourse specific ID, so we needed to build a system for storing that topic_id when we created topics. Each message (post) in Spectrum had an associated threadId, so what we really needed was a way to map those Spectrum threadIds to the topics that they became in Discourse.

To keep things simple, we created a simple CSV file that lives in the migration directory and keeps track of which Spectrum threadIDs map to which Discourse topic_ids. Every time we created a new topic in Discourse via the API, it returns the topic_id that it generated; we stored that in the CSV in the same row as the original Spectrum threadId. Here’s what that file looked like:

The last column is a boolean indicator of whether we had already created the posts attached to that topic in Discourse. Whenever a record gets created (when we created a new topic), that value gets set to FALSE, and updates to TRUE when we hit the posts endpoint for that topic_id. The reason we needed this in the first place was rate limiting, but more on that later.

To get these posts up on Discourse, we iterated through our topics CSV topic by topic:

This wasn’t quite perfect: we marked posts_posted as TRUE after the first post on that topic was created, not after all of them. So in theory if there was an error in the middle of creating all of the posts for a topic, data might get lost.

We weren’t dealing with very much data – around 400 topics or so in total – so this entire migration should have taken minutes once the scripts were written. Unfortunately, we ran into a huge issue: Discourse heavily rate limits their API even to Admin users. This makes a lot of sense product-wise, but was really inconvenient for our use case. Some specifics:

Here's what some of the Discourse rate limit settings look like:

It took a while to get a hang of exactly how to handle this, and there was a lot of manual limiting on our end (running the migration script on subsets of data). Eventually, we added a time.sleep() statement into our scripts that ran every time the API returned a rate limit error.

Like any data migration, some things went well and some didn't.

What worked:

When we started this project we weren’t sure if we were going to be able to get Spectrum data into Discourse at all. This was a huge win.

Styling Discourse posts via the API was as easy as writing Markdown and using the simple parser that we built. After a bit of manual editing, the posts in our Discourse look great.

What didn’t work:

We weren’t able to get all Spectrum posts into Discourse perfectly – missing usernames, screenshots, and character limits made things a bit too complex. Thankfully we kept our Spectrum DB running, so we were able to manually refer back and fix a lot of these issues down the road.

Working with the Discourse API was a mixed bag. Rate limiting hurt our ability to migrate smoothly and created significant hurdles. Discourse’s Admin Web UI only allows you to remove certain limits, and some are hard coded in.

If you’re interested in the source code or you’re thinking of migrating from Spectrum to Discourse, we’ve open sourced the migration scripts we built. They’re pretty niche, but hopefully they’ll be helpful to someone one day 🙂. Pull requests welcome!

https://github.com/tryretool/spectrum-to-discourse-migration