Mastodon sync to Memos

Latest script: https://gist.github.com/eallion/bf8861eb3292c2351c1067fba3198c26

Update: Added an example for the Baota panel.

TL;DR#

Jump directly to this page script content to view the script code.

Introduction#

I don't know if I'm lucky or unlucky, but just as I was preparing to make Memos my main tool in my workflow, I encountered the v0.19.0 version update, which brought a series of major issues. Besides the robustness of the new version of Memos being questioned, even the server hosting it was affected; I couldn't run it on a physical machine with 64G of memory. As netizens said, Memos seems more like a practice project. I decisively abandoned it. What’s wrong with tools like Google Keep and Obsidian? Not confining all tasks to one tool is indeed a bit troublesome, but All in one basically equals All in boom.
Now I see Memos as a way to back up my Mastodon (one of the methods).

I have always liked the proactive push solution of Webhook, which is simpler, more environmentally friendly, and more immediate than passive pull solutions like RSS and Cron jobs. Mainly, the feeling of having control is very satisfying.

Next, I will introduce how to use Webhook to sync tweets from Mastodon to Memos. I am using a Shell Script, which is a very simple script that only performs some common-sense logical judgments and may not be perfect. It can also be implemented using Node.js, Python, etc.

Tested Versions#

Memos: v0.18.2
Mastodon: v4.2.8

Mastodon requires its own instance or an account with admin privileges to create a Webhook to use this method.

Installation Tools#

Please install the tools on the server. If there are errors, please install other corresponding tools based on the error logs.

sudo apt install jq
sudo apt install lynx

Webhook Tools#

https://github.com/jkjoy/docker-webhook
https://github.com/adnanh/webhook
Baota panel - Software Store - Search for Webhook to install
1Panel - Unknown

Go to https://{INSTANCE}/admin/webhooks on Mastodon to create a Webhook.
You can select only the status:created event; replies and reblogs also count as this event.
Fill in the destination URL with the link of your deployed Webhook, such as: https://webhook.example.com/hooks/mastodon-sync-to-memos
For Baota, it is: https://webhook.mybtserver.com:8888/hook?access_key=ACCESSKEY
It is recommended to bind a domain name to the Mastodon Webhook destination URL; otherwise, Sidekiq may not be able to handle it.

Script Content#

Save the script content below to a .sh file on the server, such as ~/mastodon_sync_to_memos.sh in the current user's Home directory (~), and configure the following content. Please make sure to replace:

MEMOS_HOST=""
MEMOS_ACCESS_TOKEN=""
MEMOS_VISIBILITY=""
MASTODON_INSTANCE=""
MASTODON_ID=""
SKIP_MASTODON_REPLY=
SKIP_MASTODON_REBLOG=
HOME_DIR=~
FILE_PATH=$HOME_DIR/.mastodon_memos_id.json

Find Mastodon ID: https://INSTANCE/api/v1/accounts/lookup?acct=USERNAME

#!/bin/bash

# Tested versions: 
# Memos: v0.18.2 
# Mastodon: v4.2.8

# ======================================================
# Configuration Start

# Memos Host
MEMOS_HOST=""

# Memos Access Token
MEMOS_ACCESS_TOKEN=""

# Visibility for publishing to Memos ('PUBLIC', 'PROTECTED', 'PRIVATE') choose one
MEMOS_VISIBILITY=PUBLIC

# Mastodon Instance
MASTODON_INSTANCE=""

# Mastodon ID, Find ID: https://INSTANCE/api/v1/accounts/lookup?acct=USERNAME
MASTODON_ID=""

# Skip replies and reblogs
SKIP_MASTODON_REPLY=true
SKIP_MASTODON_REBLOG=true

# Get the current user's Home directory path and the file to save IDs, keep default, no need to change
HOME_DIR=~
FILE_PATH=$HOME_DIR/.mastodon_memos_id.json

# Configuration End
# ======================================================

# The following content does not need to be changed

# Check if the ID file exists
if [ ! -f "$FILE_PATH" ]; then
  # If the file does not exist, create the file and write JSON data
  echo '
{
  "latest_memos_id": "0",
  "latest_mastodon_id": "0",
  "bind": []
}
' > "$FILE_PATH"
  echo "Data file created: $FILE_PATH"
else
  # If the file exists, skip and proceed to the next steps
  echo "Local data exist, skipping..."
fi

# Concatenate API and Token
if [[ "$MEMOS_HOST" != */ ]]; then
  MEMOS_HOST="$MEMOS_HOST/"
fi
MEMOS_API_HOST="${MEMOS_HOST}api/v1/memo"
AUTHORIZATION="Bearer ${MEMOS_ACCESS_TOKEN}"

# Get the latest Memos ID
MEMOS_URL="${MEMOS_API_HOST}?creatorId=101&rowStatus=NORMAL&limit=1"
LATEST_MEMOS_ID=$(curl --connect-timeout 60 -s $MEMOS_URL | jq -r '.[0].id')

# Mastodon API
if [[ "$MASTODON_INSTANCE" != */ ]]; then
  MASTODON_INSTANCE="$MASTODON_INSTANCE/"
fi
CONTENT_URL="${MASTODON_INSTANCE}api/v1/accounts/${MASTODON_ID}/statuses?limit=1&exclude_replies=${SKIP_MASTODON_REPLY}&exclude_reblogs=${SKIP_MASTODON_REBLOG}"

# Latest Status ID from Mastodon
LATEST_MASTODON_ID=$(curl --connect-timeout 60 -s $CONTENT_URL | jq -r '.[0].id')

# Define LOCAL_MEMOS_ID variable
LOCAL_MEMOS_ID=$(cat "$FILE_PATH" | jq -r '.latest_memos_id')
LOCAL_MASTODON_ID=$(cat "$FILE_PATH" | jq -r '.latest_mastodon_id')

# When the Webhook is triggered, check if the latest Mastodon ID is the temporary ID to prevent duplicate sync
if [ "$LATEST_MASTODON_ID" == "$LOCAL_MASTODON_ID" ]; then
  echo "Mastodon no updated, skipping..."
  echo "Skipped: $(TZ=UTC-8 date +"%Y-%m-%d"" ""%T")"
  echo "============================="
  exit 0
fi

CONTENT=$(curl --connect-timeout 60 -s $CONTENT_URL | jq -r '.[0]')

MEDIA=$(echo $CONTENT | jq -r '.media_attachments')
# Check the content of Media
if [ "$MEDIA" != "null" ]; then
  MEDIAS=$(echo $CONTENT | jq -r '.media_attachments[] | select(.type=="image") | .url')
  # Concatenate images 
  images=""
  for url in $MEDIAS; do 
    images="$images![image]($url)\n"
  done
  TEXT=$(echo "$CONTENT" | jq -r '.content' | lynx -dump -stdin -nonumbers -nolist | tr -d '\n' | sed '/^$/N;s/\n\n/\n/g' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | sed -E 's/ {2,}/ /g')
  TEXT="$TEXT\n$images"
else
  # Normal content
  TEXT=$(echo "$CONTENT" | jq -r '.content' | lynx -dump -stdin -nonumbers -nolist | tr -d '\n' | sed '/^$/N;s/\n\n/\n/g' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | sed -E 's/ {2,}/ /g')
fi

# Check if content is empty
if [ -z "$TEXT" ] || [ "$TEXT" == "\\n" ]; then
  echo "Content is empty, skipping..."
  echo "Skipped: $(TZ=UTC-8 date +"%Y-%m-%d"" ""%T")"
  echo "============================="
  exit 0
fi

# Escape double quotes
TEXT=$(echo "$TEXT" | sed 's/"/\\"/g')

# When the Webhook is triggered, check if the latest Memos ID is the temporary ID
# After Memos has unilaterally updated, verify the binding relationship between Mastodon and Memos IDs (Todo)
#if [ "$LATEST_MEMOS_ID" == "$LOCAL_MEMOS_ID" ]; then
#  echo "Memos no updated, skipping..."
#  echo "Skipped: $(TZ=UTC-8 date +"%Y-%m-%d"" ""%T")"
#  echo "============================="
# exit 0
#fi

# Compare the MD5 values of the content from Mastodon and Memos (not necessarily precise)
# Later try to introduce GPT to compare content
CONTENT_MEMOS=$(curl --connect-timeout 60 -s $MEMOS_URL | jq '.[0].content')
CONTENT_MASTODON=$TEXT

# Get the latest Memos MD5
LATEST_MEMOS_MD5=$(echo $CONTENT_MEMOS | tr -d '"' | md5sum | cut -d' ' -f1)
# Get the latest Mastodon MD5
LATEST_TEXT_MD5=$(echo $TEXT | tr -d '"' | md5sum | cut -d' ' -f1)

# Determine if the content is duplicate based on MD5
if [ "$LATEST_TEXT_MD5" == "$LATEST_MEMOS_MD5" ]; then
  echo "Content is duplicate, skipping..."
  echo "Skipped: $(TZ=UTC-8 date +"%Y-%m-%d"" ""%T")"
  echo "============================="
  exit 0
fi

# Replace NeoDB rating Emoji
TEXT=$(echo "$TEXT" | sed "s/:star_empty:/🌑/g; s/:star_half:/🌗/g; s/:star_solid:/🌕/g")

# Remove the trailing empty line
TEXT=$(echo "$TEXT" | sed 's/\\n$//')

# Publish to Memos and get the returned JSON data
RESPONSE=$(curl -s -X POST \
  -H "Accept: application/json" \
  -H "Authorization: $AUTHORIZATION" \
  -d "{ \"content\": \"$TEXT\", \"visibility\": \"$MEMOS_VISIBILITY\"}" \
  $MEMOS_API_HOST)

# Extract the Memos ID from the returned JSON data
NEW_MEMOS_ID=$(echo "$RESPONSE" | jq -r '.id')

# Update the latest_memos_id value in the JSON file
jq ".latest_memos_id = \"$NEW_MEMOS_ID\"" "$FILE_PATH" > "${FILE_PATH}.tmp" && mv "${FILE_PATH}.tmp" "$FILE_PATH"

# Update the latest_mastodon_id value in the JSON file
jq ".latest_mastodon_id = \"$LATEST_MASTODON_ID\"" "$FILE_PATH" > "${FILE_PATH}.tmp" && mv "${FILE_PATH}.tmp" "$FILE_PATH"

# Update the binding relationship between Mastodon and Memos IDs, ensuring that the "bind" array retains unique keys, with only unique values
jq ".bind += [{\"$LATEST_MASTODON_ID\": \"$NEW_MEMOS_ID\"}] | .bind = (.bind | unique)" "$FILE_PATH" > "${FILE_PATH}.tmp" && mv "${FILE_PATH}.tmp" "$FILE_PATH"

echo "Sync Mastodon to Memos Successful!"
echo "Done: $(TZ=UTC-8 date +"%Y-%m-%d"" ""%T")"
echo "============================="

Baota Panel#

If using the Webhook plugin in the Baota panel, you can directly copy the above script content into the script of the Webhook plugin. There is no need to manually create a .sh file on the server.

JSON Data File Content#

When the script is run for the first time, it will create a file ~/.mastodon_memos_id.json in the current user's Home directory ~ and initialize it. This file will subsequently record the binding relationship between Mastodon ID and Memos ID. If you do not want to create it in the Home directory, you need to modify the HOME_DIR= and FILE_PATH= parameters.

{
  "latest_memos_id": "",
  "latest_mastodon_id": "",
  "bind": []
}

After data is generated in the production environment, an example:

{
  "latest_memos_id": "6231",
  "latest_mastodon_id": "112061852482921394",
  "bind": [
    {
      "112059053750743781": "6230"
    },
    {
      "112061852482921394": "6231"
    }
  ]
}