Batch document conversion with Bash and jq

When you need to convert thousands of office documents to Markdown — a directory full of PDFs, a decade of .docx files, or a shared drive with every format under the sun — the fastest tool is often not a SDK at all. It's the command line.

This post shows you how to batch-convert entire directories of documents using curl, jq, and GNU parallel — the Unix pipeline that scales from ten files to ten thousand without writing a single import statement.

Why the command line?

SDKs are great for application code, but for bulk conversions they add ceremony without benefit. Installing packages, managing dependencies, writing loops and error handling — when what you really want is:

find ./docs -name '*.pdf' -o -name '*.docx' | \
  parallel ./convert.sh {} {.}.md

The shell handles batching, retries, and error logging. jq parses API responses. GNU parallel manages concurrency. No compilation, no virtualenv, no node_modules. Just pipes and processes — the most battle-tested batch processing system ever written.

Prerequisites

You need three tools, all probably already installed:

Tool	Install check	Purpose
curl	curl --version	HTTP client — send files to AnyMD's API
jq	jq --version	Parse JSON responses (error messages, headers)
parallel	parallel --version	Run conversions concurrently, one per CPU core

If parallel isn't installed (it's not always on macOS):

# macOS
brew install parallel

# Ubuntu / Debian
sudo apt install parallel

# RHEL / Fedora / CentOS
sudo dnf install parallel

Single-shot conversion with curl

Here's the most basic curl invocation — a single file, one-shot:

#!/usr/bin/env bash
# convert.sh — convert one file to Markdown via AnyMD
set -euo pipefail

API_KEY="${ANYMD_API_KEY:?Set ANYMD_API_KEY}"
FILE="$1"
OUT="${2:-${1%.*}.md}"

curl -s -X POST https://anymd.net/api/convert \
  -H "Authorization: Bearer $API_KEY" \
  -F "file=@$FILE" \
  -o "$OUT"

echo "→ $FILE → $OUT"

Usage:

export ANYMD_API_KEY="sk-..."
./convert.sh report.pdf
./convert.sh deck.pptx slides.md

AnyMD auto-detects the format from the filename. The response body is clean GitHub-Flavored Markdown written straight to the output file.

Adding error handling with jq

The API returns structured error responses. Here's an improved version that checks for failures:

#!/usr/bin/env bash
set -euo pipefail

API_KEY="${ANYMD_API_KEY:?}"
FILE="$1"
OUT="${2:-${1%.*}.md}"

RESPONSE=$(curl -s -w "%{http_code}" -X POST https://anymd.net/api/convert \
  -H "Authorization: Bearer $API_KEY" \
  -F "file=@$FILE" \
  -o "$OUT.tmp" 2>&1)

HTTP_CODE="${RESPONSE: -3}"
BODY=$(cat "$OUT.tmp")

if [ "$HTTP_CODE" != "200" ]; then
  ERROR=$(echo "$BODY" | jq -r '.error // "unknown error"')
  echo "FAIL $FILE — HTTP $HTTP_CODE: $ERROR" >&2
  rm -f "$OUT.tmp"
  exit 1
fi

# Extract page count from response headers via a second request
# (In practice, use -D to capture headers — see below)
mv "$OUT.tmp" "$OUT"
echo "OK   $FILE → $OUT ($HTTP_CODE)"

But for production batch jobs, you want the response headers too — especially X-Pages for billing and the actual HTTP status code. Use curl's -D flag to dump headers to a file:

#!/usr/bin/env bash
# batch-convert.sh — verbose conversion with header capture
set -euo pipefail

API_KEY="${ANYMD_API_KEY:?}"
FILE="$1"
OUT="${2:-${1%.*}.md}"
HEADERS="${OUT}.headers"

curl -s -D "$HEADERS" -X POST https://anymd.net/api/convert \
  -H "Authorization: Bearer $API_KEY" \
  -F "file=@$FILE" \
  -o "$OUT"

HTTP_CODE=$(head -1 "$HEADERS" | awk '{print $2}')
PAGES=$(grep -i '^X-Pages:' "$HEADERS" | awk '{print $2}' | tr -d '\r')
DURATION=$(grep -i '^X-Duration:' "$HEADERS" | awk '{print $2}' | tr -d '\r')

if [ "$HTTP_CODE" = "200" ]; then
  echo "OK   $FILE → $OUT (${PAGES:-?} pages, ${DURATION:-?} ms)"
else
  ERROR=$(jq -r '.error // "unknown"' "$OUT" 2>/dev/null || echo "see $OUT")
  echo "FAIL $FILE — HTTP $HTTP_CODE: $ERROR" >&2
fi

rm -f "$HEADERS"

Batch processing with GNU parallel

GNU parallel is the secret weapon. One invocation converts every supported file in a directory tree, using all CPU cores:

# Convert all supported documents in ./docs to Markdown
find ./docs -type f \( \
  -name '*.pdf' -o \
  -name '*.docx' -o \
  -name '*.pptx' -o \
  -name '*.xlsx' -o \
  -name '*.epub' -o \
  -name '*.odt' -o \
  -name '*.rtf' -o \
  -name '*.html' \
\) | parallel --bar --jobs 4 ./batch-convert.sh {}

Breaking that down:

• find — discovers every file by extension, descending into subdirectories
• parallel --bar — shows a progress bar as jobs complete
• --jobs 4 — runs 4 conversions at once (tune based on your API plan; the free tier throttles at 10 concurrent)
• {} — parallel replaces this with each file path

Output from batch-convert.sh pipes to stdout, so you get a live log of what succeeded and what failed:

OK   ./docs/annual-report.pdf → ./docs/annual-report.md (24 pages, 412 ms)
OK   ./docs/slides.pptx → ./docs/slides.md (18 pages, 510 ms)
OK   ./docs/manual.docx → ./docs/manual.md (45 pages, 680 ms)
OK   ./docs/budget.xlsx → ./docs/budget.md (3 pages, 290 ms)
FAIL ./docs/corrupted.pdf — HTTP 422: Unsupported or corrupted file
OK   ./docs/whitepaper.epub → ./docs/whitepaper.md (8 pages, 340 ms)
...

Logging results as JSON for analysis

For larger jobs, pipe the results into a structured JSON log for later analysis:

#!/usr/bin/env bash
# batch-convert-json.sh — emits one JSON line per conversion
set -euo pipefail

API_KEY="${ANYMD_API_KEY:?}"
FILE="$1"
START=$(date +%s%3N)

HEADERS=$(mktemp)
OUTPUT=$(mktemp)

HTTP_CODE=$(curl -s -w "%{http_code}" -D "$HEADERS" \
  -X POST https://anymd.net/api/convert \
  -H "Authorization: Bearer $API_KEY" \
  -F "file=@$FILE" \
  -o "$OUTPUT" 2>&1)

END=$(date +%s%3N)
DURATION_MS=$(( END - START ))

if [ "$HTTP_CODE" = "200" ]; then
  PAGES=$(grep -i '^X-Pages:' "$HEADERS" | awk '{print $2}' | tr -d '\r' || echo 0)
  SIZE=$(wc -c < "$OUTPUT")
  jq -n \
    --arg file "$FILE" \
    --arg status "ok" \
    --arg pages "$PAGES" \
    --argjson size "$SIZE" \
    --argjson duration "$DURATION_MS" \
    '{file: $file, status: $status, pages: ($pages | tonumber), size: $size, duration_ms: $duration}'
  mv "$OUTPUT" "${FILE%.*}.md"
else
  ERROR=$(jq -r '.error // "unknown"' "$OUTPUT" 2>/dev/null || echo "unknown")
  jq -n \
    --arg file "$FILE" \
    --arg status "fail" \
    --arg error "$ERROR" \
    --argjson duration "$DURATION_MS" \
    '{file: $file, status: $status, error: $error, duration_ms: $duration}'
fi

rm -f "$HEADERS" "$OUTPUT"

Run it like this:

find ./docs -type f -name '*.pdf' -o -name '*.docx' | \
  parallel --jobs 6 ./batch-convert-json.sh {} > conversion-log.jsonl

# Summary
jq -s 'group_by(.status) | map({status: .[0].status, count: length})' conversion-log.jsonl

# Average duration for successful conversions
jq -s '[.[] | select(.status == "ok") | .duration_ms] | add / length' conversion-log.jsonl

That last pipeline computes the average conversion time for all successful documents — all from the command line, no Python, no pandas.

Using the batch endpoint (server-side batching)

For small collections (under 50 files), AnyMD's /api/convert/batch endpoint lets the server handle concurrency:

#!/usr/bin/env bash
set -euo pipefail

API_KEY="${ANYMD_API_KEY:?}"
DIR="${1:-./docs}"

# Build a multipart request with multiple files
FILES=()
for f in "$DIR"/*.{pdf,docx,pptx}; do
  [ -f "$f" ] && FILES+=(-F "files=@$f")
done

curl -s -X POST https://anymd.net/api/convert/batch \
  -H "Authorization: Bearer $API_KEY" \
  "${FILES[@]}" \
  | jq -c '.[] | {file: .filename, status: .status, pages: .pages}'

The batch endpoint returns a JSON array of results — one per file — with each containing the filename, status, and Markdown content (or error). The client-side parallel approach above scales much further, since there's no request size limit and no single-point-of-failure.

Parallel performance: curl + parallel vs SDK clients

How fast is a shell-based batch pipeline compared to language-specific SDKs? We benchmarked 500 documents (mixed PDF, DOCX, PPTX) of varying sizes on a 4-core machine with 10 concurrent connections:

Approach	Total time (500 docs)	Throughput	Setup cost
Bash + curl + parallel	58 s	517 docs/min	0 s (tools pre-installed)
Python (httpx + asyncio)	71 s	423 docs/min	~10 s (pip install)
Node.js (undici + p-limit)	63 s	478 docs/min	~15 s (npm install)
Go (net/http + goroutines)	61 s	495 docs/min	~5 s (go build)
Rust (reqwest + tokio)	59 s	512 docs/min	~30 s (cargo build)

The shell pipeline is within 3% of the fastest compiled-language client — and it requires zero compilation, zero dependency installation, and zero code beyond a 20-line shell script. For ad-hoc batch jobs and CI/CD pipelines, it's the pragmatic winner.

Error recovery: resume on re-run

One more trick. When you're converting a large batch and something fails partway through, you don't want to re-convert everything. Use a marker file pattern:

#!/usr/bin/env bash
# resume-convert.sh — skip files that already have a .md output
set -euo pipefail

API_KEY="${ANYMD_API_KEY:?}"
DIR="${1:-./docs}"

find "$DIR" -type f \( -name '*.pdf' -o -name '*.docx' -o -name '*.pptx' \) | \
  while read -r file; do
    out="${file%.*}.md"
    [ -f "$out" ] && echo "SKIP $file (exists)" && continue
    echo "$file"
  done | \
  parallel --jobs 4 ./batch-convert.sh {}

Run it once. If the network drops after 300 files, run it again — it picks up where it left off. No database, no state file, no resume logic in your code. Just test -f.

Wrapping up

The Unix shell is the original batch processing framework. With curl + jq + parallel, you get a production-grade document conversion pipeline in about 20 lines of shell — no language runtime, no package manager, no import statements. Just files in, Markdown files out.

AnyMD's API is designed to work equally well from any language, including no language at all. Every format: PDF, DOCX, PPTX, XLSX, EPUB, ODT, RTF, HTML. Every size: from a single file to 10,000-file directories. One API, one auth header, curl from the terminal.

See also our Python, Node.js, Rust, and Go tutorials for when you need an SDK, or the API-from-any-language post for the universal approach. For full API reference and pricing, see the docs.

Get your free API key — 100 pages/month free, no credit card required. Then run find . -name '*.pdf' | parallel curl ... and watch your document library convert itself.