How to speed up a slow OpenCTI TAXII import on large feeds

A large TAXII feed that keeps changing makes OpenCTI wait on missing references, so the first import slows to a crawl on an idle machine. More workers plus shorter retry waits increase import speed. A second pass after a cursor reset and "Import from date" deletion then imports the relationships the first pass had to drop.

Detailed explanation

TAXII returns objects in date_added order. When the feed re-versions an object, it moves to the end of the collection. Relationships and sightings that point to it keep their earlier position, so OpenCTI reads the reference before its target. The worker retries for about 7.5 seconds, then drops the object. The target usually arrives hundreds of pages later, so the waiting almost never pays off and the workers spend nearly all their time asleep.

If the feed re-versions more objects per hour than OpenCTI reads, the import never catches up.

Runbook

1. Check the version

The retry settings exist from OpenCTI version 7.260917. Older versions silently ignore them. Only the worker count applies there.

sudo docker exec xtm-worker-1 pip show pycti | grep Version

2. Set the worker options (recommended for 2 vCPU, 32 GB RAM, OpenCTI version 7.260921.0 on Docker)

Add this to docker-compose.override.yml next to docker-compose.yml:

services:
  worker:
    deploy:
      replicas: 18 # 1 message in flight per worker; 18–24 on 2 vCPU
    logging:
      driver: json-file
      options: { max-size: '50m', max-file: '3' } # keeps worker logs bounded
    environment:
      - OPENCTI_MISSING_REF_RETRY_EXPONENTIAL=true # enables the tunable schedule (default off)
      - OPENCTI_MISSING_REF_RETRY_INITIAL_DELAY=0.2 # default 0.5 s
      - OPENCTI_MISSING_REF_RETRY_FACTOR=1.5 # default 2.0 → ~1.6 s total wait instead of ~7.5 s

cd ~/opencti/docker
nano docker-compose.override.yml                 # add the worker block from step 2
docker compose config --quiet                    # validate
docker compose up -d --no-deps worker            # recreate the workers only

Only the workers are recreated: the platform keeps running, the feed cursor stays, and in-flight messages go back to the queue.

3. Wait until the first pass has caught up

4. Reset the cursor, delete the "Import from date" and run second pass on your feed

Do this only when the queue is empty, so no page is cut in half. In the UI before version 7.260722: Data → Ingestion → TAXII feeds → your feed → ⋮ → Reset state → ⋮ → Update → empty the "Import from date". In the UI after version 7.260722: Integrations → TAXII feeds → your feed → ⋮ → Reset state → ⋮ → Update → empty the "Import from date".

Every target now exists in OpenCTI, and references resolve by id whatever version is stored. Relationships and sightings dropped in the first pass should import on the first attempt; objects that already exist are plain upserts.

Rollback (optional)

Remove the INITIAL_DELAY and FACTOR lines (and lower replicas if needed), then run the same docker compose up -d --no-deps worker again. The workers return to the defaults 0.5 s / 2.0.