145 lines
5.4 KiB
Markdown
145 lines
5.4 KiB
Markdown
# KB Restore / Replication Runbook
|
|
|
|
This runbook is for proving Living IP KB/database redundancy on GCP.
|
|
|
|
Current source reality:
|
|
|
|
- canonical runtime DB: `/opt/teleo-eval/pipeline/pipeline.db`
|
|
- engine: SQLite WAL
|
|
- canonical Leo files:
|
|
- `/opt/teleo-eval/workspaces/main/agents/leo`
|
|
- `/opt/teleo-eval/workspaces/research-leo/agents/leo`
|
|
- `/opt/teleo-eval/agent-state`
|
|
|
|
The standby target already exists:
|
|
|
|
- project: `teleo-501523`
|
|
- instance: `teleo-pgvector-standby`
|
|
- database: `teleo_kb`
|
|
- region: `europe-west6`
|
|
- network: `teleo-staging-net`
|
|
- private IP: `10.61.0.3`
|
|
- admin password secret: `gcp-teleo-pgvector-standby-postgres-password`
|
|
|
|
Do not call this redundancy complete until source data has been restored or replicated and queried from GCP.
|
|
|
|
## Source Backup Canary
|
|
|
|
Create a consistent source backup without stopping the VPS service:
|
|
|
|
```bash
|
|
ops/backup_vps_sqlite_kb.sh
|
|
```
|
|
|
|
The script:
|
|
|
|
- uses SQLite `.backup` against `/opt/teleo-eval/pipeline/pipeline.db`;
|
|
- compresses and hashes the backup on the VPS;
|
|
- archives Leo/KB files while excluding `secrets` and logs;
|
|
- copies both artifacts locally;
|
|
- verifies SHA-256 matches;
|
|
- runs `PRAGMA integrity_check` on a local restored SQLite copy;
|
|
- records proof under `outputs/gcp-infra-hardening-20260707/proofs/`.
|
|
|
|
This proves source exportability and local restore integrity. It does not prove GCP DB redundancy until a GCP restore/import/query canary also passes.
|
|
|
|
## Local SQLite-To-Postgres Restore Canary
|
|
|
|
Before importing into Cloud SQL, prove that the current SQLite backup can be
|
|
converted and restored into PostgreSQL without row loss:
|
|
|
|
```bash
|
|
SQLITE_BACKUP=./outputs/gcp-infra-hardening-20260707/private-backups/teleo-pipeline-sqlite-<timestamp>.db.gz \
|
|
ops/run_sqlite_postgres_restore_canary.sh
|
|
```
|
|
|
|
The canary:
|
|
|
|
- generates a PostgreSQL import script with `ops/sqlite_to_postgres_dump.py`;
|
|
- recreates a shadow schema in a disposable `postgres:16-alpine` container;
|
|
- imports all user tables from the SQLite backup;
|
|
- compares source and target row counts for every table;
|
|
- writes a proof JSON under `outputs/gcp-infra-hardening-20260707/proofs/`;
|
|
- removes only its temporary canary container.
|
|
|
|
This is a local restore/parity proof, not GCP redundancy by itself. It is the
|
|
preflight that should pass before the same generated import is applied through
|
|
the approved Cloud SQL connector/VPC path.
|
|
|
|
## Required Proof
|
|
|
|
A successful restore or replication canary must retain:
|
|
|
|
- source dataset identity:
|
|
- source host or dump artifact;
|
|
- dump timestamp or replication slot timestamp;
|
|
- source schema/database name.
|
|
- transfer proof:
|
|
- dump object path in a versioned bucket, or logical replication subscription details;
|
|
- row/table counts before import where available.
|
|
- target proof:
|
|
- `teleo-pgvector-standby` readback;
|
|
- `teleo_kb` database readback;
|
|
- extension readback for `vector` if the restored schema needs pgvector;
|
|
- representative query readback for core KB tables.
|
|
- failure boundary:
|
|
- exact missing secret, source access, schema incompatibility, extension issue, or import error.
|
|
|
|
## One-Shot SQLite Export / GCP Restore Path
|
|
|
|
Use this while the canonical DB remains SQLite on the VPS and we need a GCP restore drill.
|
|
|
|
1. Run `ops/backup_vps_sqlite_kb.sh`.
|
|
2. Upload the resulting SQLite backup and Leo/KB tarball to a versioned GCS bucket such as `gs://teleo-501523-prod-backups/kb-dumps/`.
|
|
3. Run the local SQLite-to-Postgres restore canary above and retain its proof.
|
|
4. Convert or replay the SQLite data into Cloud SQL with the explicit generated migration/import script. Do not run blind string rewrites against the SQLite dump.
|
|
5. Install required extensions on Cloud SQL:
|
|
|
|
```sql
|
|
create extension if not exists vector;
|
|
```
|
|
|
|
6. From a trusted VPC runtime or Cloud SQL connector path, run readbacks:
|
|
|
|
```sql
|
|
select current_database();
|
|
select extname, extversion from pg_extension where extname = 'vector';
|
|
select schemaname, tablename from pg_tables where schemaname not in ('pg_catalog', 'information_schema') order by 1, 2 limit 50;
|
|
```
|
|
|
|
7. Retain the SQLite backup hash, GCS object generation, import/conversion operation, query output, and row-count sample.
|
|
|
|
## Logical Replication Path
|
|
|
|
Use this only if the canonical source becomes Postgres or a Postgres mirror exists. SQLite cannot be logically replicated into Cloud SQL Postgres without an intermediate conversion/sync layer.
|
|
|
|
Required source privileges:
|
|
|
|
- replication-capable source user;
|
|
- publication over the intended schemas/tables;
|
|
- network path from GCP to source, or source-to-GCP path through an approved proxy/tunnel.
|
|
|
|
Required target steps:
|
|
|
|
```sql
|
|
create extension if not exists vector;
|
|
create subscription <subscription_name>
|
|
connection '<redacted source connection string>'
|
|
publication <publication_name>;
|
|
```
|
|
|
|
Retain only redacted connection metadata. Do not commit or paste credentials.
|
|
|
|
## Current Blocker
|
|
|
|
As of this run, GCP has the standby target, a repeatable SQLite/KB export script,
|
|
and a local SQLite-to-Postgres restore canary. It does not have an approved GCP
|
|
upload/restore credential in the current local session, nor a retained Cloud SQL
|
|
import/query proof. That is why the readiness checker still reports:
|
|
|
|
- `kb_source_restore_access = blocked`
|
|
- `kb_restore_or_replication = blocked`
|
|
|
|
The next real canary is:
|
|
|
|
source SQLite/KB backup -> upload to versioned GCS bucket -> convert/import into `teleo_kb` -> install/verify `vector` if needed -> run representative KB queries on GCP -> retain proof.
|