Merge pull request #145 from living-ip/codex/gcp-cloudsql-fail-closed-postrestart-20260714
Some checks are pending
CI / lint-and-test (push) Waiting to run
Some checks are pending
CI / lint-and-test (push) Waiting to run
Keep GCP knowledge reads on Cloud SQL after restart
This commit is contained in:
twentyOne2x
GitHub
commit
794635d17f
9 changed files with 267 additions and 18 deletions
|
|
@ -71,7 +71,15 @@ Do not describe a passing Hermes memory sync as canonical KB parity.
|
|||
retrieval receipts.
|
||||
- The generated clone was deleted. The retained rollback database remains
|
||||
connection-disabled with zero sessions. The GCP gateway remained PID
|
||||
`148735`, `NRestarts=0`, active/running throughout.
|
||||
`148735`, `NRestarts=0`, active/running throughout that bounded experiment.
|
||||
- PR `#144` merged the reviewed helper/skill. After deployment, the service
|
||||
survived a controlled restart and is active/running at PID `304036`, start
|
||||
time `2026-07-14 10:40:17 UTC`. Live post-restart `status` and `search`
|
||||
returned Cloud SQL retrieval receipts with unchanged canonical counts.
|
||||
- A live regression showed the old wrapper could time out its stronger status
|
||||
probe and fall through to a different local tool. Supported GCP KB commands
|
||||
now route directly to Cloud SQL and fail closed on errors; two behavioral
|
||||
tests enforce that invariant.
|
||||
- Persistent GCP `teleo_canonical` remains the older staging copy measured at
|
||||
`52,164` rows and `26` proposals. It has not been promoted or cut over.
|
||||
|
||||
|
|
@ -82,6 +90,7 @@ Primary retained proof:
|
|||
- `docs/reports/leo-working-state-20260709/gcp-db-first-parity-current.json`
|
||||
- `docs/reports/leo-working-state-20260709/gcp-db-first-blind-claim-current.json`
|
||||
- `docs/reports/leo-working-state-20260709/gcp-db-first-cleanup-current.json`
|
||||
- `docs/reports/leo-working-state-20260709/gcp-db-first-live-deploy-restart-current.json`
|
||||
|
||||
## Required Parity Rows
|
||||
|
||||
|
|
|
|||
|
|
@ -27,9 +27,11 @@ Orient the worker before action. Build a current, proof-linked understanding of
|
|||
- GCP is a separate lane. Direct passwordless SSH and `sudo -n` work as of
|
||||
2026-07-14. The newest VPS DB was restored to a disposable private Cloud SQL
|
||||
clone with exact parity, and a real ID-free no-send reasoning turn passed.
|
||||
Persistent GCP `teleo_canonical` is still an older staging copy; Telegram
|
||||
delivery, GCP canonical mutation, ongoing replication, promotion, and cutover
|
||||
remain separate proof rows.
|
||||
The merged helper/skill were deployed, supported KB commands now fail closed
|
||||
to Cloud SQL, and the service survived a controlled restart with live
|
||||
post-restart status/search receipts. Persistent GCP `teleo_canonical` is still
|
||||
an older staging copy; Telegram delivery, GCP canonical mutation, ongoing
|
||||
replication, promotion, and cutover remain separate proof rows.
|
||||
|
||||
## Architecture Boundaries
|
||||
|
||||
|
|
@ -53,11 +55,12 @@ From the repo root, read:
|
|||
3. `docs/reports/leo-working-state-20260709/gcp-db-first-parity-current.json`
|
||||
4. `docs/reports/leo-working-state-20260709/gcp-db-first-blind-claim-current.json`
|
||||
5. `docs/reports/leo-working-state-20260709/gcp-db-first-cleanup-current.json`
|
||||
6. `docs/reports/leo-working-state-20260709/operator-surface-map.md`
|
||||
7. `docs/reports/leo-working-state-20260709/working-leo-definition-20260709.md`
|
||||
8. `.agents/skills/working-leo-m3taversal-outcomes/SKILL.md`
|
||||
9. `docs/reports/leo-working-state-20260709/approve-claim-clone-canary-current.md`
|
||||
10. `docs/reports/leo-working-state-20260709/gcp-cloud-sql-t3-live-readonly-current.md`
|
||||
6. `docs/reports/leo-working-state-20260709/gcp-db-first-live-deploy-restart-current.json`
|
||||
7. `docs/reports/leo-working-state-20260709/operator-surface-map.md`
|
||||
8. `docs/reports/leo-working-state-20260709/working-leo-definition-20260709.md`
|
||||
9. `.agents/skills/working-leo-m3taversal-outcomes/SKILL.md`
|
||||
10. `docs/reports/leo-working-state-20260709/approve-claim-clone-canary-current.md`
|
||||
11. `docs/reports/leo-working-state-20260709/gcp-cloud-sql-t3-live-readonly-current.md`
|
||||
|
||||
## Required Status Split
|
||||
|
||||
|
|
|
|||
|
|
@ -133,6 +133,10 @@ is green at its required tier:
|
|||
- The GCP helper now gives deterministic receipts for every read, handles a
|
||||
canonical-only restored DB without the optional audit schema, and returns an
|
||||
honest zero-hit result instead of crashing.
|
||||
- The reviewed helper/skill are merged and deployed. GCP survived a controlled
|
||||
restart, then live `status` and `search` reached persistent Cloud SQL with
|
||||
unchanged counts and stable receipts. Supported commands fail closed instead
|
||||
of falling back to a different local database.
|
||||
- The disposable GCP database and temporary profiles/processes were removed.
|
||||
Persistent GCP `teleo_canonical` remains the older staging copy and has not
|
||||
been promoted or cut over.
|
||||
|
|
|
|||
|
|
@ -15,7 +15,17 @@ section below.
|
|||
claim plus both expected source rows, passed `18/18` runtime checks and `6/6`
|
||||
reasoning outcomes, made no DB write, and sent no Telegram message.
|
||||
- The disposable clone was deleted. The rollback DB remains disabled with zero
|
||||
connections. The live GCP gateway stayed PID `148735`, `NRestarts=0`.
|
||||
connections. The live GCP gateway stayed PID `148735`, `NRestarts=0` during
|
||||
the disposable restore/reasoning/cleanup run.
|
||||
- PR `#144` then merged the database-first helper and skill. They were deployed
|
||||
to GCP and the service survived a controlled restart: active/running at PID
|
||||
`304036`, start time `2026-07-14 10:40:17 UTC`.
|
||||
- Post-restart status and search both routed to persistent Cloud SQL and emitted
|
||||
retrieval receipts. The search found the expected claim and both expected
|
||||
source IDs/hashes. Canonical counts remained unchanged.
|
||||
- The post-restart search exposed and repaired a wrapper fallback bug: supported
|
||||
GCP KB commands now fail closed on Cloud SQL errors instead of silently using
|
||||
a different local database. Full repository tests pass `1240/1240`.
|
||||
- Persistent GCP `teleo_canonical` is still the older staging copy measured at
|
||||
`52,164` rows and `26` proposals. No promotion or production cutover occurred.
|
||||
- Fast snapshot restore is now proven. Full reconstruction from every original
|
||||
|
|
@ -28,6 +38,7 @@ Newest GCP database-first artifacts:
|
|||
- `gcp-db-first-parity-current.json`
|
||||
- `gcp-db-first-blind-claim-current.json`
|
||||
- `gcp-db-first-cleanup-current.json`
|
||||
- `gcp-db-first-live-deploy-restart-current.json`
|
||||
|
||||
## Live Addendum - 2026-07-14
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,125 @@
|
|||
{
|
||||
"artifact": "gcp_db_first_live_deploy_restart",
|
||||
"generated_at_utc": "2026-07-14T10:50:13Z",
|
||||
"status": "pass",
|
||||
"source": {
|
||||
"database_first_merge_commit": "08371bf2a0461536c112e1235a7229b52ed270cc",
|
||||
"database_first_pull_request": 144,
|
||||
"fail_closed_wrapper_commit": "8451c51",
|
||||
"repository": "living-ip/teleo-infrastructure"
|
||||
},
|
||||
"deployment": {
|
||||
"compute": "teleo-prod-1",
|
||||
"service": "leoclean-gcp-prod-parallel.service",
|
||||
"database": "teleo_canonical",
|
||||
"database_classification": "GCP staging; not promoted or cut over",
|
||||
"files": {
|
||||
"cloudsql_memory_tool.py": {
|
||||
"before_sha256": "f4396298405b1cbc92590e148b6272c66bb0e8461178edc3ffcfd02bfed69ab4",
|
||||
"after_sha256": "7d2074a0fc5f0d48fbf8f7905a72ead8f2696c86a041fa43e078fff5500baa51",
|
||||
"owner": "teleo:teleo",
|
||||
"mode": "0755"
|
||||
},
|
||||
"teleo-kb": {
|
||||
"before_sha256": "b988079ff2f23b962c40d4c6c9f20c0b85a7bdb0ee675ae146cf36f18ea6529a",
|
||||
"after_sha256": "63cf36cc5840f51a832e7728a4e818d981e106629e44a9f44b7fe8fed6f8a46c",
|
||||
"owner": "teleo:teleo",
|
||||
"mode": "0755"
|
||||
},
|
||||
"teleo-kb-bridge/SKILL.md": {
|
||||
"before_sha256": "523bf6975523c7ea94c043b9e60be3375b122860b951f78ad13dad10fe822de8",
|
||||
"after_sha256": "703fe9aa681ab3aa88dffa4b8c9cf2282329fe6c8f9414ad2d675fb0599679b2",
|
||||
"owner": "teleo:teleo",
|
||||
"mode": "0644"
|
||||
}
|
||||
}
|
||||
},
|
||||
"restart": {
|
||||
"before": {
|
||||
"ActiveState": "active",
|
||||
"SubState": "running",
|
||||
"MainPID": "148735",
|
||||
"NRestarts": "0",
|
||||
"ExecMainStartTimestamp": "Thu 2026-07-09 07:00:04 UTC"
|
||||
},
|
||||
"after": {
|
||||
"ActiveState": "active",
|
||||
"SubState": "running",
|
||||
"MainPID": "304036",
|
||||
"NRestarts": "0",
|
||||
"ExecMainStartTimestamp": "Tue 2026-07-14 10:40:17 UTC"
|
||||
},
|
||||
"controlled_restart_completed": true,
|
||||
"rollback_used": false
|
||||
},
|
||||
"integration_regression": {
|
||||
"detected_live": true,
|
||||
"symptom": "The old auto-mode wrapper used a full receipted status call as a 30-second health probe; the stronger read exceeded that bound and the wrapper fell through to a different local tool.",
|
||||
"repair": "Supported GCP knowledge commands now route directly to Cloud SQL and fail closed on errors instead of changing databases.",
|
||||
"regression_tests": 2,
|
||||
"full_repository_tests": {
|
||||
"passed": 1240,
|
||||
"failed": 0
|
||||
}
|
||||
},
|
||||
"post_restart_status": {
|
||||
"backend": "cloudsql:teleo-pgvector-standby/teleo_canonical/public+kb_stage primary; teleo_kb/teleo_restore fallback",
|
||||
"audit_fallback_available": true,
|
||||
"canonical_counts": {
|
||||
"claims": 1837,
|
||||
"sources": 4145,
|
||||
"claim_edges": 4916,
|
||||
"claim_evidence": 4670,
|
||||
"kb_proposals": 26
|
||||
},
|
||||
"retrieval_receipt": {
|
||||
"schema": "livingip.teleoKbRetrievalReceipt.v1",
|
||||
"semantic_context_sha256": "a428a88de299d9a20a11d614f1edf7b41cbbe78ec5bb7ff4455b6de525252641",
|
||||
"artifact_state_sha256": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945",
|
||||
"read_consistency_status": "stable_content_across_wal_change_retry",
|
||||
"attempts": 2,
|
||||
"database": "teleo_canonical"
|
||||
}
|
||||
},
|
||||
"post_restart_search": {
|
||||
"query": "AI sandbagging M&A liability",
|
||||
"command": "search",
|
||||
"backend": "cloudsql:teleo-pgvector-standby/teleo_canonical/public+kb_stage",
|
||||
"expected_claim_found": "2a7ae257-d01d-46f4-b813-63f81bb9c7c7",
|
||||
"expected_sources": [
|
||||
{
|
||||
"id": "15740795-ecc6-40fa-9a01-3d6bc7c54f79",
|
||||
"hash": "341a7685cd2909ae821d8e464fe0d4fdfbb0cadccf546ccfd4ccaf037aa9f646"
|
||||
},
|
||||
{
|
||||
"id": "261c3532-fa32-47d8-a5b5-6cc45035c267",
|
||||
"hash": "9fcd40191449fe0843178e08120fb3c86fc2b35633594aee60e747ca3918cfa5"
|
||||
}
|
||||
],
|
||||
"retrieval_receipt": {
|
||||
"schema": "livingip.teleoKbRetrievalReceipt.v1",
|
||||
"semantic_context_sha256": "9d2b81918c563a5703794e7e1c52bcafdfc3a4de917eb7f31f36f33f0538ef92",
|
||||
"artifact_state_sha256": "d473c2ab362021dcf0174a4ed4aedfcd95acd705e05b63eaa3f2ee4a9b53bde7",
|
||||
"read_consistency_status": "stable_content_across_wal_change_retry",
|
||||
"attempts": 2,
|
||||
"database": "teleo_canonical"
|
||||
}
|
||||
},
|
||||
"safety": {
|
||||
"telegram_message_sent": false,
|
||||
"canonical_database_write_attempted": false,
|
||||
"database_promoted_or_cut_over": false,
|
||||
"public_cloud_sql_access_enabled": false,
|
||||
"post_restart_model_turn_run": false
|
||||
},
|
||||
"claim_ceiling": {
|
||||
"proven": "Merged database-first helper and skill deployment, fail-closed Cloud SQL routing, controlled GCP service restart, and post-restart live status/search receipts.",
|
||||
"not_proven": [
|
||||
"post-restart model reasoning turn",
|
||||
"Telegram-visible delivery",
|
||||
"canonical proposal apply",
|
||||
"production promotion or cutover",
|
||||
"continuous VPS-to-GCP replication"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -94,6 +94,27 @@ remaining target databases. The disabled rollback database still has zero
|
|||
connections. The live GCP Leo service remained PID `148735`, active/running,
|
||||
with `NRestarts=0` throughout restore, model replay, and cleanup.
|
||||
|
||||
### 5. The merged GCP path survived deployment and restart
|
||||
|
||||
PR `#144` merged the database-first helper and skill. Their exact reviewed
|
||||
hashes were installed on `teleo-prod-1`, then
|
||||
`leoclean-gcp-prod-parallel.service` was intentionally restarted. It returned
|
||||
active/running with a new PID (`304036`) and the deployed hashes remained in
|
||||
place.
|
||||
|
||||
The first post-restart search caught one real integration regression: the old
|
||||
wrapper used a full `status` read as a 30-second health probe. Stronger receipts
|
||||
made that probe exceed its bound, after which `auto` mode could silently select
|
||||
a different local tool. The wrapper now routes every supported GCP knowledge
|
||||
command directly to Cloud SQL and fails closed on errors. Two regression tests
|
||||
cover direct routing and no fallback.
|
||||
|
||||
A corrected live search then found the expected sandbagging claim plus both
|
||||
expected source rows and hashes from persistent `teleo_canonical`. A subsequent
|
||||
status read returned the same `1,837` claims, `4,145` sources, `4,916` edges,
|
||||
`4,670` evidence rows, and `26` proposals as before deployment. Both commands
|
||||
returned stable retrieval receipts. No Telegram message or DB write occurred.
|
||||
|
||||
## VPS Versus GCP
|
||||
|
||||
| Surface | Current result | Meaning |
|
||||
|
|
@ -119,6 +140,7 @@ rows as if they were source-code branches.
|
|||
| `gcp-db-first-parity-current.json` | `0173ee6707016e8412e6dd4326d61f71b6ef862bbc5b819079d62680676729f2` | Schema, row, role, extension, and performance parity passed |
|
||||
| `gcp-db-first-blind-claim-current.json` | `8a7cc3c1814eb385e858f12ef7579cd4524f37886c03fc6d9c61624b6aff2a52` | Real GCP Hermes reasoning passed with source-bound read receipts |
|
||||
| `gcp-db-first-cleanup-current.json` | `cac7e34f45653fabb696d911b6ccc921d3f291bf8cbe05a429d757e717dcafb4` | Clone absent, rollback disabled, service unchanged |
|
||||
| `gcp-db-first-live-deploy-restart-current.json` | `d70fdbac859b8c98d557f4df900139a93e6009e12f7b80c9adbd69a5655b9df7` | Merged deploy, fail-closed routing, restart, and live status/search readback passed |
|
||||
|
||||
## What This Unlocks
|
||||
|
||||
|
|
@ -141,6 +163,9 @@ rows as if they were source-code branches.
|
|||
and repository artifacts rather than restored from a database snapshot.
|
||||
4. VPS changes continuously replicate to GCP or GCP has been formally promoted.
|
||||
5. Broad reliability across many unseen claim domains and repeated trials.
|
||||
6. A model reasoning turn after the controlled GCP restart. The post-restart
|
||||
proof in this report is the real service plus live `teleo-kb` status/search
|
||||
path, not a new paid model response.
|
||||
|
||||
The next product-level proof is one natural Telegram challenge that reaches
|
||||
the same database-grounded reasoning, followed by one explicitly approved
|
||||
|
|
|
|||
|
|
@ -1,12 +1,11 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
PROFILE_DIR=/home/teleo/.hermes/profiles/leoclean
|
||||
PROFILE_DIR=${HERMES_HOME:-/home/teleo/.hermes/profiles/leoclean}
|
||||
KB_TOOL="$PROFILE_DIR/bin/kb_tool.py"
|
||||
CLOUDSQL_TOOL="$PROFILE_DIR/bin/cloudsql_memory_tool.py"
|
||||
MODE=${TELEO_KB_MODE:-auto}
|
||||
COMMAND=${1:-}
|
||||
CLOUDSQL_STATUS_TIMEOUT=${TELEO_CLOUDSQL_STATUS_TIMEOUT_SECONDS:-30}
|
||||
|
||||
cloudsql_supported() {
|
||||
case "$COMMAND" in
|
||||
|
|
@ -45,13 +44,12 @@ case "$MODE" in
|
|||
exec python3 "$KB_TOOL" "$@"
|
||||
;;
|
||||
auto)
|
||||
if [ -x "$CLOUDSQL_TOOL" ] && command -v psql >/dev/null 2>&1 && command -v gcloud >/dev/null 2>&1; then
|
||||
if cloudsql_supported; then
|
||||
if timeout "${CLOUDSQL_STATUS_TIMEOUT}s" python3 "$CLOUDSQL_TOOL" status --format json --redacted >/dev/null 2>&1; then
|
||||
# Canonical commands must fail closed on Cloud SQL errors. Falling through
|
||||
# to a different local database would make source-of-truth behavior depend
|
||||
# on health-check latency.
|
||||
if [ -x "$CLOUDSQL_TOOL" ] && command -v psql >/dev/null 2>&1 && command -v gcloud >/dev/null 2>&1 && cloudsql_supported; then
|
||||
run_cloudsql "$@"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
exec python3 "$KB_TOOL" --local "$@"
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import contextlib
|
|||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
|
@ -66,6 +67,76 @@ def test_cloudsql_wrapper_supports_expected_review_gated_commands() -> None:
|
|||
assert "canonical apply" in cloudsql_text.lower()
|
||||
|
||||
|
||||
def _install_wrapper_fixture(tmp_path: Path) -> tuple[Path, dict[str, str]]:
|
||||
profile = tmp_path / "profile"
|
||||
bin_dir = profile / "bin"
|
||||
bin_dir.mkdir(parents=True)
|
||||
wrapper = bin_dir / "teleo-kb"
|
||||
wrapper.write_bytes((BRIDGE_DIR / "teleo-kb").read_bytes())
|
||||
wrapper.chmod(0o700)
|
||||
cloudsql = bin_dir / "cloudsql_memory_tool.py"
|
||||
cloudsql.write_text(
|
||||
"import json, sys\nprint(json.dumps({'backend': 'cloudsql', 'argv': sys.argv[1:]}))\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
cloudsql.chmod(0o700)
|
||||
local = bin_dir / "kb_tool.py"
|
||||
local.write_text(
|
||||
"import json, sys\nprint(json.dumps({'backend': 'local', 'argv': sys.argv[1:]}))\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
local.chmod(0o700)
|
||||
fake_bin = tmp_path / "fake-bin"
|
||||
fake_bin.mkdir()
|
||||
for name in ("psql", "gcloud"):
|
||||
executable = fake_bin / name
|
||||
executable.write_text("#!/usr/bin/env bash\nexit 0\n", encoding="utf-8")
|
||||
executable.chmod(0o700)
|
||||
env = {
|
||||
**os.environ,
|
||||
"HERMES_HOME": str(profile),
|
||||
"TELEO_KB_MODE": "auto",
|
||||
"PATH": f"{fake_bin}:{os.environ['PATH']}",
|
||||
}
|
||||
return wrapper, env
|
||||
|
||||
|
||||
def test_cloudsql_wrapper_auto_mode_routes_supported_reads_without_slow_probe(tmp_path: Path) -> None:
|
||||
wrapper, env = _install_wrapper_fixture(tmp_path)
|
||||
|
||||
completed = subprocess.run(
|
||||
[str(wrapper), "search", "AI sandbagging M&A liability", "--format", "json"],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
env=env,
|
||||
)
|
||||
|
||||
assert completed.returncode == 0, completed.stderr
|
||||
result = json.loads(completed.stdout)
|
||||
assert result == {
|
||||
"backend": "cloudsql",
|
||||
"argv": ["search", "AI sandbagging M&A liability", "--format", "json"],
|
||||
}
|
||||
|
||||
|
||||
def test_cloudsql_wrapper_auto_mode_never_falls_back_after_supported_command_failure(tmp_path: Path) -> None:
|
||||
wrapper, env = _install_wrapper_fixture(tmp_path)
|
||||
cloudsql = Path(env["HERMES_HOME"]) / "bin" / "cloudsql_memory_tool.py"
|
||||
cloudsql.write_text("raise SystemExit(23)\n", encoding="utf-8")
|
||||
|
||||
completed = subprocess.run(
|
||||
[str(wrapper), "status", "--format", "json"],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
env=env,
|
||||
)
|
||||
|
||||
assert completed.returncode == 23
|
||||
assert "local" not in completed.stdout
|
||||
|
||||
|
||||
def test_bridge_edge_proposals_stage_strict_apply_payloads() -> None:
|
||||
cloudsql_text = (BRIDGE_DIR / "cloudsql_memory_tool.py").read_text()
|
||||
vps_text = (BRIDGE_DIR / "kb_tool.py").read_text()
|
||||
|
|
|
|||
|
|
@ -79,8 +79,11 @@ def test_gcp_skill_uses_current_operator_access_and_parity_truth() -> None:
|
|||
"gcp-db-first-parity-current.json",
|
||||
"gcp-db-first-blind-claim-current.json",
|
||||
"gcp-db-first-cleanup-current.json",
|
||||
"gcp-db-first-live-deploy-restart-current.json",
|
||||
"18/18",
|
||||
"6/6",
|
||||
"PID `304036`",
|
||||
"fail closed",
|
||||
"invalid_target",
|
||||
"older staging copy",
|
||||
):
|
||||
|
|
|
|||
Loading…
Reference in a new issue