teleo/teleo-infrastructure
8
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1

Merge pull request #209 from living-ip/codex/review-gcp-pr206-cache-isolation-20260721

Browse source 
Retain inspected digest references in shared Docker caches
This commit is contained in:
Fawaz 2026-07-21 00:14:43 -07:00 committed by GitHub
commit 2abe48b7e9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 120 additions and 127 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

29
docs/gcp-leoclean-nosend-service-package.md
View file

@ -186,13 +186,15 @@ The Docker binary, config directory, and local `unix://` daemon endpoint are
explicit inputs; inherited `DOCKER_*` overrides are removed. The pull does not explicit inputs; inherited `DOCKER_*` overrides are removed. The pull does not
start a container, install a unit, restart a service, or contact Cloud SQL. start a container, install a unit, restart a service, or contact Cloud SQL.
Registry authentication and this pull require a separate live-read approval. Registry authentication and this pull require a separate live-read approval.
The exact repository-digest reference must be absent before inspection; use a While holding an advisory lock on the explicit Docker config directory, the
fresh isolated Docker context if it is already cached. While holding an command pulls and records the dynamically observed config ID. It intentionally
advisory lock on the explicit Docker config directory, the command pulls, retains the exact repository-digest reference: Docker's image cache belongs to
records the dynamically observed config ID, then removes the reference only if the daemon, so a config-directory lock cannot prove that another client did not
that ID is still unchanged and proves it absent before publishing the bundle. start relying on the same immutable reference after the precheck. Repeated runs
Content-addressable layers may remain. A failed pull, inspection, or cleanup re-pull and re-inspect the same digest without deleting shared cache state.
returns a bounded error without forwarding Docker stderr. Content-addressable layers and the verified digest reference may remain. A
failed pull or inspection returns a bounded error without forwarding Docker
stderr.
Tags, alternate repositories, alternate projects/instances/service accounts, Tags, alternate repositories, alternate projects/instances/service accounts,
extra descriptor fields, wrong-architecture images, missing repository-digest extra descriptor fields, wrong-architecture images, missing repository-digest
@ -202,7 +204,8 @@ values are rejected.
## Offline evidence and live gate ## Offline evidence and live gate
Offline tests prove descriptor and inspection strictness, explicit local Docker Offline tests prove descriptor and inspection strictness, explicit local Docker
authority, cache-reference rollback, atomic bundle publication, artifact and synthetic identity binding, authority, non-destructive cache reuse, atomic bundle publication, artifact and
synthetic identity binding,
digest-only images, pinned runtime versions, a secret-free context, the digest-only images, pinned runtime versions, a secret-free context, the
capability-drop contract, no published ports, no Docker-socket mount, no capability-drop contract, no published ports, no Docker-socket mount, no
environment file, and no production/VPS/Telegram target. CI also builds exactly environment file, and no production/VPS/Telegram target. CI also builds exactly
@ -231,7 +234,7 @@ virtual environment.
Publishing the draft branch and stacked PR is only a review handoff. The next Publishing the draft branch and stacked PR is only a review handoff. The next
deployment-evidence step must execute one authorized remote registry read plus deployment-evidence step must execute one authorized remote registry read plus
the reversible local exact-reference cache mutation against the staging digest. the retained local exact-reference cache mutation against the staging digest.
A later installer slice must prove installation and rollback consume the same A later installer slice must prove installation and rollback consume the same
receipt. Live installation, Artifact Registry publication, service restart, receipt. Live installation, Artifact Registry publication, service restart,
IAM/secret changes, Cloud SQL access, and proposal staging remain separate IAM/secret changes, Cloud SQL access, and proposal staging remain separate
@ -240,10 +243,10 @@ approvals.
## Rollback and sunset ## Rollback and sunset
This slice's rollback boundary is removal of the unpublished atomic output This slice's rollback boundary is removal of the unpublished atomic output
bundle plus removal of the exact repository-digest reference that this command bundle. The verified repository-digest reference and content-addressable layers
introduced into an initially clean, advisory-locked cache; content-addressable may remain in the local daemon cache so the finalizer never deletes shared cache
layers may remain. It installs no unit state it cannot exclusively own. It installs no unit and changes no running
and changes no running service. Restoring a prior descriptor, unit, and digest service. Restoring a prior descriptor, unit, and digest
belongs to the later installer slice, which must never restart or edit the belongs to the later installer slice, which must never restart or edit the
production Leo, Telegram, or VPS services. production Leo, Telegram, or VPS services.

64
ops/gcp_leoclean_nosend_package.py
View file

@ -1741,36 +1741,6 @@ def _local_image_reference_config_id(
return config_id return config_id
def _remove_new_image_reference(
docker_binary: Path,
docker_host: str,
docker_config: Path,
image_reference: str,
expected_config_id: str,
) -> None:
observed_config_id = _local_image_reference_config_id(
docker_binary,
docker_host,
docker_config,
image_reference,
)
if observed_config_id is None:
return
_require(observed_config_id == expected_config_id, "new registry image reference changed before cleanup")
_run_docker(
docker_binary,
docker_host,
docker_config,
["image", "rm", image_reference],
check=True,
timeout=120,
)
_require(
_local_image_reference_config_id(docker_binary, docker_host, docker_config, image_reference) is None,
"new registry image reference cleanup did not pass",
)
def inspect_registry_image( def inspect_registry_image(
image_input: dict[str, Any], image_input: dict[str, Any],
image_reference: str, image_reference: str,
@ -1806,17 +1776,8 @@ def inspect_registry_image(
"image input differs from the expected reviewed input SHA-256", "image input differs from the expected reviewed input SHA-256",
) )
_validate_docker_authority(docker_binary, docker_host, docker_config) _validate_docker_authority(docker_binary, docker_host, docker_config)
prestate_observed = False
pulled_config_id: str | None = None
inspection: dict[str, Any] | None = None
primary_error: BaseException | None = None
with _DockerCacheLock(docker_config): with _DockerCacheLock(docker_config):
try: try:
_require(
_local_image_reference_config_id(docker_binary, docker_host, docker_config, image_reference) is None,
"exact registry image reference must be absent before inspection",
)
prestate_observed = True
_run_docker( _run_docker(
docker_binary, docker_binary,
docker_host, docker_host,
@ -1849,30 +1810,7 @@ def inspect_registry_image(
"inspected image differs from the expected reviewed config digest", "inspected image differs from the expected reviewed config digest",
) )
except (OSError, subprocess.SubprocessError, PackageError) as exc: except (OSError, subprocess.SubprocessError, PackageError) as exc:
primary_error = exc raise PackageError("immutable registry image inspection failed") from exc
finally:
try:
if prestate_observed:
if pulled_config_id is None:
pulled_config_id = _local_image_reference_config_id(
docker_binary,
docker_host,
docker_config,
image_reference,
)
if pulled_config_id is not None:
_remove_new_image_reference(
docker_binary,
docker_host,
docker_config,
image_reference,
pulled_config_id,
)
except (OSError, subprocess.SubprocessError, PackageError) as cleanup_error:
raise PackageError("immutable registry image cleanup failed") from cleanup_error
if primary_error is not None:
raise PackageError("immutable registry image inspection failed") from primary_error
_require(inspection is not None, "immutable registry image inspection failed")
return inspection return inspection

154
tests/test_gcp_leoclean_nosend_package.py
View file

@ -1064,7 +1064,7 @@ def test_local_image_inventory_rejects_inventory_and_inspection_disagreement(
@pytest.mark.parametrize("config_digest", ["sha256:" + "3" * 64, "sha256:" + "a" * 64]) @pytest.mark.parametrize("config_digest", ["sha256:" + "3" * 64, "sha256:" + "a" * 64])
def test_registry_inspection_uses_explicit_local_authority_and_removes_new_reference( def test_registry_inspection_absent_pull_dependent_retains_new_reference(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
config_digest: str, config_digest: str,
@ -1075,7 +1075,7 @@ def test_registry_inspection_uses_explicit_local_authority_and_removes_new_refer
prefix = [str(docker_binary), "--host", docker_host, "--config", str(docker_config)] prefix = [str(docker_binary), "--host", docker_host, "--config", str(docker_config)]
calls: list[list[str]] = [] calls: list[list[str]] = []
environments: list[dict[str, str]] = [] environments: list[dict[str, str]] = []
state = {"present": False} state = {"present": False, "dependent": False, "inspection_calls": 0}
payload = json.dumps( payload = json.dumps(
[docker_inspection_fixture(image_input, reference, config_digest=config_digest)] [docker_inspection_fixture(image_input, reference, config_digest=config_digest)]
).encode() ).encode()
@ -1099,6 +1099,9 @@ def test_registry_inspection_uses_explicit_local_authority_and_removes_new_refer
stderr=b"", stderr=b"",
) )
if operation[:2] == ["image", "inspect"]: if operation[:2] == ["image", "inspect"]:
state["inspection_calls"] += 1
if state["inspection_calls"] == 2:
state["dependent"] = True
return subprocess.CompletedProcess( return subprocess.CompletedProcess(
arguments, arguments,
0, 0,
@ -1109,9 +1112,7 @@ def test_registry_inspection_uses_explicit_local_authority_and_removes_new_refer
state["present"] = True state["present"] = True
return subprocess.CompletedProcess(arguments, 0, stdout=b"pulled\n", stderr=b"") return subprocess.CompletedProcess(arguments, 0, stdout=b"pulled\n", stderr=b"")
if operation[:2] == ["image", "rm"]: if operation[:2] == ["image", "rm"]:
assert "--force" not in operation raise AssertionError("digest reference must never be removed")
state["present"] = False
return subprocess.CompletedProcess(arguments, 0, stdout=b"removed\n", stderr=b"")
raise AssertionError(operation) raise AssertionError(operation)
monkeypatch.setattr(package.subprocess, "run", fake_run) monkeypatch.setattr(package.subprocess, "run", fake_run)
@ -1128,15 +1129,60 @@ def test_registry_inspection_uses_explicit_local_authority_and_removes_new_refer
) )
assert inspection["status"] == "pass" assert inspection["status"] == "pass"
assert state["present"] is False assert state == {"present": True, "dependent": True, "inspection_calls": 2}
assert [call[5:] for call in calls if call[5] == "pull"] == [ assert [call[5:] for call in calls if call[5] == "pull"] == [
["pull", "--quiet", "--platform", package.PLATFORM, reference] ["pull", "--quiet", "--platform", package.PLATFORM, reference]
] ]
assert all(not any(name.startswith("DOCKER_") for name in environment) for environment in environments) assert all(not any(name.startswith("DOCKER_") for name in environment) for environment in environments)
assert [call[5:] for call in calls if call[5:7] == ["image", "rm"]] == [["image", "rm", reference]] assert not any(call[5:7] == ["image", "rm"] for call in calls)
def test_registry_inspection_rejects_reviewed_config_digest_mismatch_and_cleans_reference( def test_registry_inspection_retains_preexisting_reference(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
image_input = image_input_fixture(tmp_path)
reference = f"{package.IMAGE_REPOSITORY}@sha256:{IMAGE_DIGEST}"
docker_binary, docker_host, docker_config = docker_authority(tmp_path)
payload = json.dumps([docker_inspection_fixture(image_input, reference)]).encode()
state = {"present": True, "pulls": 0}
def fake_run(arguments: list[str], **_kwargs):
operation = arguments[5:]
if operation[:2] == ["image", "ls"]:
return subprocess.CompletedProcess(
arguments,
0,
stdout=docker_inventory_fixture(reference, present=state["present"]),
stderr=b"",
)
if operation[:2] == ["image", "inspect"]:
return subprocess.CompletedProcess(arguments, 0, stdout=payload, stderr=b"")
if operation[0] == "pull":
state["pulls"] += 1
return subprocess.CompletedProcess(arguments, 0, stdout=b"pulled\n", stderr=b"")
if operation[:2] == ["image", "rm"]:
raise AssertionError("digest reference must never be removed")
raise AssertionError(operation)
monkeypatch.setattr(package.subprocess, "run", fake_run)
inspection = package.inspect_registry_image(
image_input,
reference,
expected_revision=image_input["runtime"]["teleo_git_head"],
expected_input_sha256=image_input["input_sha256"],
expected_config_digest="sha256:" + "3" * 64,
docker_binary=docker_binary,
docker_host=docker_host,
docker_config=docker_config,
)
assert inspection["status"] == "pass"
assert state == {"present": True, "pulls": 1}
def test_registry_inspection_rejects_config_digest_mismatch_and_retains_reference(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
@ -1180,10 +1226,10 @@ def test_registry_inspection_rejects_reviewed_config_digest_mismatch_and_cleans_
) )
assert "expected reviewed config digest" in str(raised.value.__cause__) assert "expected reviewed config digest" in str(raised.value.__cause__)
assert state == {"present": False, "removed": True} assert state == {"present": True, "removed": False}
def test_registry_inspection_failure_is_redacted_and_cleans_new_reference( def test_registry_inspection_pull_failure_is_redacted_and_does_not_remove_reference(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
@ -1226,11 +1272,11 @@ def test_registry_inspection_failure_is_redacted_and_cleans_new_reference(
docker_config=docker_config, docker_config=docker_config,
) )
assert state["present"] is False assert state["present"] is True
assert "credential=value" not in str(raised.value) assert "credential=value" not in str(raised.value)
def test_registry_inspection_validation_failure_removes_new_reference( def test_registry_inspection_validation_failure_retains_reference(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
@ -1280,10 +1326,10 @@ def test_registry_inspection_validation_failure_removes_new_reference(
docker_config=docker_config, docker_config=docker_config,
) )
assert state == {"present": False, "removed": True} assert state == {"present": True, "removed": False}
def test_registry_inspection_cleanup_failure_prevents_success( def test_registry_inspection_never_calls_image_rm_after_success(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
@ -1318,20 +1364,22 @@ def test_registry_inspection_cleanup_failure_prevents_success(
monkeypatch.setattr(package.subprocess, "run", fake_run) monkeypatch.setattr(package.subprocess, "run", fake_run)
with pytest.raises(package.PackageError, match="cleanup failed"): inspection = package.inspect_registry_image(
package.inspect_registry_image( image_input,
image_input, reference,
reference, expected_revision=image_input["runtime"]["teleo_git_head"],
expected_revision=image_input["runtime"]["teleo_git_head"], expected_input_sha256=image_input["input_sha256"],
expected_input_sha256=image_input["input_sha256"], expected_config_digest="sha256:" + "3" * 64,
expected_config_digest="sha256:" + "3" * 64, docker_binary=docker_binary,
docker_binary=docker_binary, docker_host=docker_host,
docker_host=docker_host, docker_config=docker_config,
docker_config=docker_config, )
)
assert inspection["status"] == "pass"
assert state["present"] is True
def test_registry_inspection_refuses_to_remove_changed_reference( def test_registry_inspection_has_no_post_inspection_cleanup_window(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
@ -1373,23 +1421,24 @@ def test_registry_inspection_refuses_to_remove_changed_reference(
monkeypatch.setattr(package.subprocess, "run", fake_run) monkeypatch.setattr(package.subprocess, "run", fake_run)
with pytest.raises(package.PackageError, match="cleanup failed"): inspection = package.inspect_registry_image(
package.inspect_registry_image( image_input,
image_input, reference,
reference, expected_revision=image_input["runtime"]["teleo_git_head"],
expected_revision=image_input["runtime"]["teleo_git_head"], expected_input_sha256=image_input["input_sha256"],
expected_input_sha256=image_input["input_sha256"], expected_config_digest="sha256:" + "3" * 64,
expected_config_digest="sha256:" + "3" * 64, docker_binary=docker_binary,
docker_binary=docker_binary, docker_host=docker_host,
docker_host=docker_host, docker_config=docker_config,
docker_config=docker_config, )
)
assert inspection["status"] == "pass"
assert state["present"] is True assert state["present"] is True
assert state["removed"] is False assert state["removed"] is False
assert state["inventory_calls"] == 1
def test_registry_inspection_rejects_preexisting_reference_without_mutation( def test_registry_inspection_accepts_preexisting_reference_without_removal(
tmp_path: Path, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
) -> None: ) -> None:
@ -1411,24 +1460,27 @@ def test_registry_inspection_rejects_preexisting_reference_without_mutation(
) )
if operation[:2] == ["image", "inspect"]: if operation[:2] == ["image", "inspect"]:
return subprocess.CompletedProcess(arguments, 0, stdout=payload, stderr=b"") return subprocess.CompletedProcess(arguments, 0, stdout=payload, stderr=b"")
if operation[0] == "pull":
return subprocess.CompletedProcess(arguments, 0, stdout=b"pulled\n", stderr=b"")
if operation[:2] == ["image", "rm"]:
raise AssertionError("digest reference must never be removed")
raise AssertionError(operation) raise AssertionError(operation)
monkeypatch.setattr(package.subprocess, "run", fake_run) monkeypatch.setattr(package.subprocess, "run", fake_run)
with pytest.raises(package.PackageError, match="inspection failed") as raised: inspection = package.inspect_registry_image(
package.inspect_registry_image( image_input,
image_input, reference,
reference, expected_revision=image_input["runtime"]["teleo_git_head"],
expected_revision=image_input["runtime"]["teleo_git_head"], expected_input_sha256=image_input["input_sha256"],
expected_input_sha256=image_input["input_sha256"], expected_config_digest="sha256:" + "3" * 64,
expected_config_digest="sha256:" + "3" * 64, docker_binary=docker_binary,
docker_binary=docker_binary, docker_host=docker_host,
docker_host=docker_host, docker_config=docker_config,
docker_config=docker_config, )
)
assert "must be absent" in str(raised.value.__cause__) assert inspection["status"] == "pass"
assert not any(operation and operation[0] == "pull" for operation in operations) assert any(operation and operation[0] == "pull" for operation in operations)
assert not any(operation[:2] == ["image", "rm"] for operation in operations) assert not any(operation[:2] == ["image", "rm"] for operation in operations)