lpkg/docs/PACKAGE_GENERATION.md

Package Module Generation

This document explains how harvested metadata is transformed into concrete Rust modules under src/pkgs/by_name/.

Overview

1. Harvest metadata – Use metadata_indexer harvest to capture package data from the LFS/BLFS/GLFS books. Each record is written to ai/metadata/packages/<book>/<slug>.json.
2. Refresh manifests – Run metadata_indexer refresh to ensure the jhalfs wget-list and md5sums caches are up to date. Harvesting relies on these caches for canonical source URLs and checksums.
3. Generate modules – Use metadata_indexer generate --metadata <path> --output <by_name_dir> to turn a metadata file into a full Rust module that exposes a PackageDefinition.

Generated modules leverage the existing scaffolder logic, so the command will create any missing prefix directories (e.g. bi/mod.rs) and populate the final mod.rs file with the correct code template.

Command reference

# Harvest metadata from a book page
cargo run --bin metadata_indexer -- --base-dir . harvest \
  --book mlfs \
  --page chapter05/binutils-pass1 \
  --output ai/metadata/packages/mlfs/binutils-pass-1.json

# Refresh jhalfs manifests (optional but recommended)
cargo run --bin metadata_indexer -- --base-dir . refresh

# Generate a module under the standard src tree
cargo run --bin metadata_indexer -- --base-dir . generate \
  --metadata ai/metadata/packages/mlfs/binutils-pass-1.json \
  --output src/pkgs/by_name \
  --overwrite

Flags

• --output defaults to src/pkgs/by_name. Point it to another directory if you want to stage modules elsewhere (e.g. target/generated/by_name).
• --overwrite deletes the existing module directory before scaffolding a new one.

After generation, run cargo fmt and cargo check to ensure the crate compiles with the new modules.

Implementation notes

• Metadata fields such as build, dependencies, and optimizations are mapped directly onto the scaffolder’s ScaffoldRequest type.
• Source URLs and MD5 checksums are sourced from the harvested metadata (populated via the jhalfs manifests).
• The module slug is derived from package.id (e.g. mlfs/binutils-pass-1 → src/pkgs/by_name/bi/binutils_pass_1/mod.rs).

See the code in src/pkgs/generator.rs for the full translation logic.