62 lines
2.3 KiB
Markdown
62 lines
2.3 KiB
Markdown
|
# Package Module Generation
|
|||
|
|
|
|||
|
|
This document explains how harvested metadata is transformed into concrete
|
|||
|
|
Rust modules under `src/pkgs/by_name/`.
|
|||
|
|
|
|||
|
|
## Overview
|
|||
|
|
|
|||
|
|
1. **Harvest metadata** – Use `metadata_indexer harvest` to capture package data
|
|||
|
|
from the LFS/BLFS/GLFS books. Each record is written to
|
|||
|
|
`ai/metadata/packages/<book>/<slug>.json`.
|
|||
|
|
2. **Refresh manifests** – Run
|
|||
|
|
`metadata_indexer refresh` to ensure the jhalfs `wget-list` and `md5sums`
|
|||
|
|
caches are up to date. Harvesting relies on these caches for canonical
|
|||
|
|
source URLs and checksums.
|
|||
|
|
3. **Generate modules** – Use
|
|||
|
|
`metadata_indexer generate --metadata <path> --output <by_name_dir>` to turn a
|
|||
|
|
metadata file into a full Rust module that exposes a `PackageDefinition`.
|
|||
|
|
|
|||
|
|
Generated modules leverage the existing scaffolder logic, so the command will
|
|||
|
|
create any missing prefix directories (e.g. `bi/mod.rs`) and populate the final
|
|||
|
|
`mod.rs` file with the correct code template.
|
|||
|
|
|
|||
|
|
## Command reference
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# Harvest metadata from a book page
|
|||
|
|
cargo run --bin metadata_indexer -- --base-dir . harvest \
|
|||
|
|
--book mlfs \
|
|||
|
|
--page chapter05/binutils-pass1 \
|
|||
|
|
--output ai/metadata/packages/mlfs/binutils-pass-1.json
|
|||
|
|
|
|||
|
|
# Refresh jhalfs manifests (optional but recommended)
|
|||
|
|
cargo run --bin metadata_indexer -- --base-dir . refresh
|
|||
|
|
|
|||
|
|
# Generate a module under the standard src tree
|
|||
|
|
cargo run --bin metadata_indexer -- --base-dir . generate \
|
|||
|
|
--metadata ai/metadata/packages/mlfs/binutils-pass-1.json \
|
|||
|
|
--output src/pkgs/by_name \
|
|||
|
|
--overwrite
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Flags
|
|||
|
|
|
|||
|
|
- `--output` defaults to `src/pkgs/by_name`. Point it to another directory if
|
|||
|
|
you want to stage modules elsewhere (e.g. `target/generated/by_name`).
|
|||
|
|
- `--overwrite` deletes the existing module directory before scaffolding a new
|
|||
|
|
one.
|
|||
|
|
|
|||
|
|
After generation, run `cargo fmt` and `cargo check` to ensure the crate compiles
|
|||
|
|
with the new modules.
|
|||
|
|
|
|||
|
|
## Implementation notes
|
|||
|
|
|
|||
|
|
- Metadata fields such as `build`, `dependencies`, and `optimizations` are
|
|||
|
|
mapped directly onto the scaffolder’s `ScaffoldRequest` type.
|
|||
|
|
- Source URLs and MD5 checksums are sourced from the harvested metadata
|
|||
|
|
(populated via the jhalfs manifests).
|
|||
|
|
- The module slug is derived from `package.id` (e.g.
|
|||
|
|
`mlfs/binutils-pass-1` → `src/pkgs/by_name/bi/binutils_pass_1/mod.rs`).
|
|||
|
|
|
|||
|
|
See the code in `src/pkgs/generator.rs` for the full translation logic.
|