61 lines
2.3 KiB
Markdown
61 lines
2.3 KiB
Markdown
# Package Module Generation
|
||
|
||
This document explains how harvested metadata is transformed into concrete
|
||
Rust modules under `src/pkgs/by_name/`.
|
||
|
||
## Overview
|
||
|
||
1. **Harvest metadata** – Use `metadata_indexer harvest` to capture package data
|
||
from the LFS/BLFS/GLFS books. Each record is written to
|
||
`ai/metadata/packages/<book>/<slug>.json`.
|
||
2. **Refresh manifests** – Run
|
||
`metadata_indexer refresh` to ensure the jhalfs `wget-list` and `md5sums`
|
||
caches are up to date. Harvesting relies on these caches for canonical
|
||
source URLs and checksums.
|
||
3. **Generate modules** – Use
|
||
`metadata_indexer generate --metadata <path> --output <by_name_dir>` to turn a
|
||
metadata file into a full Rust module that exposes a `PackageDefinition`.
|
||
|
||
Generated modules leverage the existing scaffolder logic, so the command will
|
||
create any missing prefix directories (e.g. `bi/mod.rs`) and populate the final
|
||
`mod.rs` file with the correct code template.
|
||
|
||
## Command reference
|
||
|
||
```bash
|
||
# Harvest metadata from a book page
|
||
cargo run --bin metadata_indexer -- --base-dir . harvest \
|
||
--book mlfs \
|
||
--page chapter05/binutils-pass1 \
|
||
--output ai/metadata/packages/mlfs/binutils-pass-1.json
|
||
|
||
# Refresh jhalfs manifests (optional but recommended)
|
||
cargo run --bin metadata_indexer -- --base-dir . refresh
|
||
|
||
# Generate a module under the standard src tree
|
||
cargo run --bin metadata_indexer -- --base-dir . generate \
|
||
--metadata ai/metadata/packages/mlfs/binutils-pass-1.json \
|
||
--output src/pkgs/by_name \
|
||
--overwrite
|
||
```
|
||
|
||
### Flags
|
||
|
||
- `--output` defaults to `src/pkgs/by_name`. Point it to another directory if
|
||
you want to stage modules elsewhere (e.g. `target/generated/by_name`).
|
||
- `--overwrite` deletes the existing module directory before scaffolding a new
|
||
one.
|
||
|
||
After generation, run `cargo fmt` and `cargo check` to ensure the crate compiles
|
||
with the new modules.
|
||
|
||
## Implementation notes
|
||
|
||
- Metadata fields such as `build`, `dependencies`, and `optimizations` are
|
||
mapped directly onto the scaffolder’s `ScaffoldRequest` type.
|
||
- Source URLs and MD5 checksums are sourced from the harvested metadata
|
||
(populated via the jhalfs manifests).
|
||
- The module slug is derived from `package.id` (e.g.
|
||
`mlfs/binutils-pass-1` → `src/pkgs/by_name/bi/binutils_pass_1/mod.rs`).
|
||
|
||
See the code in `src/pkgs/generator.rs` for the full translation logic.
|