Datasets
A language model trains on a corpus too large to fetch at once, published as
many numbered files. This chapter covers how bunsen names, pins and fetches
those files, and where they go next. The reference is
bunsen::data::datasets, whose
lifecycle links every step.
The pathway
a catalog build a dataset, a templated resource of numbered files
│ view it, select a range of shards
▼
ShardSet the set, bound to a directory
│ locate / fetch / fetch_many
▼
shard paths
│
▼
a data loader bunsen-arrow-dataloaders, in train-chat
A ShardSetDescriptor says
what a set is: its name, the base URLs its files come from, how they are
named and numbered, and whether they are pinned. It is plain data, a view of
a catalog build of a dataset contract (parquet-shards), whose shards are
one templated resource; NanoChat’s corpus
is the built-in catalog’s karpathy/fineweb-edu-100b-shuffle
(from_builtin). A
ShardSet binds a descriptor to a
directory, the disk cache’s or one the caller gives, and brings shards in.
Binding needs the cache feature; downloading needs fetch. Without
fetch, a set works from shards that are already on disk.
Pinned shards
A pinned set carries one SHA-256 per shard, and a fetch verifies each file
as its bytes land. The pins are generated, not typed: cargo catalog reads
the dataset’s file listing at a pinned revision of its Hugging Face
repository and lists each shard’s digest in the build’s resource, and the
descriptor’s base URL resolves that same revision. The files and their
digests therefore can’t drift apart.
bunsen stops at paths
bunsen’s part ends with a list of local paths. Reading them is a data
loader’s job, and the loader that reads NanoChat’s Parquet shards is a
separate crate,
bunsen_arrow_dataloaders, a preview that does
not depend on bunsen. Its
ChatDataLoader
streams the shards through text selection, tokenization and packing into
fixed-shape blocks of token ids, with an optional shuffle
(pipeline overview).
A selection of shards can also become a
ResourceSet, through
to_resource_set,
for a build or a loader that takes the shards as resources.
End to end: train-chat
examples/train-chat
joins the two halves. Its command line selects shards of NanoChat’s corpus
and a fetch policy, it fetches the selection through a ShardSet, splits the
paths into training and validation, and hands each part to a
ChatDataLoader. The shard-selection arguments come from the bunsen-app
dev crate, and the loader crate’s own dl-test example uses the same ones.