{"document_id":"1048310e808ee03ff165f8797c8d91f4fa7cb328d259d86fcaac66f46fb4758b","content_sha256":"29eeed712b5973ded36a1a1e2d28de8a237ff178e628b845b6517a321775d98a","hashed":{"address":"zetlyn://zetlyn/spec/README.md","title":"Open Zetlyn Corpus Specification","url":null,"language":"eng","published":"2026-09-12T20:52:18Z","keywords":"","text_chars":2417,"word_count":404,"text_hash":"5964cf7c40d5704b486d5f4813c9493c5ccb3cd242467929ce6d627e5aeea2d1","catalog_version":"folder-2","embed_model":"multilingual-e5-small","extra":{"bytes":2634,"extension":"md"}},"embed_method":"chunked-256w-mean-v1","text":"Open Zetlyn Corpus Specification\n\nTwo specifications, versioned separately:\n\n| | |\n|---|---|\n| Corpus 1.0 | what a corpus contains and declares, so two of them can be searched together |\n| Adapter protocol 1.0 | how a program feeds documents into a corpus |\n| Hub layout 1.0 | how a published corpus is served and fetched |\n| zetlyn-entities-1 | the entity vocabulary every conforming corpus uses |\n\nA corpus is a body of documents that can be searched. This specification says what one has to contain\nand what has to be declared about it, so that two corpora built by unrelated parties can be searched\ntogether without either being rebuilt.\n\nCombining two search indexes is easy if you re-index them into one. This is about the case where you\ncannot: the corpora belong to different people, sit on different machines, and one of them may be a\ncompany's own documents that never leave the building.\n\nWhat is not here\n\nThe Zetlyn runtime. This repository specifies the format and the interoperability contract. The engine\nthat plans a reading route over a corpus is a separate, proprietary product, documented in\nzetlynhq/docs. Building a conforming corpus needs neither it nor\npermission.\n\nThe line between the two runs on one criterion, stated in the specification rather than left to taste:\nanything that crosses a corpus boundary is specified here, anything that only orders documents inside\na single corpus is not. The adapter protocol and the hub layout are here for the same reason: anyone may write an adapter or\nrun a hub, and the ones Zetlyn provides hold no privilege over them.\n\nThe specification comes first. An implementation follows it, and where the two disagree the\nimplementation is wrong.\n\nLicence\n\nThe specification text is CC BY 4.0. Quote it, translate it, copy it into your own\ndocumentation, commercially or not.\n\nThe software in this repository is proprietary and source-available (same file, Part 2). You\nmay read it and run it to validate a corpus or an implementation of your own, including commercially.\nRedistributing it or building it into your own product needs permission.\n\nWhat the software produces is yours. A corpus built with these tools carries no obligation to anyone.\n\nFixtures carry their own licence, stated per fixture. One whose licence does not permit redistribution\ncannot be here.\n\n\"Zetlyn\" is a trademark. The format is free to implement, the name is not free to apply.","simhash":1748296438457921584,"bands":[35431939105987246,41050480919995979,45808694520845982,178317678902023520,240009370330877475,314585623659202756,368121311459871183,404928423809977364,434111974019239291,516215000109820346,537743990692164015,564328085745810079,579172778567733781,614354485793073590,668393988426222416,727885076444748655,771263534525103257,785290639101113539,838120927487270013,844321961686022234,865462175342439949,889646861716974526,1004274119240978617,1042991610959269105,1187222706737757098,1287256731044369369,1363567110054316121,1408076507425178743,1414290399894669667,1429257227879079653,1435120136442594952,1492212761005199328,1500324892619611183,1513516230060170088,1562985248474265234,1614687565083883300,1656393619356296961,1694827953142743515,1713792584413587284,1739284811875465247,1750811098264347380,1750982637268464688,1819077084648604332,1823046147155123285,1927233039829194924,2075930643799879836,2116933122615439508,2212490792848675290,2224796599655282868,2315612234286200303,2322841308808954940,2370307410125657863,2442888212855745463,2519991190963396260,2543011661347658695,2548443174907492829,2620198259562492167,2696589776057000406,2777028857919354923,2826254048069073883,2853571739484670830,2871711248635430652,2931195526770142448,3106113464717560083],"entities":["open zetlyn corpus specification two","corpus","adapter","hub","zetlyn","licence the","part"]}
{"document_id":"ded24da2d452e1c8ad4ca98a01571fcd19ebc17b11df41acb43868276f2dfbb0","content_sha256":"c67d216c7bce37ebe1418600f58cf5eb8e0ccfcc0e904fdc2deb09e28d693038","hashed":{"address":"zetlyn://zetlyn/spec/spec/zetlyn-adapter-protocol-1.0.md","title":"Zetlyn Adapter Protocol","url":null,"language":"eng","published":"2026-09-14T12:40:31Z","keywords":"","text_chars":8658,"word_count":1461,"text_hash":"9285bcb9ca7167646bf381832aa1677aa07ca70162adc149a2f1e90a35a6ca64","catalog_version":"folder-2","embed_model":"multilingual-e5-small","extra":{"bytes":8971,"extension":"md"}},"embed_method":"chunked-256w-mean-v1","text":"Zetlyn Adapter Protocol\n\nVersion 1.0.\n\nAn adapter is a program that reads one kind of source and writes documents. It is a separate process,\nit may be written in any language, and it speaks this protocol on its standard output.\n\nThere is no built-in adapter. The ones shipped with Zetlyn are ordinary programs speaking this\nprotocol, so an adapter you write is not a second-class one.\n\nRunning an adapter\n\n<command> --zetlyn-interface            → prints the versions it supports, one per line\n<command> --zetlyn-interface 1 --config <file>\n\nThe host asks for the versions first and runs with the highest it shares. An adapter asked for a\nversion it does not support exits non-zero without writing records.\n\n--config names a TOML file the host writes. It holds the corpus's settings as the operator wrote\nthem, minus the keys the host uses to find and bound the corpus, plus two the host adds:\n\ncorpus = \"suspero/handbook\"       # the corpus being filled\ncursor = \"2026-09-10T22:14:03Z\"   # absent on a first run; see `mark`\npath   = \"/srv/handbook\"          # everything else is the adapter's own\n\nAn adapter therefore never sees where its own program lives or how large the corpus may grow. Those\nare the host's business.\n\ncorpus and cursor are reserved, and an adapter may not have settings of its own under those\nnames. The host writes them into the same table as everything else, so a setting called cursor\nwould be overwritten by the mark on the second run and would never be seen at all on the first — the\nadapter would simply behave as though the operator had not written it. Reserved rather than nested\nbecause the host has no business inventing a table inside a file the adapter owns, and because two\nnames are cheap to avoid: a setting that wants to name a column calls it cursorcolumn.\n\nThe adapter is run by hand with the same arguments. That is the intended way to debug one.\n\nThe stream\n\nOne JSON object per line on standard output. Standard error is the adapter's log: the host passes it\nthrough and never parses it.\n\nSix record types: hello, doc, delete, skip, mark, done. A host ignores a type it does not\nknow, which is what lets a minor version add one.\n\nhello — first record, exactly once\n\n{\"t\":\"hello\",\"interface\":1,\"adapter\":\"suspero/jira\",\"ends\":true,\"divisible\":false,\n \"address_space\":\"per-corpus\",\"indexing\":{\"mode\":\"full_text\"},\"catalog_version\":\"v1\"}\n\n| | |\n|---|---|\n| adapter | the name, namespaced by owner |\n| ends | whether this run terminates. false for a source that does not end, such as a feed or a crawl |\n| divisible | whether several processes can read this source at once, each taking a share. Absent means false |\n| addressspace | global or per-corpus |\n| indexing | {\"mode\":\"fulltext\"}, or {\"mode\":\"keywords\",\"terms\":15} |\n| catalogversion | names the extraction that produced text, and changes whenever that extraction can produce different text |\n\ndivisible is what lets a source be filled by more than one worker. A crawl can divide its\nfrontier; a directory cannot usefully — one process walks it faster than three coordinate over it. A\nhost asked to run a coordinated worker against an adapter that reports false refuses, rather than\nquietly running one worker and looking slow.\n\nAbsent means false, so an adapter written before this field existed keeps working and is never\ndivided by accident.\n\nA hello that contradicts the corpus is refused. Address space, indexing and catalogversion are\nrecorded in the manifest on the first run, and a later run declaring different ones is not a\nreconfiguration — it changes the identity or the meaning of every document already stored. The host\nstops before reading a single record and says which field moved. Changing any of them deliberately\nmeans building the corpus again from empty.\n\nThe adapter declares these and the host records them in the corpus manifest. They are properties of the\nadapter, not settings the host passes in.\n\nRetention and the read gate are not here. Whether a corpus evicts, and what a reader may be shown,\nare the operator's decisions about one corpus rather than facts about the adapter, so the host declares\nthem itself.\n\ndoc — a document, added or replaced\n\n{\"t\":\"doc\",\"address\":\"docs/readme.md\",\"title\":\"Getting started\",\"text\":\"…\",\n \"published\":\"2026-03-04\",\"language\":\"en\",\"url\":\"https://…\",\n \"extra\":{\"external_links\":42,\"https\":true}}\n\n| | | |\n|---|---|---|\n| address | required | unique within this corpus. Under global, unique everywhere |\n| text | required | the document as plain text. The adapter does the extraction |\n| title | optional | |\n| published | optional | ISO 8601 |\n| language | optional | detected by the host when absent |\n| url | optional | where a reader can see the original, which under corpus-local addressing is not the address |\n| extra | optional | the fields only this kind of source has |\n\nAn address already in the corpus replaces what is there.\n\nextra is part of the document's identity. The host hashes it with the rest of the record, keys\nsorted, so the same document produced twice has the same hash. That is also the constraint it puts on\nyou: whatever you put there has to be derived from the source rather than from the run, and a\ntimestamp of when you read it does not belong in it.\n\nA web adapter uses it for outbound links, the scheme, images, a paywall. A database adapter uses it for\nwhatever the row has and a web page does not. Nothing compares extra across corpora — the values that\ncross a boundary are in the corpus specification, and none of them is here.\n\ndelete — one document is gone\n\n{\"t\":\"delete\",\"address\":\"docs/old.md\"}\n\nskip — something was there and could not be read\n\n{\"t\":\"skip\",\"address\":\"handbook/scan.pdf\",\"reason\":\"no extractable text\"}\n\nThe source held it, the adapter reached it, and nothing usable came out. It is not a failure of the\nrun and not a deletion: the document is simply not in the corpus, and the host counts it and reports\nthe count when the run ends.\n\nWithout this an adapter has two options for a file it cannot read — pretend it does not exist, or fail\nthe whole run — and the first is how a corpus quietly loses a tenth of a directory.\n\nA skipped address is not removed from the corpus. If it was read successfully before, the older\ndocument stands until a delete says otherwise.\n\nmark — a resume point\n\n{\"t\":\"mark\",\"cursor\":\"2026-09-11T08:00:00Z\"}\n\nThe host stores the last cursor it saw and passes it back in the next run's config. What the value\nmeans is the adapter's business; the host only carries it.\n\nSend mark only after everything up to that point has been sent. A cursor is a high-water mark: a\nrun that marks first and writes afterwards, and then fails, resumes above documents it never emitted.\nThey are skipped for good, and the run that skipped them looks like an ordinary incremental one.\n\ndone — the run finished\n\n{\"t\":\"done\",\"complete\":true}\n\ncomplete: true states that every document the source holds was emitted in this run. Only then may\nthe host remove addresses it did not see.\n\nA run that ends without done, or with complete: false, removes nothing. The documents already\nwritten stay written.\n\nAn adapter with ends: false never sends complete: true, so deletion by absence never applies to\nit. It still sends done when it stops: exiting 0 in silence is indistinguishable from crashing\nwith a zero status, and the host has to read it as a failure. A source that does not end and is asked\nfor one pass therefore ends that pass with {\"t\":\"done\",\"complete\":false} — the run finished, and\nnothing may be removed.\n\nWhy completeness is stated rather than assumed\n\nDeletion by absence means \"what I did not see is gone\". An adapter that dies halfway through a source\nhas not seen the rest, and a host that trusts the run deletes half the corpus. It looks exactly like a\nclean run over a source that lost half its documents, so nothing reports an error.\n\ndone is what separates the two cases, and it is the reason a partial run is safe.\n\nExit\n\n| | |\n|---|---|\n| 0 after done | the run succeeded |\n| 0 without done | treated as a failure |\n| non-zero | the run failed. Documents already written stay; nothing is removed |\n\nAn adapter that cannot reach its source exits non-zero. It does not emit an empty complete run, which\nwould be an instruction to delete everything.\n\nVersioning\n\nMAJOR.MINOR. A minor version may add optional fields and new record types; an adapter written against\n1.0 keeps working, and a host ignores record types it does not know. Anything that removes a field,\nchanges a meaning, or makes an optional field required is a major version, and the host negotiates it\nby asking which versions the adapter supports.","simhash":4835488389509643701,"bands":[1857850309739804,8163400483544081,18155767180662041,22493186018193977,35473505096303860,45800551876523816,50154445511434273,56435813079478656,62341015447558353,92414658583121519,100848201012322065,104416201700060146,143171868632920057,166904939273750444,204709218142176317,224411360376251007,268105674455747787,283626358438926306,291055094922327877,291499094183892765,291894013925254971,318489119749762706,330839816616930986,337081405029180649,362990213542667750,370238165086632080,380050163938308082,384133842071165153,384259347162849675,385270851759292077,399799612889653662,410057090401153123,411282861964856779,412118919858414436,418239221756760201,434347406365415502,444796263721377531,445336543953602979,448458692867350611,452210829023099173,459561778526306195,471055357441333346,474582313590739091,507820647090566412,512157264411603691,513811763640486055,524712731202573479,534454648102553963,558998874299487526,559814779610412760,568041657480986645,579170738004272188,583691030668294262,590273181295907260,613954358867808409,614804259995586243,617082064590146776,631401639150271886,638309392731865661,674135823242117833,692532185849406677,704753555879476472,726339270684271114,734051949052662522],"entities":["zetlyn adapter protocol version","zetlyn","toml","2026-09-10t22:14:03z","json","t\":\"doc\",\"address\":\"docs/readme.md\",\"title\":\"getting","iso","t\":\"mark\",\"cursor\":\"2026-09-11t08:00:00z","deletion","major.minor"]}
{"document_id":"d2cd58ea98193d84510b10fcf988a72ffa71a48dde11af0310c3178f876f87e5","content_sha256":"4dddef803472da0ca7c3321fdc846aff87c707f0e513a98d25336ab3ce9fc20e","hashed":{"address":"zetlyn://zetlyn/spec/spec/zetlyn-corpus-spec-1.0.md","title":"Open Zetlyn Corpus Specification","url":null,"language":"eng","published":"2026-09-14T13:26:01Z","keywords":"","text_chars":63204,"word_count":10277,"text_hash":"53f2af28f7e1d8bb148ffdd0db2693d2a6accd5bd5caa07b8e468a60e25f93cf","catalog_version":"folder-2","embed_model":"multilingual-e5-small","extra":{"bytes":64955,"extension":"md"}},"embed_method":"chunked-256w-mean-v1","text":"Open Zetlyn Corpus Specification\n\nVersion 1.0.\n\nThe rule the rest follows from\n\nA quantity may cross a corpus boundary only if it is a fact about a document. A quantity derived over\nthe corpus's own membership may not.\n\nA term's inverse document frequency is a fact about the corpus holding it, not about the document. Two\ncorpora produce two different values, and comparing them gives a number that looks ordinary and means\nnothing. The same goes for a publisher label computed from whichever domains a corpus happens to hold,\nand for a quality score fitted to one corpus's contents.\n\nFor each quantity that fails the rule, the specification names a fact about a document that carries the\nsame meaning. Section 2 has the cases.\n\nConformance is decided at three levels, specified in section 12.\n\nConventions\n\nMUST, MUST NOT, SHOULD, SHOULD NOT and MAY are read as in RFC 2119 and RFC 8174, and appear in capitals\nonly where they are normative.\n\nRules carry the reason they exist. Most of them come from something else being tried first and failing\nwithout an error message.\n\nA hash over a named list\n\nThree values in this specification identify a set of named things rather than one run of bytes: the\nversion of a corpus, its rule set, and its display rules. All three are computed the same way.\n\nH(entries) = SHA-256 of the concatenation, over entries sorted by name as UTF-8 bytes, of\n             name  LF  value  LF\n\nLowercase hex. Names are unique within a list, and an empty list hashes to the SHA-256 of the empty\nstring, e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.\n\nThe sort is over bytes and not over a locale's collation, and the separator is a byte that cannot\noccur in a name, so ab + c and a + bc cannot reach the same digest. A construction that\nconcatenated without one has been the source of this class of defect for as long as the class has\nexisted.\n\nContents\n\n| | |\n|---|---|\n| 1 · Scope and terms | corpus, adapter, scope, document, publisher, and what this document does not cover |\n| 2 · The comparability rule | fact-about-a-document versus derivation-over-membership, and why it decides the rest |\n| 3 · Identity and address space | what names a document, and why two corpora cannot collide |\n| 4 · The document record | required and optional fields; which of them the content hash covers |\n| 5 · Declared countables | extraction, embedding (model and method), fingerprints, canonical entities — the declarations that make comparison legal |\n| 6 · The corpus manifest | id, version, owner, licence, provenance, rule set, and the comparability declarations |\n| 7 · Index artifacts | what a corpus publishes so others can search it: lexical index, vector column, entity postings and their frequency table |\n| 8 · Relations | the rule set, rule ids and versions, what is stored and what is derived at read time |\n| 9 · Permissions and rights | the read gate, attribution, and what a subscriber owes a publisher |\n| 10 · Composition | searching several corpora as one: fusion, relevance magnitude, cross-corpus publisher identity, cross-corpus relations |\n| 11 · Distribution | packaging, pointers, deltas, integrity, update cadence |\n| 12 · Conformance | the three levels, and how a claim is made |\n| 13 · Versioning | what is a breaking change, and what a corpus declares |\n| A · Schemas | the normative JSON schemas |\n| B · Rule registry | every relation rule id with its versions |\n| C · Reserved names | corpus names and manifest keys that are not yours to use |\n\n1 · Scope and terms\n\nTerms\n\nDocument. One addressable unit of text with the fields of section 4. A web page, a wiki page, a\nfile, a database row.\n\nAdapter. A data form together with what rules apply to it: the open web, a news feed, a file\nshare, a database, a wiki.\n\nThe name is free and is namespaced by its owner, suspero/wiki. What a reader acts on is not the name\nbut three properties the adapter declares: the address space (section 3), what a document is findable\nby, and the extraction that produced its text. Two corpora naming the same adapter and declaring\ndifferent properties are two different things, and only the properties are consulted.\n\nRetention and the read gate are not the adapter's. Whether a corpus evicts, and what a reader may\nbe shown, are the operator's decisions about this corpus — the same adapter serves one corpus that\nevicts and another that does not. They are declared by the corpus, in the manifest.\n\nCorpus. One configured instance of an adapter, with a store of its own and a name. One adapter can\nserve several: two customers under folder are two corpora.\n\nScope. A named set of corpora searched as one, chosen per query rather than per deployment.\n\nPublisher. Who a document comes from. Inside a corpus this is a cluster of domains under one\noperator; across corpora, section 10.\n\nCountable. A number derived from a document that another party compares against: a fingerprint, a\nfrequency, a hash. Section 5.\n\nRelation. A stated connection between two documents, produced by a named rule at a named version.\nSection 8.\n\nWhat this specifies\n\nThe contents of a corpus, what it declares about itself, and what has to hold for two corpora built by\nunrelated parties to be searched together.\n\nWhat it does not\n\nHow documents are ranked inside one corpus. Which documents a corpus keeps and which it evicts. How a\nreading route is planned over the results. How a crawler is paced. None of these cross a corpus\nboundary, and a corpus that does them differently is still a corpus.\n\nNothing here requires a particular storage engine, index structure or programming language. Where a\nparameter is fixed, it is fixed because two parties have to agree on it, not because one implementation\nhappens to use it.\n\n2 · The comparability rule\n\nA quantity may cross a corpus boundary only if it is a fact about a document. A quantity derived over\nthe corpus's own membership may not.\n\nA derived quantity is computed against whatever documents a corpus happens to hold. Two corpora\nholding different documents produce different values for the same input, and comparing them gives a\nnumber in the right range with no meaning behind it. Nothing reports an error.\n\nThe cases below are the ones that matter for search. Each has a replacement that is a fact about a\ndocument.\n\nRetrieval score\n\nA BM25 score depends on the corpus's inverse document frequencies and average document length. A term\nthat is rare in a 400,000-page web crawl is ordinary in a standards corpus, so the two scores are\ndifferent scales.\n\nReplacement: rank. A composer combines results by position, never by raw score. Section 10.\n\nPublisher label\n\nA publisher is a cluster of domains under one operator, and the cluster is labelled by one of its\nmembers. Which member depends on which domains that corpus holds, so two corpora holding the same\npublisher can label it two ways.\n\nReplacement: the set of registered domains (eTLD+1) the cluster covers. A registered domain is a fact\nabout a page. Two clusters from different corpora are one publisher when their sets intersect.\n\nEntity distinctiveness\n\nWhether an entity is distinctive depends on its document frequency in the corpus.\n\nReplacement: the published frequency table, and an absolute cap that makes the threshold nearly\nindependent of corpus size. Section 5.\n\nQuality\n\nAny score fitted to one corpus's contents — what earns a slot, what is evicted, how a page ranks\nagainst its genre — is a derivation over membership, and there is no replacement for it.\n\nQuality does not cross. A corpus's internal ordering is its own, and section 10 combines corpora\nwithout reading it.\n\n3 · Identity and address space\n\nA document is identified by one string, its address. Edges, permissions, route steps and moderation\ndecisions all attach to the hash of that address. Two documents with the same address are one document\neverywhere.\n\nAddress spaces\n\nAn adapter declares one of two, and the choice belongs to the adapter, not to a corpus.\n\n| | |\n|---|---|\n| global | A native address names one document in the world. The same address in two corpora is the same document. |\n| per-corpus | A native address is unique only inside one corpus. |\n\nThe open web is global. A URL identifies a page on the internet, so two corpora that both hold it\nhave to collide.\n\nA file share is per-corpus. Two companies both hold /docs/readme.md and those are two documents.\nDeclared global instead, they become one: one set of edges, one permission decision, one takedown,\napplied to both companies. Nothing reports an error.\n\nForming an address\n\nglobal      → the native address, byte for byte\nper-corpus  → zetlyn://<owner>/<name>/<native>\n\nAn implementation refuses rather than substitutes when the native address is empty, when it already\nbegins with zetlyn://, or when the corpus name is invalid.\n\nA global address that is a URL is written in the normal form below. Byte for byte is what makes\ntwo corpora collide, and two adapters written by two people do not produce the same bytes by\naccident: https://example.org/a, HTTPS://Example.org:443/a and https://example.org/a#top are\none page and would be three documents, with no edge between them and nothing to report it.\n\n| | |\n|---|---|\n| scheme | lowercased. Only http and https; anything else is not an address |\n| userinfo | removed. It is not part of what is being named, and publishing the corpus would publish it |\n| host | lowercased, a trailing dot removed |\n| port | removed when it is the scheme's default |\n| path | . and .. resolved; an empty path is / |\n| query | kept as written; a ? with nothing after it is removed |\n| fragment | removed. It is the reader's position in a document, not a document |\n\nFour things are deliberately not normalised, because each would merge documents that are not the\nsame on some real server: the case of the path, a trailing slash, the order of query parameters, and\ntracking parameters. The last is the tempting one — a list of parameters to strip is a maintained\nlist, two implementations drift apart, and utmsource is a routing decision on somebody else's\nserver rather than a fact about the address. What puts those copies back together is the\nnear-duplicate relation of section 8, from the text, where the evidence is.\n\nScoping an already-scoped address produces a second identity for one document. Both ids resolve and\nneither knows about the other.\n\nzetlyn:// is not dereferenced. It marks which space the address belongs to, and the corpus name and\nthe native address are recovered by taking the two segments after the prefix; everything after them\nis native.\n\nTwo, fixed, and not \"up to the last slash\". A native address carries slashes of its own, so any rule\nthat searched for one would cut in a different place for every document.\n\nCorpus names\n\nA corpus name is owner/name. Both parts are non-empty and at most 64 bytes. The name holds a-z,\n0-9, .,  and -; the owner holds the same minus the dot, so that a hub reference can tell an\nowner from a host by looking for one.\n\nThe owner is a namespace, so two publishers can both offer a handbook without collision. An\nunqualified name is refused rather than resolved against a default owner: a bare handbook would mean\nsomething different on every deployment that accepted it.\n\nNo uppercase, so one corpus cannot be written two ways. Two corpora in one deployment must not share a\nname.\n\nThe document id\n\ndocument_id = SHA-256(address as UTF-8), lowercase hex\n\nThe address is hashed unchanged, so a global corpus keeps its ids when it is given a name.\n\n4 · The document record\n\nThe content hash\n\nEvery record carries contentsha256: SHA-256, lowercase hex, of the JSON serialization of an object\nwith these keys in this order, no insignificant whitespace, absent optional values as null.\n\naddress  title  url  language  published  keywords\ntext_chars  word_count  text_hash  catalog_version  embed_model  extra\n\nThe order is normative. A reader holding the document has to be able to recompute the hash and get the\nsame value.\n\nEleven of those are true of any document, whatever it came from: where it is, what it is called,\nwhat language it is in, when it was published, how long it is, what text it holds, and which extraction\nand model produced the derived values. url is optional and is where a reader can see the original —\nunder corpus-local addressing that is not the address.\n\nextra — what only this adapter knows\n\nThe twelfth key is an object the adapter supplies, and it is hashed with the rest.\n\n\"extra\": {\"external_links\": 42, \"https\": true, \"image_count\": 3, \"paywalled\": false}\n\nA web page has outbound links, a scheme, images, a paywall and structured-data types. A row in an\norders table has none of those and has an order status instead. An earlier draft of this specification\nput the web's fields in the record itself, so a folder of files carried paywalled: false and\nimagecount: 0 — inside the content hash, which made a statement about images part of a text file's\nidentity.\n\nIts keys are serialized sorted, so two producers of the same content reach the same hash.\n\nNothing compares extra across corpora. It is hashed because it is part of what the document is, not\nbecause anybody reasons over it: section 5 holds the values that cross a boundary, and none of them is\nhere.\n\nWhat is outside the hash\n\nThe embedding vector, the simhash, the shingle bands, embedmethod, and any label a later model\nattaches.\n\nAll of them derive from the text, and the text is already pinned by texthash. Putting a derivation\nin the hash marks every stored document as changed on the day the derivation is added, which means\nre-fetching a corpus to record something that did not change.\n\nThe hash identifies a document. It does not certify that two documents' derived values can be compared;\nthat is what the declarations of section 5 are for.\n\nThe fields\n\naddress is the identity of section 3. texthash is the hash of the extracted main text and is what\nmakes two documents exact duplicates. catalogversion names the extraction that produced the text and\nthe counts beside it. embedmodel names the model that produced the vector.\n\nkeywords is what a keyword-indexed corpus is findable by, beside its title. It is the document's own\nmost frequent terms, and it is a field rather than a derivation the reader performs:\n\nSplit the extracted text on whitespace and lowercase each token.\nTrim from each end every character that is neither a letter nor a digit. Drop what is left empty.\nDrop every token the stop list of section 5 holds for this document's language. A document whose\nlanguage that list does not cover keeps every token it has.\nCount. Keep the adapter.indexing.terms highest counts, ties broken by the token ascending as\nUTF-8 bytes.\nJoin them with single spaces, in that order.\n\nUnder fulltext the field is the empty string: the whole text is indexed, so no subset of it is the\ndocument's key.\n\nThe terms come from the document and from nothing else. A weighting against the corpus — every form of\ntf-idf — is a derivation over the corpus's own membership, and section 2 keeps those out of anything\nthat crosses a boundary. This one is inside the content hash, which is the strongest form of that rule.\n\nStep 3 is the exception that proves the rule, and it is why the stop list is pinned and named rather\nthan left to the builder. Without it the most frequent terms of a prose document are its function\nwords: a corpus findable by fifteen terms spends nine of them on the and in is a to of at, which is\nmost of the index of a feed or a crawl. The list is not a weighting against the corpus — it is a fact\nabout a language, it is the same for every corpus in that language, and it travels by name.\n\nA change in any field is a change in what the document is, so a corpus whose extraction improves\nproduces new hashes. That is what catalogversion is for.\n\nembedmethod\n\nEvery record carries embedmethod beside embedmodel, outside the hash.\n\nThe model does not determine the vector. The same model applied to a page's whole text under a\n512-token limit, and applied to 256-word chunks with the mean taken, produce different vectors.\nMeasured over one corpus, 64.7% of pages were truncated, at a median of 394 words.\n\nTwo vectors whose records name different embedmethod values are not comparable, and a reader\ntreats them as such. The field is per record rather than per corpus because a corpus built over\nmonths can change method partway, and a single declaration in its manifest would then be false for\npart of it.\n\n5 · Declared countables\n\nA countable is a number derived from a document that another party compares against. The comparison\nonly holds if both sides computed it the same way, so a corpus declares the parameters of every\ncountable it carries. Countables computed under different parameters are not compared.\n\nExtraction\n\ncatalogversion names which text is the document's main text, what is dropped as navigation or\nboilerplate, and how words and characters are counted. Two corpora on different catalog versions hold\ndifferent text for the same page.\n\nThe identifier is fixed here, and so is the requirement that it change whenever the extracted text can\nchange. The implementation is not.\n\nEmbedding\n\nDeclared: model identifier, method identifier, dimension, and whether vectors are normalized.\n\nThe reference values are multilingual-e5-small and chunked-256w-mean-v1: the text split into\n256-word chunks, each embedded, the mean taken. Section 4 covers why the method is declared.\n\nSimHash\n\n64 bits, over word trigrams.\n\nSplit the extracted text on whitespace, lowercase each token.\nFor each 3-token window, take the FNV-1a 64-bit hash of the window joined by single spaces.\nEach hash casts a ±1 vote per bit position. A bit is set when its sum is positive.\nUnder 3 tokens, hash the tokens individually. Empty text gives 0, and 0 never matches.\n\nTrigrams rather than single words. Unigram voting is dominated by function words, and two unrelated\nprose pages in the same language come out close.\n\nNear-duplicate: Hamming distance of 3 or less, and the two are not exact duplicates. An exact duplicate\nis reported as one.\n\nThe threshold holds for documents, not for fragments. Replacing one token at the middle of a document\nand measuring the distance: at 16 tokens it moved 18 bits, at 32 twelve, at 64 seven, at 128 three, at\n512 one. A short document has too few trigrams for a vote to be deep, so an edit flips bits a longer\none absorbs, and two short near-duplicates are not reported as near.\n\nThis is a property of the countable rather than a defect in it, and it is stated because the opposite\nbelief is the natural one. A corpus of short records — titles, rows, log lines — gets nothing from\nnearduplicate and should not be read as having none.\n\nShingle bands\n\nShingle length 12 tokens. FNV-1a 64-bit over the shingle joined by single spaces. A document keeps the\n64 smallest values.\n\nTwo documents share a band value exactly when they share that 12-token run, so an overlap counts\nverbatim spans instead of estimating similarity.\n\nQuotation requires 2 shared bands. One shared 12-gram can be a boilerplate line.\n\nFNV-1a is named rather than left open. A language's default hasher is usually not stable across\nreleases, and an unstable hash stops fingerprints agreeing without reporting anything.\n\nEntity distinctiveness\n\nAn entity is distinctive in a corpus when its document frequency is at most min(⌈√n⌉, 32), for n\ndocuments.\n\nAbove about 1000 documents the threshold is 32 whatever the corpus size, so the gate barely depends on\nthe corpus at all.\n\nFrequency over two corpora is the sum of the two, and neither side can derive the other's, so a corpus\npublishes its entity frequency table for its entities to take part in cross-corpus relations.\n\nThe entity vocabulary\n\nThe vocabulary is not declared. It is fixed here, like the fingerprint parameters above, and it is\nnamed zetlyn-entities-1:\n\n| | |\n|---|---|\n| recognizer | a named model, pinned by content hash |\n| noise filter | which spans are dropped before anything else |\n| cap | at most 512 distinct entities per document, in document order, keeping the first mention |\n\nAn entity is the recognizer's span, lowercased, and nothing is mapped onto anything else. Version 1\nhas no alias table: one has to be derived from documents rather than written from memory, and merging\ntwo things that are not the same would do it in every conforming corpus at once.\n\nThe contract is vocabulary/zetlyn-entities-1/ in this repository, and the artifact is published\nat <https://download.zetlyn.com/zetlyn-entities-1.json>, with its\nSHA-256 stated there and here:\n\n7e6732e23d92fd8b496d04c40a6b7ac005dafa8c622167c82bb6bffcb1b786df\n\nThe hash is what a corpus is conforming against, not the address — a corpus built from different\nbytes under this name is not conforming, wherever they came from, and one built from these bytes is,\nhowever it obtained them. The address is stated so that there is somewhere to get them.\n\nThe cap is not a haircut. One long document can yield tens of thousands of spans, and without a bound\na single file decides a corpus's entity graph; keeping the first mentions in document order rather\nthan a set's iteration order is what makes the surviving entities reproducible.\n\nWhy this one is a constant and not a declaration\n\nA declaration lets two parties canonicalize differently and find out afterwards. That looked\nacceptable until the relation the routes actually use was counted: over 24 questions and 192 route\nsteps, 96 steps rested on a relation to an earlier step, and 89 of those 96 were the\nentity-mediated one. Citations, which need no shared vocabulary at all, accounted for three.\n\nSo the relation that carries almost all of the route's structure is exactly the one a free choice of\nrecognizer would break, and it would break silently — two corpora would simply share fewer distinctive\nentities than they should, and nothing would report it.\n\nPinning the vocabulary has a price, and it is worth saying plainly: the format is tied to a specific\nmodel, and a model has a lifetime. That is what the version in the name is for. Two corpora on\ndifferent vocabulary versions do not derive entity relations between them, which is the same rule that\ngoverns every other parameter here.\n\nThe stop list\n\nAlso fixed here, for the same reason and by the same construction, and named zetlyn-stopwords-1. It\nis what step 3 of keywords drops, and it is a set per language, not one list:\n\nlanguages   deu  eng\nsha256      8a7de7d08415184f00d76c06ccc1072fde13905ac24087d8a54cd51544653ae7\n\nThe hash is the named-list hash of the Conventions, over one entry per language whose value is that\nlanguage's words joined by single spaces, in the order they appear below. A corpus computing\nkeywords against different words under this name is not conforming; the words are printed in full\nso that there is nothing to obtain.\n\nIt applies to a document whose language it covers and to no other. A list is a fact about one\nlanguage, and French or on son a car are all English function words carrying their own meaning — a\ndocument in a language this does not name keeps every token it has. language is already inside the\ncontent hash, so nothing new is pinned by depending on it.\n\nThere is no length floor. It is the cheap way to write a stop list and it is wrong: ai, go, eu\nand 5g are among the terms a technical or a news corpus most needs to be findable by.\n\neng\n\na about after again against all also am an and any are as at be because been before being between\nboth but by can could did do does doing down during each few for from further had has have having\nhe her here hers herself him himself his how i if in into is it its itself just me more most my\nmyself no nor not now of off on once only or other our ours ourselves out over own same she should\nso some such than that the their theirs them themselves then there these they this those through\nto too under until up very was we were what when where which while who whom why will with would\nyou your yours yourself yourselves\n\ndeu\n\naber alle allem allen aller alles als also am an auch auf aus bei bin bis bist da damit dann das\ndass dazu dein deine dem den denn der des dessen dich die dies diese diesem diesen dieser dieses\ndir doch dort du durch ein eine einem einen einer eines er es etwas euch euer für gegen gewesen\nhab habe haben hat hatte hatten hier hin hinter ich ihm ihn ihnen ihr ihre ihrem ihren ihrer ihres\nim in indem ins ist jede jedem jeden jeder jedes jene jenem jenen jener jenes jetzt kann kein\nkeine keinem keinen keiner keines können machen man manche mein meine mich mir mit muss musste\nmüssen nach nicht nichts noch nun nur ob oder ohne schon sehr sein seine seinem seinen seiner\nseines selbst sich sie sind so solche soll sollte sondern sonst um und uns unser unsere unter viel\nvom von vor war waren warst was weg weil weiter welche wenn werde werden wie wieder wir wird wirst\nwo wollen wollte während würde würden zu zum zur zwar zwischen über\n\nA second version is a second vocabulary. Adding a language moves the content hash of every\nkeyword-indexed document built with the first, and the two are then not comparable — which is the\nmechanism of section 2 working rather than a cost of using it.\n\n6 · The corpus manifest\n\nOne JSON object per published version of a corpus. It travels inside the artifact, not beside it, so a\nmanifest that arrives by a copied key or a mis-synced bucket still says what the payloads are.\n\nIdentity and build\n\n| | |\n|---|---|\n| specversion | the version of this specification the corpus was built against |\n| corpus | the name, per section 3 |\n| version | content hash of the payloads this manifest covers |\n| base | on a delta, the full version it extends. A delta applied to a different base reintroduces documents the newer build dropped |\n| builtat | unix seconds, supplied by the builder and never read from a clock, so a build is reproducible |\n| documents, relations | counts |\n| rulesetid | section 8 |\n\nAppendix A has the schema.\n\nAdapter\n\n| | |\n|---|---|\n| adapter.name | free, namespaced by owner: suspero/wiki |\n| adapter.addressspace | global or per-corpus |\n| adapter.indexing | keywords with a term count, or fulltext |\n\nA reader acts on the properties. The name is documentation.\n\nThe adapter declares these, and countables.catalogversion below, when it runs. Nothing else in the\nmanifest comes from it.\n\nPolicy\n\n| | |\n|---|---|\n| retention.evicts | whether the corpus drops documents to stay within a size, and on what basis |\n| readgate.default | open or closed: what may be shown for a source with no rule |\n| readgate.rulesversion | the content hash of the display rules in force, section 9 |\n\nThese are the operator's, not the adapter's. One adapter serves a corpus that evicts and another that\nkeeps everything.\n\nindexing is not a tuning choice. A web-shaped corpus indexes the title and the most frequent terms\nbecause recall is cheap and precision is the problem. A private corpus indexes the whole text, because\nthe caller knows the word is in section 7.3 and not finding it is not a worse ranking.\n\nPayloads\n\n\"payloads\": {\n  \"documents.jsonl.zst\": {\"bytes\": 48211904, \"sha256\": \"…\"},\n  \"vectors.f32\":         {\"bytes\": 79200000, \"sha256\": \"…\"}\n}\n\nEvery file this version consists of, with its size and its hash. A consumer fetches what the manifest\nlists and checks each file against it; anything else served under this version is not part of it.\n\nWithout this list, \"verify the payloads against the manifest\" has nothing to verify against — which is\nwhat section 11 and the hub layout both assume it does.\n\nCountables\n\nThe declarations of section 5, verbatim, so a consumer can decide comparability without fetching a\ndocument.\n\ncountables.catalog_version\ncountables.embed.model\ncountables.embed.method          ← may be absent when records disagree; see section 4\ncountables.embed.dimension\ncountables.embed.normalized\ncountables.simhash.bits\ncountables.simhash.shingle_tokens\ncountables.bands.shingle_tokens\ncountables.bands.hash\ncountables.bands.keep\ncountables.entities             ← \"zetlyn-entities-1\"\ncountables.stopwords.name       ← \"zetlyn-stopwords-1\"\ncountables.stopwords.sha256\ncountables.stopwords.languages\n\nembed.method in the manifest is a summary. The record's field decides, because a corpus built over\nmonths can hold both.\n\nDistribution — L3 only\n\n| | |\n|---|---|\n| owner | who published it, and where to reach them |\n| licence | what a subscriber may do with the documents, and with answers derived from them |\n| attribution | what has to appear wherever a document's title or text is shown |\n| provenance | per document: where it came from and when it was captured |\n\nA corpus that omits these is L1 or L2 and is not offered to others. Each of them is something the\nrecipient cannot establish alone.\n\n7 · Index artifacts\n\nA published corpus ships the documents and the structures needed to search them. A consumer that had\nto rebuild an index would be re-indexing, which is what this specification exists to avoid.\n\nThe payloads\n\n| file | format | |\n|---|---|---|\n| documents.jsonl | one record per line | the records of section 4 |\n| relations.jsonl | one relation per line | the stored relations of section 8 |\n| vectors.f32 | raw little-endian IEEE 754 binary32 | documents × dimension values, no header |\n| entities.jsonl | one entity per line | the postings: which documents mention it |\n| frequencies.jsonl | one entity per line | entity to document count |\n| publishers.jsonl | one cluster per line | its registered domains, for section 10 |\n| rights.jsonl | one source rule per line | section 9 |\n| statistics.json | one object | the document count and the term statistics the lexical index was built from |\n\nA payload may be zstd-compressed, in which case its name carries .zst. The manifest lists the names\nas published.\n\nDocuments are numbered by their position in documents.jsonl, counting from zero, and every other\npayload refers to them by that number rather than by id. Row i of the vector column is document i;\na posting names 3, 41, 900; a relation names two positions. A version never changes, so a position\ninside one is stable, and an integer is an order of magnitude smaller than a 64-character hash repeated\nacross millions of postings.\n\n{\"entity\":\"cve-2026-1188\",\"documents\":[3,41,900]}\n{\"entity\":\"cve-2026-1188\",\"documents\":3}\n\nA frequency is the length of its posting list, and both are published anyway: a consumer computing\na distinctiveness threshold over several corpora needs the counts and not the postings, and the counts\nare a rounding error beside them. That they must agree is a conformance check, not a convention.\n\nVectors are a payload of their own rather than a field on each record. A consumer that ranks lexically\nand never embeds does not pay for them, and a consumer that only needs the sample of section 10 fetches\none payload instead of every record.\n\nAddressing and integrity\n\nPayload keys carry the version, and the version is the content hash of the payloads. Two corpora reach\nthe same key only by holding the same bytes, so a key is unambiguous across every corpus there will be.\n\nversion = H({ payload name : SHA-256 of that payload's bytes as published })\n\nThe named-list hash of the Conventions, over exactly the entries the manifest's payloads object\nlists. The bytes are the ones served: a payload published compressed is hashed compressed, because\nthat is what a consumer holds and what it can check without unpacking anything.\n\nThe manifest is not one of the entries. It carries the version, so it cannot be inside it.\n\nPointers are different. A name whose value changes — which version is current, which is being\nprepared — must be namespaced per corpus, or two publishers into one store overwrite each other's idea\nof what is current.\n\nThe manifest travels inside the artifact. A prefix is a naming convention; the manifest is what a\nreader checks, and it refuses a payload whose manifest names another corpus.\n\nLexical index\n\nWhat a document is findable by follows adapter.indexing from section 6: title and the most frequent\nterms, or the whole extracted text.\n\nThe index structure is not specified. What is specified is that the corpus publishes the statistics its\nscores were derived from, so a consumer can see that those scores are corpus-relative and must not be\ncompared across corpora.\n\nVector index\n\nNot published. An approximate-nearest-neighbour structure is built from the vector column by whoever\nserves the corpus, and a consumer that chunks documents differently needs a different structure over\nthe same vectors.\n\nWhat must hold is the correspondence: position i in the vector column is the i-th document in the\ndocument payload. A consumer takes a deterministic sample by position, which is what section 10\nrequires.\n\n8 · Relations\n\nA relation is a connection between two documents, produced by a named rule at a named version. Every\nrelation carries the rule id, the rule version, and a support value: the countable that fired it.\n\nRules compare hashes, counts, string identity or a Hamming distance. None of them reads a distance out\nof embedding space. A relation is a measurement of form, not of meaning: quotes says a verbatim run\nappears on both pages, not that one endorses the other.\n\nThe rule set\n\n| id | version | fires when | support |\n|---|---|---|---|\n| duplicateof/texthash | 1 | the extracted text hashes are identical | — |\n| nearduplicate/simhash | 1 | SimHash Hamming distance ≤ 3, and not an exact duplicate | the distance |\n| quotes/minhashshingle | 1 | ≥ 2 shared shingle bands, different publishers, and not whole-page reuse | shared bands |\n| cites/outlink | 1 | an outlink resolves to a live document of the corpus, different publishers | — |\n| citesdomain/rollup | 1 | the same, rolled up to the registered domain | — |\n| mentions/nercanonical | 2 | the document names the entity | — |\n| sharesentity/distinctive | 3 | both name an entity with document frequency between 2 and min(⌈√n⌉, 32), different publishers | the frequency |\n| obsoletes/rfc-index | 1 | a publisher's own index declares it | — |\n| updates/rfc-index | 1 | the same | — |\n\nduplicateof is emitted regardless of publisher. The others require two different publishers: an\nintra-publisher link is navigation, and intra-publisher reuse is syndication, which duplicateof and\nnearduplicate already state more precisely.\n\nobsoletes and updates are not computed from either document. They are read from an index the\npublisher maintains, and the edge id carries that index's content hash.\n\nWhat a stored relation looks like\n\nOne JSON object per line of relations.jsonl, naming two document positions in the sense of\nsection 7.\n\n{\"from\":3,\"to\":41,\"rule\":\"quotes/minhash_shingle\",\"version\":1,\"support\":{\"shared_bands\":4,\"of\":64}}\n\nsupport is the countable that fired the rule, and it is absent for the rules that have none. The\npair is written once, with from the lower position: these relations are symmetric, and writing both\ndirections would double the payload to say the same thing twice.\n\nmentions is the exception and is not in this file. It relates a document to an entity rather than\nto a document, so it has no second position to name — it is stored as the entity postings of\nsection 7, which is the same fact in the form a reader needs it. A corpus ships it in\nentities.jsonl and nowhere else.\n\nStored, and derived at read time\n\nsharesentity is not stored. Its predicate is corpus-relative, so a reader recomputes it from the\ncurrent entity postings and never trusts a stored value. What a corpus ships instead is the postings\nand the frequency table.\n\nEvery other rule above produces a stored edge.\n\nThe consequence for a publisher: an entity-mediated relation costs nothing at build time, and a corpus\nthat omits its frequency table simply does not take part in one.\n\nRule set identity\n\nA corpus declares rulesetid: a content hash over the rule ids and versions it was built with. A\nreader serving an explanation quotes it, so the explanation travels with the data rather than being\nread from a rule set the reader may no longer have.\n\nrule_set_id = H({ rule id : its version in decimal })\n\nThe named-list hash of the Conventions. A corpus built with no rules at all — one that stores no\nrelations — has the empty list's digest, which is a statement that no rule fired and not a missing\nfield.\n\nA version changes whenever the predicate changes, including a moved threshold. sharesentity is at\nversion 3 because its cap gained an absolute ceiling, which changed which pairs fire.\n\nAcross corpora\n\nA relation whose endpoints are in two corpora is derived at read time and never stored. Neither corpus\nknows about the other, and a stored cross-corpus edge would be a fact about a pairing that exists only\nin one composer's configuration.\n\nThe precondition is agreement. Most of what a relation rests on is fixed by section 5 and therefore\nthe same everywhere; what remains free is the extraction. A cross-corpus relation is derived only when\nboth corpora agree on what it rests on. Where they differ, the relation is not\nderived. It is not derived with a caveat, and it is not derived on the assumption that the difference\nis small.\n\n| rule | needs the same | and |\n|---|---|---|\n| duplicateof | catalogversion | — |\n| nearduplicate | catalogversion | the simhash parameters are fixed for everyone |\n| quotes | catalogversion | the band parameters are fixed; publishers differ, per section 10 |\n| sharesentity | the vocabulary version | fixed within one version of this specification; frequencies summed, see below |\n| cites, citesdomain | — | both corpora global |\n| obsoletes, updates | — | both corpora global |\n| mentions | — | never crosses; it relates a document to an entity |\n\nduplicateof compares text hashes, which are hashes of the extracted text, so two corpora on\ndifferent catalog versions hold different text and a match would be an accident.\n\nsharesentity is corpus-relative and so is its threshold. Over a set of corpora the frequency of an\nentity is the sum of its frequency in each, and n is the total document count of the set. Both come\nfrom the frequency tables of section 7, so the composer needs no document to compute them.\n\ncites requires an outlink from one corpus to resolve against the other's addresses. Under\nper-corpus addressing an address does not identify a document outside its own corpus, so the relation\nhas no meaning there. This is a boundary, not a missing feature.\n\nWhose rules fired. A derived cross-corpus relation carries the rulesetid of the composer that\nderived it, not of either corpus. A reader asking why it exists is asking about the composer.\n\n9 · Permissions and rights\n\nA corpus carries rules about what a reader may be shown. They are per source, not per document, and\nthey are declared rather than inferred.\n\nWhat a source rule decides\n\n| | |\n|---|---|\n| title | whether the publisher's headline may be shown |\n| summary | whether the publisher's teaser or an extract may be shown |\n| attribution | text that must appear wherever this source's material appears |\n| processing | whether the text may be sent to a generative model |\n\nOne JSON object per line of rights.jsonl, keyed by the source it governs.\n\n{\"source\":\"example.com\",\"title\":true,\"summary\":false,\"attribution\":\"© Example Ltd\",\"processing\":false}\n\nA source is the publisher identity of section 10 — the registered domain where a document carries\none, and the corpus's own name where it does not, because a corpus with no domains has one voice as\nfar as a rule is concerned. An absent key falls to the corpus's default, so a rule states what it\nchanges and nothing else. A document whose source has no line at all is governed by the default\nalone.\n\nA withheld title is blanked, not replaced. There is nothing to substitute, and a reconstructed label\nasserts something the publisher declined to release. A document that never had a title is a different\ncase: nothing is being withheld, and an identifier taken from its address may stand in.\n\nAttribution travels with the text rather than being looked up again by whatever renders it, so a\nconsumer cannot show the one without the other.\n\nThe posture of a corpus\n\nA corpus declares a default for sources it has no rule for. Two defaults are in use: open, where a\nsource with no rule may be shown in full, and closed, where it may not.\n\nThe default is the answer for most documents in most corpora, so declaring it is not a formality.\n\nRule version\n\nA corpus declares the content hash of the rule set in force. A served answer can then be matched\nagainst the rules that governed it, which is what makes a display decision checkable after the fact.\n\nrules_version = SHA-256 of the uncompressed bytes of rights.jsonl, lines in published order\n\nThe uncompressed bytes, so that republishing the same rules under a different compression does not\nread as a change of rules. A corpus with no rules declares the SHA-256 of the empty string.\n\nThe rules themselves travel with the corpus, as rights.jsonl in section 7. A hash alone would\ndeclare that rules exist and leave a subscriber unable to honour them — the failure would be a\nsubscriber showing what a publisher withheld, with a manifest that says the rules were in force.\n\nOver a scope\n\nThe rules of the corpus a document came from apply. Section 10.\n\nWhat a subscriber owes — L3\n\nA licence covering the documents, and answers derived from them. Attribution as the source requires.\nWhatever retention or deletion the publisher's terms impose. A withdrawal is a term of the licence, not\na mechanism of the format; section 11 has the reason.\n\n10 · Composition\n\nA composer answers one query over several corpora. It is not a corpus and does not conform at a level;\nwhat follows are the obligations a corpus's declarations place on anything that combines it with\nanother.\n\nScores do not cross, ranks do\n\nA composer must not compare raw retrieval scores from two corpora. Section 2 has the reason.\n\nCombination is by position in each corpus's own ordering.\n\nRelevance magnitude\n\nA rank says a document is first. It does not say whether it answers the question, so a corpus holding\nnothing relevant still contributes its best. A composer that only fuses ranks will fill an answer from\na corpus that has nothing to say.\n\nA composer therefore needs a magnitude that means the same thing in every corpus: a scale on which one\nend is a genuine match and the other is a document no better than one picked at random.\n\nThis specification does not fix how that is computed. It fixes what a corpus has to make possible:\n\nthe vectors of the candidates it returns for a query, and\na sample of its document vectors, drawn by a rule that gives the same sample for the same query\nand corpus.\n\nWithout the sample a consumer cannot establish what \"unrelated\" scores like inside that corpus, and\nthe magnitude cannot be derived from anything the corpus already returns.\n\nOne publisher, two corpora\n\nA publisher is a cluster of domains under one operator, and the cluster's label is computed from the\ndomains that corpus happens to hold. Two corpora can label the same publisher differently.\n\nClusters from different corpora are the same publisher when their sets of registered domains (eTLD+1)\nintersect. The relation is transitive: {a, b} in one corpus and {b, c} in another are one publisher\nspanning all three.\n\nA registered domain is a fact about a document. The cluster stays inside its corpus, where it is worth\nhaving, and only the domain sets cross.\n\nTwo limits, stated rather than discovered. A cluster that is wrong inside its corpus is wrong here too,\nover more domains. And a corpus whose documents have no domains — a file share, a database — supplies\nno publisher identity at all, so its documents are each their own voice unless the corpus declares\notherwise.\n\nThe read gate belongs to the document\n\nA document is shown under the rules of the corpus it came from, not the rules of the corpus the query\nhappened to start in. The gate travels with the candidate.\n\nA composer that applies one corpus's posture to another's documents produces a well-formed answer that\nviolates a publisher's terms, and reports nothing.\n\nRelations across corpora\n\nDerived at read time, never stored, and only where both corpora declare the same parameters for the\ncountable involved. Section 8 has the conditions per rule.\n\n11 · Distribution\n\nHow a corpus is handed to somebody else. L3 only; a corpus that is never distributed can ignore this\nsection.\n\nA version is a set of bytes\n\nA published version is the payloads of section 7 plus the manifest, addressed by the content hash of\nthe payloads. A version is never modified. A correction is a new version.\n\nWhat the manifest declares is the exception, and it has to be. Most of the manifest is derived\nfrom the payloads — documents, relations, rulesetid, adapter, countables, payloads and\nthe version itself — and a change to any of those is a different set of bytes and therefore a\ndifferent version. The rest is declared rather than derived:\n\nbuilt_at   retention   read_gate   owner   licence   attribution   cadence\n\nNone of them is in a payload, so the version does not cover them, and an operator or a publisher who\nchanges one cannot express it as a new version — the payloads are identical, so the hash is. Refusing\nthe change would mean a posture and a licence could only ever be set before the first build, or\nnever.\n\nSo those may be restated for a version that exists, and nothing else may. A subscriber that cached a\nmanifest and fetches it again may therefore see different terms under the same hash, which is a fact\nabout what a hash addresses and not a defect: it addresses the documents, and the terms are what\nsomebody says about handing them over.\n\nTwo builds from the same documents with the same parameters produce the same version. A builder that\nreads a clock, iterates a hash map, or embeds a path makes that false without reporting anything, so\nbuiltat is supplied rather than read and orderings are document order.\n\nOn one machine, with one build of the producer. The vector column is floating point, and a\nfloating-point reduction split across threads is summed in whatever order the threads finish in — so\ninference has to be pinned to a fixed threading before the sentence above is true at all. Across\narchitectures it may stay false however it is pinned: the same model on two CPUs can differ in the\nlast bit, and a version is the hash of the bytes.\n\nWhat this costs is worth naming. A subscriber can check that the bytes it fetched are the version the\nmanifest claims, which is what the check exists for. It cannot rebuild the corpus from the same\ndocuments on its own hardware and expect the same version, and a publisher who needs that guarantee\npublishes the bytes rather than the recipe.\n\nPointers\n\nA publisher maintains a small number of named pointers whose values change: which version subscribers\nshould use, and which one is being prepared. They are namespaced per corpus.\n\nA subscriber that follows a pointer gets a version. A subscriber that pins a version gets the same\nbytes forever.\n\nDeltas\n\nA delta carries the documents added, changed and removed since a named base, and states that base. A\ndelta applied to a different base reintroduces documents the newer build had already dropped.\n\nA publisher offering deltas still publishes full versions, because a subscriber joining later cannot\nreplay an unbounded chain.\n\nCadence\n\nA corpus declares how often it publishes and how long a version stays fetchable. A subscriber that\npins needs to know when the bytes go away; one that follows the pointer needs to know how stale it can\nbe.\n\nWithdrawal\n\nThere is no recall mechanism, and this is deliberate.\n\nA subscriber who has fetched a version holds the bytes. Nothing in the format can take them back, and a\nformat that claimed otherwise would be making a promise it cannot keep. What a publisher can do is stop\npublishing the document in later versions and state its terms in the licence, which is where an\nobligation to delete belongs.\n\n12 · Conformance\n\nA corpus conforms at a level. Each level buys a capability rather than marking maturity, and a\npublisher can stop at any of them.\n\nWhat the suite does\n\nIt recomputes what a corpus claims about itself, from the material the corpus ships.\n\n| | checked by |\n|---|---|\n| documentid | hashing the address again |\n| contentsha256 | serializing the fields of section 4 in order and hashing them again |\n| simhash, shingle bands | deriving them from the extracted text again |\n| entities | applying the declared canonicalisation and cap to the document again |\n| entity frequencies | summing the postings |\n| vectors | count against documents, length against the declared dimension, position against document order |\n| version | hashing the payloads again |\n| every payload | its size and hash against the manifest's list |\n| frequencies | against the length of each posting list |\n| the build itself | building the corpus a second time and comparing the version |\n\nEvery countable in this specification is derived from text the corpus already carries, so a corpus\ncannot state a number that disagrees with its own documents and still pass. That is what the levels\nbelow rest on, and it is why the parameters of section 5 are fixed rather than described.\n\nBuilding twice is a check and not a formality. Section 11 states that two builds from the same\ndocuments with the same parameters produce the same version, and a build that reads a clock, iterates\na hash map or embeds a path makes that false while producing a corpus that passes every other check.\nA second build is the only thing that notices.\n\nA suite is an implementation of the checks above. Conformance is the checks passing, not a particular\nprogram's approval, so two suites that disagree have a defect between them rather than two opinions.\n\nA suite is only trustworthy against corpora that are known to fail, one per check. One that has never\nrejected anything has not been shown to reject anything.\n\nL1 — Searchable\n\nThe corpus can answer a query, and its answers can be combined with another corpus's.\n\nDocuments with addresses per section 3 and records per section 4.\nThe extraction and embedding declarations of section 5.\nA manifest with identity, build and countables, per section 6.\nThe document payload, the vector column and the statistics of section 7, with the position\ncorrespondence section 10 depends on.\nA manifest listing every payload with its size and hash.\n\nL2 — Related\n\nRelations can be derived between this corpus's documents and another's.\n\nL1.\nThe extracted text of each document. Without it the fingerprints cannot be recomputed, and a\ncorpus whose numbers cannot be checked cannot be reasoned across. A corpus that does not ship its\ntext stops at L1 — which is a real decision for a corpus of other people's pages, where shipping\nthe text is a different act from searching it.\nThe fingerprint declarations of section 5: simhash and shingle bands, with their parameters.\nCanonical entities under zetlyn-entities-1, the entity postings, and the entity frequency table.\nPublisher clusters with their registered domains, where the documents carry domains at all. A\nfile share and a database do not, and section 10 says what follows: each document is its own voice.\nThe requirement is to publish what you have, not to invent it.\n\nL3 — Publishable\n\nThe corpus can be handed to somebody else and used under stated terms.\n\nL2.\nPer-document provenance, licence, attribution and the rules of section 9.\nContent-addressed versions, pointers and the cadence of section 11.\n\nA private corpus usually stops at L2: it is never distributed, so L3 costs it something and buys\nnothing. Each L3 requirement is something a recipient cannot establish alone.\n\nClaiming it\n\nA claim names the level and the specification version. Both are checkable by re-running the checks\nagainst the corpus, which is the point of building them out of recomputation.\n\n13 · Versioning\n\nThis specification is versioned MAJOR.MINOR. A corpus declares the version it was built against, as\nspecversion in its manifest.\n\nThis is version 1.0.\n\n| Change | Version |\n|---|---|\n| A corpus valid under the old version is still valid, and reads the same | MINOR |\n| A corpus valid under the old version becomes invalid | MAJOR |\n| A corpus stays valid but reads differently | MAJOR |\n\nThe third row is the one that matters. A change that leaves every corpus parsing correctly while\naltering which documents a query returns, or which relations fire, is breaking. Moving a threshold is\nthe usual case: sharesentity went to version 3 because its cap gained a ceiling, and every corpus\nkept validating.\n\nA MAJOR version is not retracted. Old versions stay specified and stay testable, because corpora built\nagainst them keep existing.\n\nRule versions are separate\n\nA relation rule carries its own version (section 8) and can move without the specification moving. What\nthe specification fixes is that the version changes when the predicate does, and that it is stamped on\nevery relation so a reader sees the change per edge rather than by diffing a graph.\n\nCountable parameters\n\nA change to any parameter of section 5 is MAJOR. Two corpora built under different parameters cannot\nhave those countables compared, so the change splits the population in half — which is what a major\nversion is for.\n\nA · Schemas\n\nJSON Schema, draft 2020-12. These cover what this specification requires. An implementation may carry\nmore fields on a record; a consumer ignores what it does not know.\n\nThe hashed object\n\nThe object of section 4, whose serialization is hashed. Property order in the schema is the order the\nhash uses.\n\n{\n  \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n  \"title\": \"zetlyn.document.hashed\",\n  \"type\": \"object\",\n  \"additionalProperties\": false,\n  \"required\": [\"address\",\"title\",\"url\",\"language\",\"published\",\"keywords\",\n               \"text_chars\",\"word_count\",\"text_hash\",\"catalog_version\",\"embed_model\",\"extra\"],\n  \"properties\": {\n    \"address\":         { \"type\": \"string\", \"minLength\": 1 },\n    \"title\":           { \"type\": [\"string\",\"null\"] },\n    \"url\":             { \"type\": [\"string\",\"null\"] },\n    \"language\":        { \"type\": [\"string\",\"null\"] },\n    \"published\":       { \"type\": [\"string\",\"null\"] },\n    \"keywords\":        { \"type\": \"string\" },\n    \"text_chars\":      { \"type\": \"integer\", \"minimum\": 0 },\n    \"word_count\":      { \"type\": \"integer\", \"minimum\": 0 },\n    \"text_hash\":       { \"type\": \"string\", \"pattern\": \"^[0-9a-f]{64}$\" },\n    \"catalog_version\": { \"type\": \"string\", \"minLength\": 1 },\n    \"embed_model\":     { \"type\": \"string\", \"minLength\": 1 },\n    \"extra\":           { \"type\": \"object\",\n                         \"description\": \"the adapter's own fields; serialized with its keys sorted\" }\n  }\n}\n\nThe record\n\nThe hashed object plus what a corpus ships beside it.\n\n{\n  \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n  \"title\": \"zetlyn.document\",\n  \"type\": \"object\",\n  \"required\": [\"document_id\",\"content_sha256\",\"hashed\",\"embed_method\"],\n  \"properties\": {\n    \"document_id\":   { \"type\": \"string\", \"pattern\": \"^[0-9a-f]{64}$\" },\n    \"content_sha256\":{ \"type\": \"string\", \"pattern\": \"^[0-9a-f]{64}$\" },\n    \"hashed\":        { \"$ref\": \"zetlyn.document.hashed\" },\n    \"embed_method\":  { \"type\": \"string\", \"minLength\": 1 },\n    \"text\":          { \"type\": \"string\",\n                       \"description\": \"the extracted main text. Required when adapter.indexing is full_text\" },\n    \"simhash\":       { \"type\": \"integer\", \"minimum\": 0, \"maximum\": 18446744073709551615 },\n    \"bands\":         { \"type\": \"array\", \"maxItems\": 64, \"items\": { \"type\": \"integer\" } },\n    \"entities\":      { \"type\": \"array\", \"items\": { \"type\": \"string\" } }\n  }\n}\n\ntext is required at fulltext and optional otherwise, but a corpus that omits it cannot have its\nfingerprints recomputed, so it cannot reach L2.\n\nThe manifest\n\n{\n  \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n  \"title\": \"zetlyn.manifest\",\n  \"type\": \"object\",\n  \"required\": [\"spec_version\",\"corpus\",\"version\",\"built_at\",\"documents\",\"relations\",\n               \"rule_set_id\",\"adapter\",\"countables\",\"retention\",\"read_gate\",\"payloads\"],\n  \"properties\": {\n    \"spec_version\": { \"type\": \"string\", \"pattern\": \"^[0-9]+\\\\.[0-9]+$\" },\n    \"payloads\": {\n      \"type\": \"object\",\n      \"minProperties\": 1,\n      \"additionalProperties\": {\n        \"type\": \"object\",\n        \"required\": [\"bytes\", \"sha256\"],\n        \"properties\": {\n          \"bytes\":  { \"type\": \"integer\", \"minimum\": 0 },\n          \"sha256\": { \"type\": \"string\", \"pattern\": \"^[0-9a-f]{64}$\" }\n        }\n      }\n    },\n    \"corpus\":       { \"type\": \"string\", \"pattern\": \"^[a-z0-9._-]{1,64}/[a-z0-9._-]{1,64}$\" },\n    \"version\":      { \"type\": \"string\", \"pattern\": \"^[0-9a-f]{64}$\" },\n    \"base\":         { \"type\": \"string\", \"pattern\": \"^[0-9a-f]{64}$\" },\n    \"built_at\":     { \"type\": \"integer\" },\n    \"documents\":    { \"type\": \"integer\", \"minimum\": 0 },\n    \"relations\":    { \"type\": \"integer\", \"minimum\": 0 },\n    \"rule_set_id\":  { \"type\": \"string\", \"pattern\": \"^[0-9a-f]{64}$\" },\n\n    \"adapter\": {\n      \"type\": \"object\",\n      \"required\": [\"name\",\"address_space\",\"indexing\"],\n      \"properties\": {\n        \"name\":          { \"type\": \"string\", \"pattern\": \"^[a-z0-9._-]+/[a-z0-9._-]+$\" },\n        \"address_space\": { \"enum\": [\"global\",\"per-corpus\"] },\n        \"indexing\": {\n          \"type\": \"object\",\n          \"required\": [\"mode\"],\n          \"properties\": {\n            \"mode\":  { \"enum\": [\"keywords\",\"full_text\"] },\n            \"terms\": { \"type\": \"integer\", \"minimum\": 1 }\n          }\n        },\n      }\n    },\n\n    \"retention\": {\n      \"type\": \"object\",\n      \"required\": [\"evicts\"],\n      \"properties\": {\n        \"evicts\": { \"type\": \"boolean\" },\n        \"basis\":  { \"type\": \"string\" }\n      }\n    },\n    \"read_gate\": {\n      \"type\": \"object\",\n      \"required\": [\"default\",\"rules_version\"],\n      \"properties\": {\n        \"default\":       { \"enum\": [\"open\",\"closed\"] },\n        \"rules_version\": { \"type\": \"string\", \"pattern\": \"^[0-9a-f]{64}$\" }\n      }\n    },\n\n    \"countables\": {\n      \"type\": \"object\",\n      \"required\": [\"catalog_version\",\"embed\",\"simhash\",\"bands\",\"entities\",\"stopwords\"],\n      \"properties\": {\n        \"catalog_version\": { \"type\": \"string\", \"minLength\": 1 },\n        \"embed\": {\n          \"type\": \"object\",\n          \"required\": [\"model\",\"dimension\",\"normalized\"],\n          \"properties\": {\n            \"model\":      { \"type\": \"string\", \"minLength\": 1 },\n            \"method\":     { \"type\": \"string\", \"minLength\": 1 },\n            \"dimension\":  { \"type\": \"integer\", \"minimum\": 1 },\n            \"normalized\": { \"type\": \"boolean\" }\n          }\n        },\n        \"simhash\": {\n          \"type\": \"object\",\n          \"required\": [\"bits\",\"shingle_tokens\"],\n          \"properties\": {\n            \"bits\":           { \"const\": 64 },\n            \"shingle_tokens\": { \"const\": 3 }\n          }\n        },\n        \"bands\": {\n          \"type\": \"object\",\n          \"required\": [\"shingle_tokens\",\"hash\",\"keep\"],\n          \"properties\": {\n            \"shingle_tokens\": { \"const\": 12 },\n            \"hash\":           { \"const\": \"fnv1a-64\" },\n            \"keep\":           { \"const\": 64 }\n          }\n        },\n        \"entities\": { \"type\": \"string\", \"pattern\": \"^zetlyn-entities-[0-9]+$\" },\n        \"stopwords\": {\n          \"type\": \"object\",\n          \"required\": [\"name\",\"sha256\",\"languages\"],\n          \"properties\": {\n            \"name\":      { \"type\": \"string\", \"pattern\": \"^zetlyn-stopwords-[0-9]+$\" },\n            \"sha256\":    { \"type\": \"string\", \"pattern\": \"^[0-9a-f]{64}$\" },\n            \"languages\": { \"type\": \"array\", \"items\": { \"type\": \"string\", \"minLength\": 3 } }\n          }\n        }\n      }\n    },\n\n    \"owner\":       { \"type\": \"object\",\n                     \"required\": [\"name\",\"contact\"],\n                     \"properties\": { \"name\": {\"type\":\"string\"}, \"contact\": {\"type\":\"string\"} } },\n    \"licence\":     { \"type\": \"string\" },\n    \"attribution\": { \"type\": \"string\" },\n    \"cadence\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"publishes\":  { \"type\": \"string\" },\n        \"retains_for\":{ \"type\": \"string\" }\n      }\n    }\n  }\n}\n\nowner, licence, attribution and cadence are required at L3 and absent otherwise. The schema\ndoes not require them, because a corpus that is never distributed is still valid; section 12 does.\n\nThe const values under simhash and bands are the parameters of section 5. They are constants\nrather than free numbers because a corpus that used others would produce fingerprints nothing can\ncompare against, and a declaration of that fact would only make the incompatibility legible.\n\nB · Rule registry\n\nThe rule versions this specification defines. Section 8 has their predicates; this is the register, so\na relation's stamped version can be resolved to a meaning.\n\nA rule's version changes when its predicate changes, including a moved threshold. It is stamped on\nevery relation, so a threshold that moves is visible per relation instead of by diffing a graph.\n\n| id | version | what it means |\n|---|---|---|\n| duplicateof/texthash | 1 | initial |\n| nearduplicate/simhash | 1 | initial |\n| quotes/minhashshingle | 1 | initial |\n| cites/outlink | 1 | initial |\n| citesdomain/rollup | 1 | initial |\n| obsoletes/rfc-index | 1 | initial |\n| updates/rfc-index | 1 | initial |\n| mentions/nercanonical | 2 | the noise filter and the per-document cap are applied when the document is written, so the entities a document has are the entities it contributes |\n| sharesentity/distinctive | 3 | the distinctiveness threshold is min(⌈√n⌉, 32); the absolute ceiling is what keeps it nearly independent of corpus size |\n\nA relation stamped with a version not listed here was produced before this specification and is not\ncovered by it. A consumer may read it; nothing here says what it means.\n\nrulesetid is the content hash over the ids and versions a corpus was built with. A corpus declares\nit, and a reader quotes it when explaining a relation.\n\nC · Reserved names\n\nLittle is reserved, because little needs to be. Corpus names are scoped to a deployment and adapter\nnames are scoped to their owner, so neither is a shared namespace.\n\nThe address prefix\n\nzetlyn:// is reserved. A native address may not begin with it, and section 3 refuses one that does.\n\nAdapter names\n\nThe zetlyn/ owner namespace is reserved: zetlyn/web, zetlyn/news and anything else under it.\n\nEvery other name is yours, and it has to carry an owner: suspero/wiki, not wiki. An unqualified\nadapter name is reserved and a consumer refuses it, because a bare name is a claim on a namespace\nnobody allocated.\n\nManifest keys\n\nUnqualified top-level keys are reserved for this specification, including ones it does not use yet.\n\nAn implementation adding its own uses an x- prefix. A consumer ignores keys it does not recognise.\nThat combination is what lets a MINOR version add a key without invalidating anything.\n\nCorpus names\n\nThe zetlyn/ owner namespace is reserved, as it is for adapters. Every other owner is yours, and a\ncorpus name without one is refused.","simhash":3545974178770022902,"bands":[2996456272093803,3621599008139487,4860956552758168,5400423114923559,5787705063122255,6734452027185846,6961166791911930,7815756382544993,10605609097917107,12154328684126375,13024236893551780,13284181933264190,13868077537972747,14440851745677642,18736187158129351,19074238141129240,19353689030948950,20281725666016990,21935006337763103,25326137500200174,25537378205211810,26365052950742846,27446497904444124,31077777709867664,38841473069194395,39980315917412905,40247199291981279,40436893800394377,40850682063476462,41216145198837113,41356441993584486,52039604229230821,52675028453473385,57197447391298235,57665332119887517,59044442807826300,61547068737647605,66313220521581833,68433077042384527,68520186715360984,70362859181078637,71780212694260578,72306265293177253,72577474473697762,77314522014624082,77705486128224085,77909745699595081,80896944272624017,84789881383570630,86972792411756376,87932057843839254,89161645005193186,98421486731755877,104204234865436071,106038416746865537,108234632441561485,109850212363363446,112612580091186957,114053930019403152,116209377310236694,126225683370666230,128110892886640790,128810570235771036,129733063509474429],"entities":["open zetlyn corpus specification version","section","conformance","conventions","must","should","rfc","rules","sha-256","utf-8","lf lowercase","scope","identity","declared","index","relations","permissions","composition","distribution","versioning","schemas","json","b rule","c reserved","terms document","whether","corpus","publisher","which","none","nothing","bm25","replacement","etld+1","quality any","https://example.org:443/a","four","sha-256(address","split","without","embedding declared","simhash","fnv-1a","near-duplicate hamming","shingle","french","english","appendix a","countables the","distribution l3","ieee","payload","relations a","hamming","simhash hamming","example ltd\",\"processing\":false a","l3 a","composition a","derived","distribution how","cpus","pointers a","deltas a","cadence a","withdrawal there","conformance a","l1 searchable the","l2 related relations","l3 publishable the","versioning this","major.minor","version a","minor a","major a","major the","major","json schema","additionalproperties","minlength","maxitems","minproperties","rule","reserved","little","unqualified","minor"]}
{"document_id":"67f991d3b1dac9da32a8314a25c867945daac67cd1c508cdf5d25bacd6058b9f","content_sha256":"0ae8d0cb8a521232a97365e6be67da27eaa505a4fece556dc0f97b361292403f","hashed":{"address":"zetlyn://zetlyn/spec/spec/zetlyn-hub-1.0.md","title":"Zetlyn Hub Layout","url":null,"language":"eng","published":"2026-09-14T10:45:33Z","keywords":"","text_chars":6857,"word_count":1122,"text_hash":"f485150cb3e6c305bad35f9a50865a2196a7f2d5aa8ac3f13d617dc2b3fa6460","catalog_version":"folder-2","embed_model":"multilingual-e5-small","extra":{"bytes":7032,"extension":"md"}},"embed_method":"chunked-256w-mean-v1","text":"Zetlyn Hub Layout\n\nVersion 1.0.\n\nA hub is where published corpora are fetched from. It is a directory layout served over HTTPS and\nnothing more: any web server, object store or static host can be one, and a mirror is a file copy.\n\nhub.zetlyn.com is one hub. It is not privileged by this specification; it is the default a client\nuses when a reference names no host.\n\nReferences\n\n[host/]owner/name[@tag]\n\nsuspero/handbook                       the default hub\nsuspero/handbook@v3                    a tag\nhub.example.com/suspero/handbook@v3    another hub\n\nThe first segment is a host when it contains a dot. That is why an owner may not contain one: with\nsus.pero/handbook a reader could not tell which it was looking at. Names may — suspero/handbook.v2\nis a corpus, not a host.\n\nA reference is not a corpus name. Two hubs may both serve suspero/handbook, and the manifests\ninside them carry the same corpus name, which a deployment cannot hold twice. A subscriber that wants\nboth renames one locally; the name inside the artifact is the publisher's, the name in a deployment is\nthe operator's, and a host in a reference says where the bytes came from rather than what the corpus is\ncalled.\n\nA reference with no tag resolves the tag latest. There is nothing automatic about latest; it is an\nordinary tag the publisher sets, and a publisher who sets none has no untagged reference.\n\nA tag is at most 64 bytes of a-z, 0-9, .,  and -, and may not begin with a dot. It\nbecomes a path segment on the way to tags/{tag}, so a tag carrying a slash or .. is a request to\nread somewhere else on the host serving it, and one beginning with a dot is a hidden file. A client\nrefuses a tag it cannot spell before it asks for it.\n\nThe layout\n\nA hub carries two kinds of thing, and they are separated at the top because they are fetched for\ndifferent reasons and by different commands.\n\n/corpora/{owner}/{name}/tags/{tag}                     → one line: a version hash\n/corpora/{owner}/{name}/versions/{hash}/manifest.json\n/corpora/{owner}/{name}/versions/{hash}/manifest.sig   → optional\n/corpora/{owner}/{name}/versions/{hash}/{payload}\n/corpora/{owner}/{name}/index.json                     → optional\n\n/adapters/{owner}/{name}/tags/{tag}                    → one line: a version hash\n/adapters/{owner}/{name}/versions/{hash}/manifest.json\n/adapters/{owner}/{name}/versions/{hash}/{target}/{file}\n\n{hash} is the version: the content hash of what the version consists of, as the corpus\nspecification's named-list hash defines it — the payloads for a corpus, the files for an adapter. A\nversion directory is written once and never changed.\n\nAn owner may not hold a corpus and an adapter under one name. The two trees keep them apart on the\nhub, and nothing keeps them apart in a deployment: zetlyn/folder names one thing there, and a name\nthat meant a corpus in one command and a program in another would be a name nobody could read.\n\nAn adapter version\n\n{\"spec_version\":\"1.0\",\"adapter\":\"zetlyn/folder\",\"version\":\"…\",\"interface\":1,\n \"files\":{\"linux-x86_64/adapter-folder\":{\"bytes\":2841064,\"sha256\":\"…\"},\n          \"darwin-aarch64/adapter-folder\":{\"bytes\":2370704,\"sha256\":\"…\"}}}\n\ninterface is the adapter protocol version the program speaks, stated so that a host can refuse\nbefore it downloads rather than after. {target} is the platform the file runs on, written as the\narchitecture and the system it was built for; a host fetches the one it is, and a version that does\nnot carry it is a version that does not run here.\n\nAn adapter is a program and its manifest says nothing about what it produces. Whether it can fill a\ncorpus that reaches a conformance level is a question about the corpus, and the corpus answers it.\n\nA tag is mutable, a version is not. Moving a tag is how a publisher releases; it is also the only\nthing a compromised hub can do to a subscriber who follows one. A subscriber who pins a hash is\nunaffected by anything the hub does afterwards.\n\nindex.json carries what a search needs — names, descriptions, licences, the latest tag. It is\noptional because a bucket cannot generate one, and a hub without it simply cannot be searched.\n\nFetching\n\nA subscriber, in order:\n\nresolve the reference to a host, owner, name and tag, and the tree the thing is in;\nread {tree}/{owner}/{name}/tags/{tag} for the version hash;\nread versions/{hash}/manifest.json;\nrefuse unless the manifest names this corpus — or this adapter — and this version;\nverify manifest.sig when a key is pinned for this owner;\nfetch the files the manifest lists, and check each against it;\nrefuse unless their hash together is {hash}.\n\nStep 7 is what makes the transport irrelevant. A hub that serves the wrong bytes fails the check, and\nso does a mirror that lost one. It is also the only step that catches a hub lying consistently — one\nthat replaces a file and updates the manifest entry beside it passes every per-file check, and\ncannot make the set hash to the version it is serving them under.\n\nAn adapter is fetched the same way and then run. What a hash cannot tell you is whether the program\nis one you want to run at all; that is what pinning a key is for, and what a subscriber who has not\npinned one does not have.\n\nSignatures\n\nOptional, and Ed25519. The publisher signs the bytes of manifest.json; the detached signature is\nmanifest.sig beside it.\n\nA subscriber pins a public key per owner. A pinned key that does not verify is a refusal, not a\nwarning. Without a pinned key the signature is not checked — an unchecked signature is decoration, and\nthis specification does not pretend otherwise.\n\nWhere the key comes from is outside this specification, and deliberately. A key fetched from the\nhub it is meant to defend against defends against nothing. Publishers hand it over the way they hand\nover anything else that has to be trusted — on their own site, in a contract, in a mail somebody can\ncheck — and a subscriber that has not done that has a corpus verified by its hash and not by its\nauthor. A hub may serve a key for convenience; taking it from there is trust on first use, which is\nweaker than a pin and should not be called one.\n\nSigning defends the case content addressing does not: a hub that moves a tag to different bytes. If\nevery subscriber pins a hash, signing buys nothing; if subscribers follow tags, it is the only defence.\n\nPublishing\n\nWriting to a hub is outside this specification. A static host is written to by whatever writes to that\nhost — aws s3 sync, scp, a CI job.\n\nhub.zetlyn.com allocates owner namespaces on registration, first come, and requires a token to write\nunder one. Another hub may do as it likes; nothing here depends on it.\n\nWhat a hub is not\n\nIt does not answer queries, hold a scope, or know what a subscriber does with a corpus. It serves\nfiles. A corpus fetched from a hub is read locally and no question ever reaches the hub.","simhash":12345356738332145005,"bands":[11935661926511583,12028195456951392,29378169662484825,36250256453192757,44453687962197676,63643284914890493,148455153606266158,154154520722062399,154395956213780667,159646932587937912,167181702001041302,169364266953539290,173969216675397400,187196785126653781,190964451991121007,232398604397238103,238721729380900713,239712402766985673,246985668817043391,248276201908576494,287385459995251267,311316174055995206,311929442277040787,312097138588999897,323363110477824484,325264186832664870,330664749449850655,334460222419880990,338595090872776551,343818490926595898,348897522070165401,350174781821391025,350498486886931818,376452377644648289,402555780543914807,412305094152323212,438688789173981904,441877495631596835,451573997710504895,477634249240058705,477868225924340012,481492651129990309,505459967475236748,538758509862572357,569820462094458522,579362308259892556,599464310079987220,611841402318508067,625582826860076304,675758479388459784,698756553440035627,702638835484858320,742680707491425993,746055648539493782,756676546576985844,758586327415935219,778688788268473561,798783028646225455,804815092011845730,812821370709067541,815984780871557695,851962223947085345,870441464266997898,872326892274237794],"entities":["zetlyn hub layout version","fetching a","signatures optional","ed25519","publishing writing"]}
{"document_id":"bad851bd8da8982ff6eb0aa8ac03e0fd71212b7ab3457a6f973d9c4812bad919","content_sha256":"380d061bf73a32cc6e1de7756448930bbd0cc8eb65e5172beba53b7de4adb653","hashed":{"address":"zetlyn://zetlyn/spec/vocabulary/zetlyn-entities-1/README.md","title":"zetlyn-entities-1","url":null,"language":"eng","published":"2026-09-12T21:30:16Z","keywords":"","text_chars":2243,"word_count":371,"text_hash":"8147fe180f79b59dc18c8d8edb93d93d3c3b973c5c3e64f09d70aeeae51d4e86","catalog_version":"folder-2","embed_model":"multilingual-e5-small","extra":{"bytes":2290,"extension":"md"}},"embed_method":"chunked-256w-mean-v1","text":"zetlyn-entities-1\n\nThe entity vocabulary every conforming corpus uses. Section 5 of the corpus specification fixes it\nrather than letting a corpus declare one, because the relation that carries almost all of a route's\nstructure is the entity-mediated one, and two parties canonicalising differently would share fewer\nentities than they should without anything reporting it.\n\nTwo parts, and a corpus uses both or neither:\n\n| | |\n|---|---|\n| recognizer.md | what a recognizer has to produce, and how the artifact is pinned |\n| the cap | at most 512 distinct entities per document, document order, first mention kept |\n\nAn entity is the recognizer's span, lowercased. Nothing is mapped onto anything else.\n\nVersioning\n\nThe name carries the version, and the version covers all three parts together. zetlyn-entities-2\nwould be a different vocabulary; corpora on different versions do not derive entity relations between\nthem, which is the rule every other fixed parameter follows.\n\nAnything that changes which entities a document has changes the version. That is expensive on purpose:\na vocabulary that shifted under a published corpus would silently change what that corpus shares with\nevery other one.\n\nPinning\n\nA manifest names zetlyn-entities-1 and nothing more. What that name means is the content of this\ndirectory at the release that published it, and a release states the hash of each file. A corpus built\nagainst a different file with the same name is not conforming, and the hashes are what makes that\ncheckable rather than assumed.\n\nWhat is not settled\n\nThe alias table here is a starting table, not a vocabulary. It covers software and technology,\nbecause that is what the first corpus held. A regulator's corpus, a medical corpus and a legal corpus\nshare almost nothing with it, and an entity they both name — a directive, a substance, a court — is\ncanonicalised by neither.\n\nThat is the open question this directory raises rather than answers: a vocabulary fixed for everyone\nhas to be good enough for everyone, and one that is not simply moves the failure from two parties\ndisagreed to neither party was covered. Growing it is not a matter of adding rows; it is the\nquestion of who curates a shared vocabulary and on whose authority.","simhash":16175095528026906677,"bands":[46268443012284033,154166374341420222,175686399872098751,218761906797175262,267441034112772847,304977475460073947,370012814491182625,458285030427684746,467056857738914793,511741754579159187,531895351313557877,555404870128026193,590358031965371391,693845500518368071,724208518619194599,725353868364353191,742320362635159146,779126143985946399,797901600625378360,876262463049772311,966867243855788595,977021972968473972,996124206333198960,1097017380919059044,1160640759969058525,1185321129914876683,1202054337913391822,1219497623523392742,1278794338351645332,1280759947888288571,1287853539156566140,1314249501730943817,1318213539836628458,1331285589581244977,1353082559672701638,1395519167440568788,1412475928577239932,1476762930419860008,1495169383772888319,1505713173407159597,1540362824734606465,1568842540027224704,1630338371483056447,1865682610880346536,2007666946628426507,2088908036940219131,2212936355934150688,2230265484887478594,2307370024527804772,2428886278352103957,2527228208300789014,2531447451977255946,2701778017689446344,2704288520438392347,2749700844430099930,2760988365802212321,2771546097216711798,2821697322850794197,2830188468447346343,2913660037294147354,3024254353201244800,3059014057874568621,3099359928109556876,3139037551522915465],"entities":["versioning the","pinning a"]}
{"document_id":"1218b19b318a220e47234b1361eb47768edab84a77f3a77f76e1e166a8d97eef","content_sha256":"b774cd2bc156700469a7349cee53fe8cc2a91b8579d6ffc37ccf2c87ef710031","hashed":{"address":"zetlyn://zetlyn/spec/vocabulary/zetlyn-entities-1/recognizer.md","title":"The recognizer","url":null,"language":"eng","published":"2026-09-14T09:01:30Z","keywords":"","text_chars":2506,"word_count":417,"text_hash":"50840931f56624f3c3aa6ae658443062d75cbd0df8ed07829c9393f23efd2cd4","catalog_version":"folder-2","embed_model":"multilingual-e5-small","extra":{"bytes":2547,"extension":"md"}},"embed_method":"chunked-256w-mean-v1","text":"The recognizer\n\nWhat a conforming implementation has to produce, and how the artifact that produces it is pinned.\n\nWhat it does\n\nTakes the extracted text of one document and returns the spans that name a specific thing — a person,\nan organisation, a place, a product, a standard, a law. It does not return topics, common nouns, or\nanything a reader would call a subject rather than a name.\n\nOutput is a sequence of spans in document order. Each span is lowercased and deduplicated, keeping the\nfirst mention. At most 512 survive.\n\nWhat the specification pins\n\nNot a method and not an architecture. The artifact, by content hash.\n\nTwo implementations that agree on a fixture do not agree on arbitrary text, and entity relations are\nderived over arbitrary text. A vocabulary is therefore only shared if everyone runs the same\nrecognizer, and the only way to state \"the same\" is to name the bytes.\n\nA release of this vocabulary publishes the artifact and states its SHA-256. A corpus built with\nanything else is not conforming under zetlyn-entities-1, whatever it declares.\n\nThe artifact\n\nhttps://download.zetlyn.com/zetlyn-entities-1.json\n7e6732e23d92fd8b496d04c40a6b7ac005dafa8c622167c82bb6bffcb1b786df   1,424 bytes\n\nNot in this repository, and the reason is what the pin is for. The hash is the contract; the address\nis a convenience. A copy fetched from anywhere, carried on a disk or built into a distribution is the\nsame artifact if its bytes hash to that value, and a copy sitting in the right directory is not the\nartifact if they do not. Putting the file here as well would create a second place to look and a\nsecond thing to keep in step, and neither would be more authoritative than the number above.\n\nIt is 1,424 bytes: a per-token BIO classifier over twelve lexical features with a logistic head. No\nembedding and no GPU, which is what makes it reproducible on any machine — see Determinism below.\n\nWhat it costs to pin is worth saying. The model was trained on one body of documents, and it carries\nthat body's idea of what a name looks like. A retrain is not an improvement to zetlyn-entities-1;\nit is zetlyn-entities-2, and two corpora on different versions derive no entity relations between\nthem.\n\nDeterminism\n\nThe same bytes of text give the same spans, on any machine, in any order of documents. A recognizer\nthat batches, that depends on a thread count, or that carries state between documents produces a\ncorpus nobody can rebuild — and section 12 checks exactly that, by building twice.","simhash":10671431232392776343,"bands":[34892914459252879,167232428697973758,229877547208443906,312707735691387809,353404865277848240,399540989219339021,465284896950403066,497496498809085309,511429010870825012,553812338644688983,661925598148466139,664559662731696647,693192852983458947,735218843642509310,835245591127457748,902322560845397896,917962367266546277,950460528794105572,958683143101864560,1011089155479981051,1019689568147838369,1035585552632883285,1116000589284302942,1116102963939235577,1148837928543332357,1165769081598682191,1213589718839505218,1252902677200421778,1295808051292937727,1347439454781995839,1354286734049627085,1447192210358270439,1449544181120705152,1449722557995088653,1461908823719750364,1591330428622976609,1625749973871308830,1679364141407670670,1704914541386014019,1760304889118882711,1796799554305308786,1829585408875384644,1860585046390869345,1865856913343426712,1962830568641642014,1983662855966582929,1986046119369112529,2092130433717629369,2107778103034455467,2154415837316535598,2203798782893770739,2209916648338177149,2334832876038122747,2350248787697749030,2413311981774159439,2476640718337098097,2551515463829418687,2603039504868663136,2754976984906425542,2774099400704773022,2800068618530314904,2827004098736257157,2837655154130118686,2863568898497126526],"entities":["takes","sha-256","bio","determinism","determinism the"]}