Repository navigation
feat: dataset controller for rows, DataFrames and distributions - #14
Merged
Merged
Conversation
- Dataset gains about, data, generated_by and parts - Distribution gains file, access_url, byte_size, media_type, compression_format, package_format, conforms_to and table_schema - Distribution.download_url is now optional - File is imported from opensemantic.core instead of regenerated
- inline dict payload is written to a LocalFile attached via Distribution while Dataset.data is cleared - File -> Dataset.data round trip restores the payload unchanged - a serialized dataset keeps only a node reference to the file, which a resolver dereferences before the payload is read back - runs against both the pydantic v1 and v2 models
…ions - to_df/from_df convert the dataset's rows to a pint-pandas frame - materialize_data fills data from a distribution's file - externalize_data writes data to a file as JSON or CSV and attaches the distribution - file access goes through the core file controllers - export BaseDataView and COLORS from the view package
Keeps the rows reachable once the dataset is reloaded and its file is only a reference.
A controller may take metadata from the stream, the wiki one derives the uploaded page title from its name.
- from_df keeps a unit the model does not declare, so a to_df round trip no longer reinterprets magnitudes, and reports a column it cannot build - _declared_unit returns None for a field that is not a model - a URL-only distribution resolves RemoteFile on the dataset's own layer - materialize_data guards a missing row model and tries the next distribution when a read fails - serialize_data dumps v2 rows through model_dump, not the deprecated dict - _unwrap leaves a multi-member union alone and unwraps a mapping to its value type - _file_url reports an unreadable attribute instead of swallowing it - tests drive DatasetControllerMixin directly, on both layers - pin opensemantic.core to the branch carrying the file controllers
A caller hands the file controller what it needs to read the file - a wiki session, credentials - without knowing which controller answers.
- a row field that is not a model takes the cell as it stands, instead of being forced through float() into a value wrapper - materialize_data resolves the controller inside the try, so an unreadable file entity moves on to the next distribution, and the final error chains the cause that was collected - _file_url no longer lets an unexpected failure out after the upload - _unwrap sees through Annotated; a sequence cell is a value, not a gap - a missing optional extra is reported at info, not warning
- shared x-range at figure creation, and the reset event bridge - shadow-DOM stylesheets, content scrolling owned by panelini - why HTML export builds detached figures - unique column names for the unit-aware export - what the Playwright browser test has to get right
pint-pandas dequantify/quantify require every column to have a unit, but a table row also carries a label, a count, a timestamp. _pint_csv keeps the unit row for the columns that have one and passes the rest through, and a CSV from elsewhere is read with its single header row instead of losing its first data row to a header. - _pint_csv is shared with the view's export, which had its own copy - examples/dataset_from_remote_csv.py wraps a CSV on the web in a Dataset and reads the rows back as AddressRow
A Dataset a caller builds, or a schema generates, now converts and materializes itself; subclassing it is enough. DataToolController and the database controllers keep their own names: the first indexes channels, warns and may open an archive database on construction, and of the latter three answer for one Database, one of them needing a db_path. Also: the archive auto-init reads storage_locations only when it will use it. That is a link field, so reading it resolves IRIs, and building a tool that does not archive must not need a resolver.
The file controllers are on core's main; the floor replaces the pin once a version tag is cut.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DatasetControllerMixin: the conversions between a dataset's inline rows, apandas/pint DataFrame and a file behind a
Distribution.Dataset controller
to_df()/from_df(df, label=...)- one column per row field, each in thefield's declared unit. A field without a declared unit keeps the column's own
unit, so a round trip does not reinterpret magnitudes, and a field that is not
a quantity (a timestamp, a label, a count) is carried through unchanged.
materialize_data()/externalize_data(file, fmt=...)- move the rowsbetween
dataand a file, in JSON or CSV. File access goes through theopensemantic.corefile controllers, so a local file, an HTTP URL and a wikipage are read the same way.
materialize_data(**kwargs)passes a session orcredentials on to the controller.
tried; the final error chains the cause.
Datasetnames the controller that implements it, so a dataset a callerbuilds, or a schema generates, converts and materializes itself.
DataToolControllerand the database controllers keep their own names: thefirst indexes channels and may open an archive database on construction, and
of the latter three answer for one
Database.CSV with a unit header
_pint_csvwrites the two-row(field, unit)header for the columns that havea unit and passes the rest through, where pint-pandas requires every column to
carry one. A CSV from elsewhere is read with its single header row instead of
losing its first data row to a header. The view's export shares this code.
Also
examples/dataset_from_remote_csv.pywraps a CSV on the web in a dataset andreads the rows back as a typed row model.
tests/test_dataset_controller.py: 58 tests over both pydantic layers.test has to get right.
Needs the file controllers from opensemantic.core, merged but not yet released,
so the dependency is pinned to its
mainuntil a version tag is cut.