Repository navigation
perf: read a distribution straight into a DataFrame - #16
Merged
Merged
Conversation
- materialize_df skips building a model per cell, which is what made a large dataset take minutes; materialize_data keeps the typed rows - to_df reads a column's magnitudes directly when its values agree on one unit, and converts per value only when they differ (#15) - both read paths share one distribution loop 20k rows: materialize_df 0.25 s against 11.07 s for the rows (csv), to_df 4.19 s to 0.57 s
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #15.
Loading a cycling dataset for the dashboard took minutes. The cause is
structural: the dataset was read into one pydantic model per cell and
immediately flattened back into a DataFrame, and only the DataFrame is
plotted.
materialize_df
Reads a distribution straight into a frame, without constructing rows.
materialize_datais unchanged and still returns the typed rows, and both nowshare one distribution loop (
_read_distribution), so the retry, the warningper unreadable distribution and the chained error behave identically.
CSV goes through
_pint_csv.read_csv. JSON is read column by column: aserialized value is
{"value": x, "unit": "Item:OSW..."}, so when a columnagrees on one unit the magnitudes are taken directly and the unit is attached
once.
to_df vectorized
A column almost always holds one unit. When that unit is the target, the stored
magnitudes already are the answer, so the column is read without pint at all.
A column that mixes units keeps the per-value
to_pint().to(unit)conversion,covered by a test with a kelvin and a Celsius value in one column.
This also speeds up
serialize_data("csv"), which uploads use.Measured, 20 000 rows
materialize_df(csv)materialize_data(csv)materialize_df(json)to_dfFor reference,
pd.read_csvof the same data is 0.02 s.Tests: 234 pass, 16 new. The two failures in
test_controller.pyneed a livePostgREST and fail the same way on main.