Repository navigation
17 clean service oriondb ingest and map penelopedb sensor data - #44
diyakadakia merged 29 commits into
Conversation
…Id and time bounds
…or-data' of https://github.com/GenerateNU/orion into 17-clean-service-oriondb-ingest-and-map-penelopedb-sensor-data
…or-data' of https://github.com/GenerateNU/orion into 17-clean-service-oriondb-ingest-and-map-penelopedb-sensor-data
…, first 4 tests passed, need to test last 2
| t.c.dataTypeName == data_type_name, | ||
| ) | ||
| with self.engine.connect() as conn: | ||
| return conn.execute(stmt).scalar() No newline at end of file |
There was a problem hiding this comment.
What is this method for?
| return self._fetch(stmt) | ||
|
|
||
| def get_all_paginated(self, batch_size: int = 5000): | ||
| def get_all_paginated(self, batch_size: int = 5000, stmt=None): |
There was a problem hiding this comment.
The variable being stmt is really ambiguous also injecting a SQL query to our get method is questionable. I would much prefer if you pass it a list of sensor names you wish to query then have a helper construct what its pulling.
| return written | ||
|
|
||
|
|
||
| def ingest_run( |
There was a problem hiding this comment.
I think I need a code walkthrough on monday. There is a lot going on here
| Column("sensor", String, ForeignKey("sensors.name"), nullable=False) | ||
| ) | ||
|
|
||
| signals_table = Table( |
There was a problem hiding this comment.
What is the point of the signals table?
| tables=[sensors_table, sensor_sources_table, cleaned_data_table], | ||
| checkfirst=True, | ||
| ) | ||
| with engine.begin() as conn: |
There was a problem hiding this comment.
If this is a service, why are there connections to the db? That should happen in a repository. Services should only have business logic
| "vcu_eth_a_gyro", | ||
| }) | ||
|
|
||
| def _flag_investigation_tags(run_id: str, sensor_names: set[str]) -> set[str]: |
There was a problem hiding this comment.
What does this part do? The needs investigation part was supposed to be you guys doing research to see if you believe that they should be included in our algorithm or if you think they are unimportant for what we want.
…igation tags, and fixed return type hints
TKDPenguin
left a comment
TKDPenguin
left a comment
There was a problem hiding this comment.
Looks good to me. Thanks for updating it
5 participants
Overview
Created ingest.py in orion folder and manual_ingest_test.py in the testing folder to test the ingest. Normalized bms soc sensor to 1-100, fixed GPS lag, mapped to clean sensor names, and separated the multivalue sensors into different rows with suffix _0/_1/_2 depending on which value it is.
Technical Description
Queried Penelope DB to get raw data from a run id and timestamp first, then ingest file has helper functions to normalize, map clean sensors, and separate multivalue columns. These are then called in ingest_run function. The sensor_sources and sensors tables are also filled with the signal_catalog mappings.
There is one problem which is that when we suffix the sensors, the suffix ones are not mapped back to a source. However, the base name is mapped back to the source and is also present in sensors. We weren't sure how we wanted to fix this...maybe a base sensor column in the sensor_sources? or changing the signal catalog?
How Has This Been Tested?
Tested with custom tests in manual_ingest_testing.py; they test the full ingest and some reading and rollback functions. All passed meaning that reading penelope, rollback, cleaning, and writing to orion DB works as expected
Ticket Review
Answer the following: