Test instructions here.
- Python version: 3.12.2
- OS: Windows 11, using WSL2 with Ubuntu 22.04.3
Running docker compose (Source DB - postgresql, Target DB - postgresql, and Source DB API - FastAPI):
docker compose upTo setup (from repo root folder):
cd etl_script
python -m venv env
source env/bin/activate
python instal -r requirements.txtTo run:
python main.py -d <date: yyyy-mm-dd>Example:
python main.py -d "2023-01-10"To setup (from repo root folder):
cd my_dagster
python -m venv env
source env/bin/activate
python instal -r requirements.txtTo run:
dagster devTable schema:
- data
- timestamp TIMESTAMP,
- wind_speed DECIMAL,
- power DECIMAL,
- ambient_temperature DECIMAL,
- PRIMARY KEY (timestamp)
Script used to generate data: source_db/create_mock_data.sql.
This script generated data with 1-min interval, between 2024-01-01 01:30:00 and 2024-01-11 01:29:00.
This data can be found in: source_db/dataset/data.csv.
Exposed port to host: 5432
Tables schema:
- signal
- id INT,
- name VARCHAR(20),
- name VARCHAR(20),
- PRIMARY KEY (id)
- data
- timestamp TIMESTAMP,
- signal_id INT,
- value DECIMAL,
- PRIMARY KEY (timestamp, signal_id),
- FOREIGN KEY (signal_id) REFERENCES signal(id)
Exposed port to host: 5433
Exposed port to host: 8081
Exposed endpoints:
Example:
Method: POST
URL: http://localhost:8081/data
Body:
{
"start_datetime": "2024-01-08T07:01:01",
"end_datetime": "2024-01-08T08:01:01",
"fields": ["wind_speed", "power"]
}
Used references: