This package implements the Green Algorithms methodology to estimate component-wise energy consumption and carbon footprints of computational workloads.
Important
This package is open-source and directly usable, but is mainly intended as a base package to be used by other Green Algorithms tools.
Divided primarily into two pipelines—Ingestion and Computation—it processes raw job logs extracted directly from workload managers (such as SLURM) or structured CSV files. The pipeline produces aggregated output per user and per day. Additionally, the computation module exposes underlying tools to calculate energy consumption, carbon footprints, and carbon emission equivalents directly.
⚠️ Status: This package is in its early developmental stages and is expected to evolve significantly.
- Flexible Ingestion: Extract logs from SLURM using sacct (with provision to extend for other workload managers) or structured logs in CSV format.
- Energy Calculation: Models power draw based on CPU/GPU architecture, core counts, memory usage, and runtime.
- Carbon Footprint Calculation: Uses CarbonIntensity regional API (UK only) to integrate dynamic carbon intensity (CI) with estimated energy consumption.
-
Contextual Metrics: Translates raw
$\text{CO}_2\text{e}$ metrics into intuitive real-world equivalents (e.g., tree-months for sequestration, car miles driven). -
Storage Backend Agnostic: Supports custom storage backends (via
CIStorageBackend) to cache or persist carbon intensity values.
- Python:
>= 3.9(Python 3.14 recommended) - Core Dependencies:
pandasnumpyrequests
Note
We recommend using the Python version 3.14 but this package is compatible from Python 3.9 onwards depending on what version is available in your environment. Please note that Python 3.9 and 3.10 are now deprecated, more info on Python versions here.
Currently, Green-Algorithms-core can be installed directly from this Git repository using pip.
pip install git+https://github.com/Cambridge-Sustainable-Computing-Lab/Green-Algorithms-core.gitpip install git+https://github.com/Cambridge-Sustainable-Computing-Lab/Green-Algorithms-core.git@v0.1.0
pip install git+https://github.com/Cambridge-Sustainable-Computing-Lab/Green-Algorithms-core.git@main💡 Refer to tests/conftest.py for examples of required configuration parameters.
import pandas as pd
from ga_core import HPCDataProcessor
# Define Configurations and Fixed Parameters
cluster_info = {
# Add cluster-specific information here...
}
config_data = {
# Add main configurations here...
}
fixed_params = {
# Add hardware constants and baseline parameters here...
}
# Initialize the Processor
processor = HPCDataProcessor(
config_data=config_data,
cluster_info=cluster_info,
fixed_params=fixed_params,
all_users_access=False # Set to True if you have access to multiple user's logs
)
# Extract and Enrich Data
raw_df = processor.extract_data()
enriched_df = processor.enrich_data(raw_df)
# View Results
print(enriched_df[["UserX", "StartDatetimeX", "carbonFootprint", "energy_CPUs", "energy_GPUs"]].head())You can bypass direct interaction with the workload manager by reading raw logs from a file instead. HPCDataProcessor.extract_data() accepts raw log bytes and handles cleaning and preparation for the enrichment step.
with open('sample_raw_logs.txt', 'rb') as f:
logs_raw = f.read() # Read raw logs
# Extract and enrich data
raw_df = processor.extract_data(logs_raw)
enriched_df = processor.enrich_data(raw_df)If you are contributing to or extending the codebase locally:
# Clone the repository
git clone https://github.com/Cambridge-Sustainable-Computing-Lab/Green-Algorithms-core.git
cd Green-Algorithms-core
# Install in editable mode with development dependencies
pip install -e ".[dev]"💡 Using .[dev] installs the package in editable mode along with testing tools like pytest.
Execute the test suite to verify your local setup:
pytest .- Fork the repository and clone your fork locally.
- Create a new branch off
mainfor your change:
git checkout main
git checkout -b feature/<your-feature-name>-<your-username>- Make your changes, then run
pytest .to make sure nothing's broken. - Commit your changes with a clear message, push to your fork, and open a Pull Request against
main.
Important
Please open an issue for larger changes.
The tests/ directory also serves as a reference for inputs, configurations, and expected pipeline outputs:
- Pipeline Examples: Check
tests/test_hpc_data_pipeline.pyto see how the pipeline handles completed, failed, and unfinished jobs. - Sample Data: Inspect
tests/testdata/slurm/for raw SLURM log fixtures and their corresponding golden CSV outputs. - Fixture Configurations: Refer to
tests/conftest.pyfor baseline configuration dictionaries (cluster_info, config_data, fixed_params).
This package is built and maintained by the Cambridge Sustainable Computing Lab at the University of Cambridge, UK.
This work is licensed under the GNU General Public License v3.0.