Skip to content

fix: convert pyarrow-backed ds to datetime64 before fitting - #2780

Open
vishalhabib99 wants to merge 1 commit into
meta:mainfrom
vishalhabib99:fix/pyarrow-ds
Open

vishalhabib99 wants to merge 1 commit into
meta:mainfrom
vishalhabib99:fix/pyarrow-ds

Conversation

@vishalhabib99

Copy link
Copy Markdown

fit fails when ds is a pyarrow-backed timestamp column, which is what pd.read_parquet(..., dtype_backend="pyarrow") returns for a datetime column written by pandas.

import numpy as np
import pandas as pd
from prophet import Prophet

df = pd.DataFrame({"ds": pd.date_range("2021-01-01", periods=100), "y": np.arange(100.0)})
df = df.astype({"ds": "timestamp[ns][pyarrow]"})
Prophet().fit(df)
  File ".../prophet/forecaster.py", line 1199, in set_auto_seasonalities
    min_dt = dt.iloc[cast(np.ndarray, dt.values).nonzero()[0]].min()
AttributeError: 'ArrowExtensionArray' object has no attribute 'nonzero'

Why

setup_dataframe parses ds with pd.to_datetime, which leaves timestamp[ns][pyarrow] as pyarrow (with pandas 3.1.1 it converts timestamp[us][pyarrow] to datetime64[us], but not the ns variant). Two places then assume a numpy-backed column:

  • set_auto_seasonalities calls .nonzero() on dt.values, which is an ArrowExtensionArray.
  • Past that, fourier_series calls .dt.total_seconds(), which on pyarrow durations casts the nanosecond integers to double and raises ArrowInvalid: Integer value ... not in range because they're above 2**53.

Fix

A small _to_datetime helper runs pd.to_datetime and, when the result has an ArrowDtype, wraps it in pd.DatetimeIndex to get a numpy datetime64 column. It's used in the two places that parse ds: setup_dataframe (fit and predict) and history_dates in preprocess. A timezone-aware pyarrow column becomes datetime64[ns, tz], so the existing "Column ds has timezone specified" check still raises. For anything that isn't pyarrow-backed the behavior is unchanged. The getattr(pd, "ArrowDtype", ()) keeps it working on pandas versions before 1.5.

With the change, a model fit on the pyarrow column gives the same seasonalities and the same yhat as one fit on the numpy column. predict, cross_validation, performance_metrics, model_to_json/model_from_json, plot and plot_components all work on the pyarrow-backed data.

Tests

  • test_fit_predict_pyarrow_ds[ns|us]: fits on timestamp[ns] and timestamp[us] pyarrow ds, checks seasonalities and yhat against the numpy-backed model, and also predicts on a pyarrow-backed future frame. The ns case fails on main with the AttributeError above. The us case already passes on main because pd.to_datetime converts that one, and it's there to keep it working.
  • test_pyarrow_ds_with_timezone: a tz-aware pyarrow ds still raises the timezone ValueError.
  • Both use pytest.importorskip("pyarrow"). CI installs .[dev,parallel], which brings in pyarrow through dask[dataframe], so they run there.

Full suite (test_prophet, test_diagnostics, test_serialize, test_utilities, test_metrics, test_plot, test_backend_loading): 141 passed, 6 skipped with the change; on main the only failure is the new ns test. pyrefly check reports the same 5 errors before and after, all missing optional modules in my environment (dask, plotly).

Tested on macOS arm64, Python 3.12, pandas 3.1.1, pyarrow 26.0.1, cmdstanpy backend, running main's Python code on the 1.5.0 wheel's compiled Stan model.

A ds column of type timestamp[ns][pyarrow] (for example from
pd.read_parquet(..., dtype_backend="pyarrow")) stays pyarrow after
pd.to_datetime, and fit then fails in set_auto_seasonalities with
"'ArrowExtensionArray' object has no attribute 'nonzero'", and in
fourier_series because pyarrow cannot cast the nanosecond values to
float. Convert pyarrow-backed timestamps to numpy datetime64 where ds is
parsed (setup_dataframe and history_dates). Timezone-aware columns keep
their timezone, so the existing timezone check still raises.
@meta-cla meta-cla Bot added the CLA Signed label Nov 8, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant