etc

detdevlib.utils.etc

get_fetch_range

get_fetch_range(
    latest: datetime | None,
    refetch: datetime | None,
    hist_cutoff: datetime | None,
    future_cutoff: datetime | None,
) -> tuple[datetime | None, datetime | None]

Calculates the incremental fetch range [S, E) for a time-series ETL pipeline.

This function determines the optimal start and end times to fetch data, balancing progress tracking (watermarks) with mandatory lookbacks and hard policy boundaries.

The logic treats None as 'no bound' (effectively -infinity for lower bounds and +infinity for upper bounds) to avoid instability issues associated with datetime.min or datetime.max.

Parameters:

Name	Type	Description	Default
latest	datetime | None	The watermark (L); the latest point in time for which local data exists. Defaults to hist_cutoff if None.	required
refetch	datetime | None	The lookback point (R); a point in the past to account for data volatility. Defaults to latest if None.	required
hist_cutoff	datetime | None	The historical floor (H); the absolute earliest data we care to fetch.	required
future_cutoff	datetime | None	The future ceiling (F); the absolute latest data we care to fetch.	required

Returns:

Type	Description
datetime | None	A tuple (Start, End) defining the range:
datetime | None	Start (S): Defined as max(H, min(min(L, R), F)).
tuple[datetime | None, datetime | None]	End (E): Simply the future_cutoff (F).

safe_int

safe_int(x: str) -> int | None

Converts a string to int if possible, otherwise returns None.

clean_dict

clean_dict(d: dict) -> dict

Removes all None values from a dictionary.

is_dst_adj_day

is_dst_adj_day(date_obj: date) -> bool

Checks if a given date is a day when Daylight Saving Time (DST) adjustment occurs.

Assumes Amsterdam locality.

safe_reset_index

safe_reset_index(
    df: DataFrame, inplace: bool = False
) -> pd.DataFrame

Reset index safely by renaming index columns if they conflict with existing columns.

coerce_dataframe_columns

coerce_dataframe_columns(
    df: DataFrame, target_columns: List[str]
) -> None

Coerces the dataframe columns in place to match the expected columns.

standardize_to_utc

standardize_to_utc(series: Series) -> pd.Series

Converts a Series (object or datetime) to UTC datetime.

Logic: 1. Converts object/string to datetime. 2. Naive datetimes are localized to UTC (Assumed UTC). 3. Aware datetimes are converted to UTC.

time_range

time_range(
    start_dt: datetime,
    end_dt: datetime,
    step: Callable[[datetime], datetime],
)

Generates a sequence of datetimes from start_dt up to (but not including) end_dt.

time_blocks

time_blocks(
    start_dt: datetime,
    end_dt: datetime,
    step: Callable[[datetime], datetime],
)

Generates contiguous time intervals (blocks) covering the duration from start_dt to end_dt.

safe_localize

safe_localize(
    dt_naive: datetime, tz: tzinfo | None
) -> datetime

Localizes a naive datetime to a specific time zone.

Handles ambiguous and non-existent times (DST transitions) deterministically.

to_timezone

to_timezone(dt: datetime, tz: tzinfo) -> datetime

Converts a datetime to a specific time zone.

If the datetime is naive, it is localized to the given time zone (using safe_localize). If the datetime is aware, it is converted to the given time zone.

start_of_second

start_of_second(dt: datetime) -> datetime

Floors to the beginning of the current second (removes microseconds).

start_of_minute

start_of_minute(dt: datetime) -> datetime

Floors to the beginning of the current minute (:00).

start_of_hour

start_of_hour(dt: datetime) -> datetime

Floors to the beginning of the current hour (:00:00).

start_of_day

start_of_day(dt: datetime) -> datetime

Floors to the beginning of the current day (00:00:00).

start_of_week

start_of_week(
    dt: datetime, week_start_day: int = 0
) -> datetime

Floors to the beginning of the week based on the specified start day.

0=Monday, ..., 6=Sunday. Defaults to 0 (Monday).

start_of_month

start_of_month(dt: datetime) -> datetime

Floors to the beginning of the current month (1st, 00:00:00).

start_of_year

start_of_year(dt: datetime) -> datetime

Floors to the beginning of the current year (Jan 1, 00:00:00).

start_of_next_second

start_of_next_second(dt: datetime) -> datetime

Start of the next second.

start_of_next_minute

start_of_next_minute(dt: datetime) -> datetime

Start of the next minute.

start_of_next_hour

start_of_next_hour(dt: datetime) -> datetime

Start of the next hour.

start_of_next_day

start_of_next_day(dt: datetime) -> datetime

Start of the next day.

start_of_next_week

start_of_next_week(
    dt: datetime, week_start_day: int = 0
) -> datetime

Start of the next week based on the specified start day.

0=Monday, ..., 6=Sunday. Defaults to 0 (Monday).

start_of_next_month

start_of_next_month(dt: datetime) -> datetime

Start of the next month.

start_of_next_year

start_of_next_year(dt: datetime) -> datetime

Start of the next year.