SQLREC ARM compatibility project

This project builds two replacement wheels, pyfg and graphlearn, with the import surface needed by the pipeline currently generated by SQLREC. SQLREC uses FG_NORMAL, unweighted item: ID features, integer num_buckets, and string hash_bucket_size. The ID path accepts scalar strings and Arrow lists of strings, such as MovieLens genres, as multi-value IDs. SQLREC sets USE_FARM_HASH_TO_BUCKETIZE=true, which this package requires for hash features so bucket IDs remain portable between x86 and ARM.

Supported pyfg calls are FgArrowHandler(..., 1), process_arrow, direct handler calls for default values, reset_executor, set_env, and unset_env. Other feature configurations, handler methods, and graphlearn sampling calls raise UnsupportedAPIError with the unsupported name. The graphlearn modules exist so TorchEasyRec can import its sampler module when no sampler is configured.

Develop and test

Use Python 3.11 on ARM or x86:

python3.11 -m venv .venv
.venv/bin/python -m pip install -e ./packages/pyfg -e ./packages/graphlearn 'pytest>=8,<10'
.venv/bin/python -m pytest -q

On an ARM Docker engine, run the same tests with:

docker build -t sqlrec-arm-compat:test .
docker run --rm sqlrec-arm-compat:test

The differential test skips unless SQLREC_ORIGINAL_PYTHON names a separate x86 Python environment with Alibaba's original pyfg and graphlearn wheels. It compares ID values, row lengths, NumPy dtypes, encoded defaults, and the graphlearn import surface directly:

python3.11 -m venv /path/to/original-x86-venv
/path/to/original-x86-venv/bin/python -m pip install \
  'numpy<2' 'pyarrow==17.0.0' \
  'https://tzrec.oss-accelerate.aliyuncs.com/third_party/pyfg-1.0.5-cp311-cp311-linux_x86_64.whl' \
  'https://tzrec.oss-accelerate.aliyuncs.com/third_party/graphlearn/graphlearn-1.3.8-cp311-cp311-linux_x86_64.whl'
SQLREC_ORIGINAL_PYTHON=/path/to/original-x86-venv/bin/python \
  .venv/bin/python -m pytest -q tests/test_differential.py

The source is pure Python. Its pyfarmhash dependency contains a native extension, so installation on ARM needs a C++ compiler or a prebuilt ARM wheel for that dependency. The optional TorchEasyRec integration test runs when tzrec and its runtime dependencies are installed.

SQLREC image CI

Push this source repository to sqlrec/sqlrec-arm-compat on the master branch. The SQLREC image workflow owns the CI for this project; this repository does not need its own workflow. For each TZRec image run, SQLREC resolves the selected arm_compat_ref branch once, runs the differential tests on a native x86 runner against the original pyfg and graphlearn wheels, then checks out the same revision on a native ARM runner. The ARM job runs this project's tests, builds the pyfg, graphlearn, and pyfarmhash wheels, and installs them in the TZRec ARM image. The manual SQLREC image workflow can select another branch; release builds use master.

Build both replacement wheels from the project root:

.venv/bin/python -m pip wheel --no-deps -w dist ./packages/pyfg ./packages/graphlearn

On an ARM Docker engine, the wheel stage builds all three wheels in one step:

docker build --platform linux/arm64 --target wheels --output type=local,dest=dist .