English | 中文

A recommendation engine that supports SQL-based development. The goal is to enable data scientists, including data analysts, data engineers, and backend developers, to quickly build production-ready recommendation systems. The system architecture is shown in the figure below. SQLRec encapsulates underlying component access, model training, inference, and other processes using SQL, allowing upper-level recommendation business logic to be described using only SQL.

SQLRec has the following features:

• Cloud-native, comes with minikube-based deployment scripts for one-click deployment of the SQLRec system and related dependency services
• Extended SQL syntax, making it possible to describe recommendation system business logic using SQL
• Implemented an efficient SQL execution engine based on Calcite, meeting the real-time requirements of recommendation systems
• Built on existing big data ecosystem, easy to integrate
• Easy to extend, supports custom UDFs, Table types, and Model types

For detailed information, refer to the SQLRec User Manual.

Run the dependency-free Docker demo:

docker run --rm -d --name sqlrec-demo \
  -p 30000:30000 \
  -p 30001:30001 \
  sqlrec/sqlrec-demo:latest

The demo includes CSV data for five users with three interests each, five categories (pc, phone, book, sports, home), and 25 hot items. Each process loads it into memory on first use. Run cli.sh inside the container directly from the host to execute SQL at the sqlrec> prompt; end each statement with a semicolon:

docker exec -it sqlrec-demo /app/cli.sh

show tables;
select * from demo_user_interest_category;

cache table quick_start_user as
select cast(1000001 as bigint) as user_id;

call demo_rec(quick_start_user);

quick_start_user supplies the input row to the recommendation function, and CALL returns its recommendations. Press Ctrl+D to leave the SQL CLI. Its in-memory data is separate from the HTTP service's data. From the host, you can also call the built-in recommendation API with user IDs 1000001 through 1000005:

curl -X POST http://localhost:30001/api/v1/demo_rec \
  -H "Content-Type: application/json" \
  -d '{"data":{"user_info":[{"user_id":1000001}]}}'

The API records exposures in memory and excludes previously returned items. After the sample items are exhausted, restart the container to reset the demo.

In the Docker demo, tables, SQL functions, and APIs are loaded from .sql files under SQL_SCHEMA_DIR. For example, prepare this directory on the host:

sql/
├── table/hot_item.sql
├── function/recommend.sql
└── api/recommend.sql

Define a table in sql/table/hot_item.sql:

CREATE TABLE hot_item (
  item_id BIGINT,
  score FLOAT,
  PRIMARY KEY (item_id) NOT ENFORCED
) WITH (
  'connector' = 'filesystem'
);

Define a SQL function in sql/function/recommend.sql:

CREATE OR REPLACE SQL FUNCTION recommend;

DEFINE INPUT TABLE user_info (
  user_id BIGINT
);

CACHE TABLE result AS
SELECT item_id, score
FROM hot_item
ORDER BY score DESC
LIMIT 10;

RETURN result;

Publish the function in sql/api/recommend.sql:

CREATE OR REPLACE API recommend WITH recommend;

Stop the built-in demo, then mount the complete definition directory and restart SQLRec:

docker stop sqlrec-demo

docker run --rm -d --name sqlrec-custom \
  -p 30000:30000 \
  -p 30001:30001 \
  -v "$(pwd)/sql:/workspace/sql:ro" \
  -e SQL_SCHEMA_DIR=/workspace/sql \
  sqlrec/sqlrec-demo:latest

Local metadata mode does not accept DDL through the CLI or /sql/v1. Edit the SQL files and restart the container whenever a definition changes. Filesystem table data is held in memory and is intended only for demos and tests.

Insert data and call the newly published API:

curl -X POST http://localhost:30001/sql/v1 \
  -H "Content-Type: application/json" \
  -d '{"sqls":["insert into hot_item values (1001, 0.9), (1002, 0.8)"]}'

curl -X POST http://localhost:30001/api/v1/recommend \
  -H "Content-Type: application/json" \
  -d '{"data":{"user_info":[{"user_id":1000001}]}}'

Open http://localhost:30001/ui/static/index.html to inspect the loaded tables, functions, APIs, and execution DAG. Stop and remove the demo when finished:

docker stop sqlrec-custom

For more CLI, data-loading, and API examples, see the Docker Quick Start.

Compared with the Docker demo's local metadata mode, the complete service mode primarily adds persistent, mutable metadata. Through Beeline, JDBC, or another Hive Thrift client, you can execute and retain management statements such as:

• CREATE TABLE, CREATE SQL FUNCTION, and CREATE API;
• CREATE MODEL, TRAIN MODEL, and EXPORT MODEL;
• CREATE SERVICE and other model-service lifecycle operations.

The bundled Minikube scripts are intended for development and testing. System requirements and component versions evolve with the scripts; use the current deploy/ configuration as authoritative. See Service Deployment for the complete deployment process and production considerations.

Versions before 1.0 are beta versions, not recommended for production use, and interface compatibility is not guaranteed. There is no planned release date yet. It will be released after the following features are completed:

• Comprehensive unit test, integration test, and effectiveness test coverage
• Code quality optimization, many details still need to be polished
• Support for degradation and timeout configuration
• Complete version management method, easy to roll back to previous versions
• Metric monitoring system improvement
• C++ model serving

• Frontend UI for viewing current execution DAG, SQL code, statistics, etc.
• Further optimize SQL syntax compatibility and runtime performance
• More ready-to-use UDFs, models, etc.
• Support for more external data sources, such as JDBC, MongoDB, etc.
• Tensorboard visualization of model training process
• GPU training and inference support
• Support for authentication and authorization
• Best practice tutorials, including search, recommendation, etc.