nlp4j-python

Python tools for NLP4J-related data processing.

This repository contains small, practical command-line tools for working with text, JSONL, and NLP-oriented datasets.

🎯 Overview

nlp4j-python provides lightweight, zero-dependency tools designed for:

• Data processing and ETL pipelines
• NLP dataset preparation and manipulation
• Search index management
• RAG (Retrieval-Augmented Generation) pipeline workflows

🛠️ Tools

JSONL Editor

A zero-dependency CLI tool for editing JSONL files with interactive, batch command, and traditional batch modes.

Location: tools/jsonl-editor

Key Features:

• 🔄 Three execution modes: Interactive REPL, batch commands, traditional CLI
• 🔑 Key operations: Remove, rename, select, add with expression support
• 📝 Safe expression DSL: concat(), coalesce() without using eval()
• 💾 Stream processing: Handle large files efficiently
• 🔒 Safe in-place editing: With optional backup
• 📦 Zero dependencies: Uses only Python standard library

Quick Start (No Installation)

cd tools/jsonl-editor
python nlp4j-jsonl.py

Install as Package

cd tools/jsonl-editor
pip install .

Or after publishing to PyPI:

pip install nlp4j-jsonl

Usage Examples

Interactive Mode:

nlp4j-jsonl
>> load test.jsonl
>> remove key1
>> write output.jsonl
>> exit

Batch Command Mode:

nlp4j-jsonl -c "load test.jsonl; remove key1; write output.jsonl"

Traditional Batch Mode:

nlp4j-jsonl remove --input test.jsonl key1 --inplace
nlp4j-jsonl add --input test.jsonl key4 "concat(key2, ' ', key3)" --inplace

More Examples:

# Chain multiple operations
nlp4j-jsonl -c "load data.jsonl; remove id; rename name full_name; add email 'concat(username, \"@example.com\")'; write result.jsonl"

# Select specific fields
nlp4j-jsonl select --input data.jsonl title body --output filtered.jsonl

# Rename with backup
nlp4j-jsonl rename --input data.jsonl old_key new_key --inplace --backup

📖 Full Documentation

📋 Repository Policy

• ✅ Prefer zero or minimal dependencies
• ✅ Prefer standard Python libraries when practical
• ✅ Keep each tool independently installable
• ✅ Keep tools useful for data processing, NLP, search, and RAG pipelines
• ✅ Maintain simple and predictable behavior
• ✅ Provide both programmatic and CLI interfaces

🔧 Requirements

• Python 3.10 or higher
• No external dependencies (standard library only)

📄 License

Apache License 2.0

🤝 Contributing

Contributions are welcome! Please ensure:

• Tools remain zero or minimal dependency
• Code follows the repository policy
• Tests are included for new features
• Documentation is updated

🗺️ Roadmap

Future tools may include:

• Text preprocessing utilities
• Embedding format converters
• Search index builders
• RAG pipeline helpers

---

Author: Hiroki OYA