hankcs/HanLP

HanLP is a comprehensive natural language processing library that provides a modular, pipeline-based architecture for building and training NLP models. It supports a wide range of tasks including tokenization, part-of-speech tagging, named entity recognition, dependency and constituency parsing, semantic role labeling, and text classification. The system offers both PyTorch and TensorFlow backends, enabling users to construct complex workflows by chaining components, and includes utilities for model compression, multi-task learning, and evaluation metrics.

The project underwent a significant architectural shift towards a modular, component-based design, introducing a unified pipeline system and abstract base classes for NLP components. This period also focused on expanding framework support by adding comprehensive TensorFlow implementations for existing PyTorch models, alongside building out a rich ecosystem of layers, metrics, and dataset loaders.

28 changes

This period was defined by the introduction of a comprehensive Multi-Task Learning (MTL) framework, enabling joint training across diverse NLP tasks such as parsing, tagging, and NER. The codebase expanded significantly with new components for knowledge distillation, semantic textual similarity, and masked language modeling, alongside extensive demo scripts for multiple languages.

27 changes

hankcs/HanLP was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point the surveyor at a repository you know and see whether you agree with it.