Spacy-BIST Parser

Raw text parser based on Spacy and BIST parsers

The parser uses Spacy’s english model for sentence breaking, tokenization and token annotations (part-of-speech, lemma, NER). Dependency relations between tokens are extracted using BIST parser. The BIST parser is described here, and its code is documented here.

Usage

To use the module, import it like so:

from nlp_architect.pipelines.spacy_bist import SpacyBISTParser

Training

By default, the parser uses a pre-trained BIST model and Spacy’s English model (en). A pre-trained BIST model is automatically downloaded (on-demand) to spacy_bist/bist-pretrained/ and then loaded from that directory. To use other models, supply a path or link to each model at initialization (see example below).

For instructions on how to train a BIST model, see BIST documentation. For instructions on how to get spaCy models or how to train a model see spaCy training instructions

Example

parser = SpacyBISTParser(spacy_model='/path/or/link/to/spacy/model', bist_model='/path/to/bist/model')

Parsing

The parser accepts a document as a raw text string encoded in UTF-8 format and outputs a CoreNLPDoc instance which contains the annotations (example output below).

Example

parser = SpacyBISTParser()
parsed_doc = parser.parse(doc_text='First sentence. Second sentence')
print(parsed_doc)

Output

{
    "doc_text": "First sentence. Second sentence",
    "sentences": [
        [
            {
                "start": 0,
                "len": 5,
                "pos": "JJ",
                "ner": "ORDINAL",
                "lemma": "first",
                "gov": 1,
                "rel": "amod",
                "text": "First"
            },
            {
                "start": 6,
                "len": 8,
                "pos": "NN",
                "ner": "",
                "lemma": "sentence",
                "gov": -1,
                "rel": "root",
                "text": "sentence"
            },
            {
                "start": 14,
                "len": 1,
                "pos": ".",
                "ner": "",
                "lemma": ".",
                "gov": 1,
                "rel": "punct",
                "text": "."
            }
        ],
        [
            {
                "start": 16,
                "len": 6,
                "pos": "JJ",
                "ner": "ORDINAL",
                "lemma": "second",
                "gov": 1,
                "rel": "amod",
                "text": "Second"
            },
            {
                "start": 23,
                "len": 8,
                "pos": "NN",
                "ner": "",
                "lemma": "sentence",
                "gov": -1,
                "rel": "root",
                "text": "sentence"
            }
        ]
    ]
}

References

[1]	Kiperwasser, E., & Goldberg, Y. (2016). Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations. Transactions Of The Association For Computational Linguistics, 4, 313-327. https://transacl.org/ojs/index.php/tacl/article/view/885/198