The 15th IEEE International Conference on Knowledge and Systems Engineering (KSE2023) October 18-20, 2023 | Ha Noi, Vietnam
14th IEEE International Conference on Knowledge and Systems Engineering (KSE 2022)
October 19-21, 2022 | Nha Trang, Vietnam

Workshop Session: Automated Legal Question Answering Competition (ALQAC 2022)

Automated Legal Question Answering Competition (ALQAC)
Run in association with the International Conference on Knowledge and Systems Engineering

ALQAC-2022 CALL FOR TASK PARTICIPATION

ALQAC-2022 Workshop: October 19-21, 2022
ALQAC-2022 Registration due: May 30, 2022

Sponsored by

Japan Advanced Institute of Science and Technology (JAIST)

University of Engineering and Technology (VNU-UET)

Zalo Vietnam

Overview

As an associated event of KSE 2022, we are happy to announce the 2nd Automated Legal Question Answering Competition (ALQAC 2022). ALQAC includes 2 tasks: (1) Legal Document Retrieval, and (2) Legal Question Answering. For the competition, we introduce the Legal Question Answering dataset – a manually annotated dataset based on well-known statute laws in the Vietnamese Language. Through the competition, we aim to develop a research community on legal support systems.

Prize

We have 2 tasks. For each task, we have

One First prize, the winning team will receive 250$.

One Second prize, the runner-up team will receive 150$.

Two Third prizes, each receives 50$.

In total, the prize is 1000$. Besides, the winning team of each task will have the KSE conference fee of the presenter covered.

Dataset

The dataset file formats are shown via examples as follows.

[
    {
      "id": "45/2019/QH14",
      "articles": [
            {
                "text": "The content of legal article",
                "id": "1"
            }
        ]
    }
]
[
    {
        "question_id": "q-1",
        "text": "The content of question or statement",
        "answer": <span of text>,
        "relevant_articles": [
            {
                "law_id": "45/2019/QH14",
                "article_id": "1"
            }
        ]
    }
]

Tasks

Tasks Description
Task 1: Legal Document Retrieval

Task 1’s goal is to return articles that are related to a statement. An article is considered “relevant” to a statement iff the statement can be answered/verified by the article.

Specifically, the input samples consist of:

[
    {
        "question_id": "q-1",
        "text": "The content of question or statement"
    }
]

The system should retrieve all the relevant articles as follows:

[
    {
        "question_id": "q-1",
        "text": "The content of question or statement",
        "relevant_articles": [
            {
                "law_id": "45/2019/QH14",
                "article_id": "1"
            }
        ]
    }
]

Note that “relevant_articles” are the list of all relevant articles for the questions/statements.

The evaluation methods are precision, recall, and F2-measure as follows:

Precisioni = the number of correctly retrieved articles of query ith
             the number of retrieved articles of query ith ,


Recalli = the number of correctly retrieved cases(paragraphs) of query ith
           the number of relevant cases(paragraphs) of query ith ,

F2i= (5 x Precisioni x Recalli)
       (4Precisioni + Recalli)

F2 = average of (F2i)

In addition to the above evaluation measures, ordinal information retrieval measures such as Mean Average Precision and R-precision can be used for discussing the characteristics of the submission results.

In ALQAC 2022, the method used to calculate the final evaluation score of all queries is macro-average (evaluation measure is calculated for each query and their average is used as the final evaluation measure) instead of micro-average (evaluation measure is calculated using results of all queries).

Task 2: Legal Question Answering
Factoid Question

Given a legal question, the goal is to produce a span of text which determines the exact answer to the question.  That is, for each question-answer pair, if the character sequence of the model’s prediction exactly matches the character sequence of the true answer, the predicted answer is considered correct; otherwise, it is considered incorrect.

Specifically, the input samples consist of the question as follows:

[
    {
        "question_id": "q-1",
        "text": "The content of question",
    }
]

The system should answer whether the question/statement is a span via “answer” in JSON format as follows:

[
    {
        "question_id": "q-1",
        "text": "The content of question",
        "answer": <span of text>
    }
]

The evaluation measure will be accuracy, with respect to whether the question was correctly confirmed:

Accuracy = (the number of queries which were correctly confirmed as true or false)
(the number of all queries)

In addition to the above evaluation measures, ordinal information retrieval measures such as Mean Average Precision and R-precision can be used for discussing the characteristics of the submission results.

Note: The output submitted by teams will be published to a public GitHub repository so that legal and AI experts can refer to this information for analysis purposes. Expert evaluation is the official metric to decide the systems’ performance of the teams.

Submission Details

Participants are required to submit a paper on their method and experimental results. The participants have to submit via e-mail files containing the results of each task, separately. For each task, participants can submit a maximum of 3 results corresponding to 3 different settings/methods. The code for evaluation is published on Google Colab (https://colab.research.google.com/drive/17tEVE2C56kHXxdfeooBPSA5mG4vZVTxf).

In this framework, we defined the mentioned input/output data structure and evaluation methods for all 2 tasks.

Note: Participants have the responsibility to warranty their result files to follow the required format.

These examples are outputs of 2 tasks that the participants’ model needs to generate for evaluation methods:

Task 1: Legal Document Retrieval
[
    {
        "question_id": "q-193",
        "relevant_articles": [
            {
                "law_id": "100/2015/QH13",
                "article_id": "177"
            }
        ]
    },
    ...
]
Task 2: Legal Question Answering
[
    {
        "question_id": "q-193",
        "answer": <span of text>
    },
    ...
]

At least one of the authors of an accepted paper has to present the paper at the ALQAC workshop of KSE 2022.

The papers authored by the task winners will be included in the main KSE 2022 proceedings if ALQAC organizers admit the paper novelty after the review process.

Papers should conform to the standards set out on the KSE 2022 webpage (section Submission) and be submitted to EasyChair.

Application Details

Potential participants in ALQAC-2022 should respond to this call for participation by submitting an application via: tinyurl.com/ALQAC2022Registration.

Schedule (Timezone: AOE)

May 6, 2022: Call for participation
June 1, 2022: Training data release
June 20, 2022: Testing data release
August 15, 2022: Submission deadline for Task 1 & 2
August 20, 2022: Announcements of rankings/assessments
August 31, 2022: Paper/Technical report of your methods for the tasks
September 7, 2022: Notification of Acceptance
September 14, 2022: Camera-ready Submission
September 24, 2022: KSE Registration Deadline
October 19-21, 2022: KSE 2022

Task winners

– Task 1 (this task has 2 first prizes and 2 third prizes):

Team Miko

First prize

Team QA12

First prize

Team NG Team

Third prize

Team VLH

Third prize

– Task 2 (this task has 1 first prize, 1 second prize, and 1 third prize):

Team QA12

First prize

Team VLH

Second prize

Team Miko

Third prize

Questions and Further Information

Email: chau.nguyen@jaist.ac.jp with the subject [ALQAC-2022] <Content>

Program Committee