Xây dựng đường ống RAG trong R với ragnar và ellmer: Từ nhúng dày đặc đến sinh câu trả lời có căn cứ

1library(quanteda)
2library(quanteda.textstats)
3
4# Hai câu ngữ nghĩa giống hệt nhau nhưng từ vựng hoàn toàn khác
5sentences <- c(
6  doc1 = "The borrower defaulted on their loan obligations.",
7  doc2 = "The client failed to meet repayment commitments.",
8  doc3 = "The borrower defaulted on their loan obligations."  # giống hệt doc1
9)
10
11corp <- corpus(sentences)
12
13sim_tfidf <- corp |>
14  tokens(remove_punct = TRUE) |>
15  tokens_tolower() |>
16  tokens_remove(stopwords("english")) |>
17  dfm() |>
18  dfm_tfidf() |>
19  textstat_simil(method = "cosine", margin = "documents")
20
21as.matrix(sim_tfidf)

1     doc1 doc2 doc3
2doc1    1    0    1
3doc2    0    1    0
4doc3    1    0    1

1# Nhúng cùng ba câu bằng mô hình Ollama cục bộ
2embed_fn <- embed_ollama(model = "nomic-embed-text")
3
4embeddings <- embed_fn(as.character(sentences))
5
6# Hàm tính cosine similarity
7cosine_sim <- function(a, b) sum(a * b) / (sqrt(sum(a^2)) * sqrt(sum(b^2)))
8
9sim_12 <- cosine_sim(embeddings[1, ], embeddings[2, ])
10sim_13 <- cosine_sim(embeddings[1, ], embeddings[3, ])
11
12tibble(
13  pair         = c("doc1 vs doc2 (khác từ, cùng ý)",
14                   "doc1 vs doc3 (giống hệt)"),
15  tfidf_cosine = c(0.000, 1.000),
16  dense_cosine = round(c(sim_12, sim_13), 3)
17)

1# A tibble: 2 × 3
2  pair                                 tfidf_cosine dense_cosine
3  <chr>                                       <dbl>        <dbl>
41 doc1 vs doc2 (khác từ, cùng ý)                  0        0.733
52 doc1 vs doc3 (giống hệt)                        1        1

1credit_docs <- tibble(
2  id = 1:15,
3  source = c(
4    rep("IFRS 9", 5),
5    rep("Basel III", 5),
6    rep("Credit Policy", 5)
7  ),
8  text = c(
9    # Trích dẫn IFRS 9
10    "A financial asset is in Stage 1 if there has been no significant increase in credit risk since initial recognition.",
11    "Stage 2 classification is triggered when there is a significant increase in credit risk relative to initial recognition, even if no actual default has occurred.",
12    "Stage 3 assets are those for which objective evidence of impairment exists, including default events or bankruptcy proceedings.",
13    "The 12-month Expected Credit Loss is the portion of lifetime ECL resulting from default events possible within 12 months of the reporting date.",
14    "Lifetime ECL represents the expected credit losses resulting from all possible default events over the expected life of the financial instrument.",
15
16    # Trích dẫn Basel III
17    "The Capital Conservation Buffer requires banks to hold a minimum Common Equity Tier 1 ratio of 2.5% above the regulatory minimum.",
18    "Counterparty credit risk arises from the failure of a counterparty to meet its contractual payment obligations.",
19    "The Liquidity Coverage Ratio ensures banks hold sufficient high-quality liquid assets to survive a 30-day stress scenario.",
20    "Pillar 2 requirements allow supervisors to impose additional capital requirements above the Pillar 1 minimum based on institution-specific risk profiles.",
21    "The Net Stable Funding Ratio measures the proportion of long-term assets funded by stable sources of funding over a one-year horizon.",
22
23    # Chính sách tín dụng
24    "Loan applicants with a debt-to-income ratio exceeding 45% are automatically declined under the current credit policy.",
25    "A payment more than 90 days past due triggers a non-performing loan classification and requires full provisioning.",
26    "Collateral revaluation must occur at minimum annually for secured exposures exceeding the materiality threshold.",
27    "The credit scorecard is recalibrated quarterly to reflect shifts in the applicant population and macroeconomic conditions.",
28    "Early arrears management is initiated when a borrower misses two consecutive scheduled payments."
29  )
30)
31
32credit_docs

1# Thực tế với văn bản lớn:
2# doc_text |> ragnar::markdown_chunk(level = 2)  # tách theo tiêu đề ##
3
4# Với bộ ngữ liệu này, thêm metadata cho từng chunk
5chunks <- credit_docs |>
6  mutate(
7    chunk_id   = paste0("chunk_", id),
8    char_count = nchar(text)
9  )
10
11chunks |> select(chunk_id, source, char_count, text)

1# Tạo kho trong bộ nhớ (dùng đường dẫn file để bền vững)
2store <- ragnar_store_create(
3  location = ":memory:",
4  embed    = embed_ollama(model = "nomic-embed-text"),
5  version  = 1
6)
7
8# Chèn tài liệu
9ragnar_store_insert(
10  store,
11  chunks |> mutate(origin = source, hash = chunk_id) |> select(origin, hash, text)
12)
13
14# Xây dựng chỉ mục vectơ
15ragnar_store_build_index(store)

1# Lấy toàn bộ chunk đã lưu từ DuckDB
2all_embeddings <- DBI::dbReadTable(store@con, "chunks")
3
4# PCA giảm chiều xuống 2D
5emb_matrix <- all_embeddings$embedding
6pca_result <- prcomp(emb_matrix, center = TRUE, scale. = FALSE)
7
8tibble(
9  source = chunks$source,
10  label  = stringr::str_trunc(chunks$text, 50),
11  PC1    = pca_result$x[, 1],
12  PC2    = pca_result$x[, 2]
13) |>
14  ggplot(aes(x = PC1, y = PC2, colour = source, label = label)) +
15  ggrepel::geom_text_repel(size = 2.5, max.overlaps = 8) +
16  geom_point(size = 3, alpha = 0.8) +
17  labs(
18    title    = "Không gian nhúng: Bộ ngữ liệu Chính sách tín dụng",
19    subtitle = "Shoot PCA của nhúng nomic-embed-text",
20    colour   = "Nguồn"
21  ) +
22  theme_minimal()

1query <- "What triggers a move to Stage 2 under IFRS 9?"
2
3# Kết hợp (VSS + BM25)
4results_hybrid <- ragnar_retrieve(store, query, top_k = 2)
5
6# Chỉ VSS
7results_vss  <- ragnar_retrieve_vss(store, query, top_k = 2)
8
9# Chỉ BM25
10results_bm25 <- ragnar_retrieve_bm25(store, query, top_k = 2)
11
12bind_rows(
13  results_vss    |> mutate(method = "VSS",    rank = row_number()),
14  results_bm25   |> mutate(method = "BM25",   rank = row_number()),
15  results_hybrid |> mutate(method = "Hybrid", rank = row_number())
16) |>
17  select(method, rank, origin, text) |>
18  arrange(method, rank)

1# A tibble: 7 × 4
2  method  rank origin        text                                               
3  <chr>  <int> <chr>         <chr>                                              
41 BM25       1 IFRS 9        Stage 2 classification is triggered when there is …
52 BM25       2 Credit Policy A payment more than 90 days past due triggers a no…
63 Hybrid     1 IFRS 9        Stage 2 classification is triggered when there is …
74 Hybrid     2 IFRS 9        A financial asset is in Stage 1 if there has been …
85 Hybrid     3 Credit Policy A payment more than 90 days past due triggers a no…
96 VSS        1 IFRS 9        Stage 2 classification is triggered when there is …
107 VSS        2 IFRS 9        A financial asset is in Stage 1 if there has been …

1# Khởi tạo chat (Ollama cục bộ)
2chat <- chat_ollama(
3  model  = "llama3.1",
4  system = "You are a credit risk analyst. Answer questions using only the
5            information retrieved from the document store. If the answer is
6            not in the retrieved context, say so explicitly."
7)
8
9# Đăng ký kho vectơ như một công cụ truy xuất
10ragnar_register_tool_retrieve(chat, store)

1query <- "What triggers a loan to move to Stage 2 under IFRS 9?"
2
3# Có RAG (đã đăng ký công cụ truy xuất)
4answer_rag <- chat$chat(query)

1The information retrieved does not specifically provide what triggers a loan to
2move to Stage 2 under IFRS 9. However, it suggests that the criteria for moving
3to Stage 2 might be closely related to an increase in default likelihood, as 
4stated in one of the search results referencing "a significant increase in 
5credit risk relative to initial recognition".
6
7Based solely on this result we cannot definitively state what triggers a loan 
8under Stage 2 yet.

1# Không RAG (chat mới, không công cụ, không ngữ cảnh)
2chat_naive   <- chat_ollama(model = "llama3.1")
3answer_naive <- chat_naive$chat(query)

1Under International Financial Reporting Standards (IFRS) 9, a loan is 
2classified as an impaired asset and moved to Stage 2 when it experiences a 
3significant increase in credit risk since the initial recognition of the loan.
4
5To be specific, a loan moves to Stage 2 if there are indicators that suggest a 
6significant deterioration in the debtor's ability to repay the loan, which is 
7considered to be more than insignificant. This can be due to various factors 
8such as:
9
10* A decline in payments or any other sign that the borrower may default
11* A material increase in off-balance-sheet credit risk, such as a guarantee 
12from a high-risk counterparty
13* A significant fall in credit ratings of the debtor
14
15Stage 2 is also known as "probable losses" (under IFRS 9) or "substandard 
16loans" under many banking regulator's rules.

1tibble(
2  approach = c("RAG (có căn cứ)", "Không RAG (nguyên sinh)"),
3  answer   = c(answer_rag, answer_naive)
4)

1questions <- c(
2  "What triggers an IFRS 9 Stage 2 migration?",
3  "When is a loan classified as non-performing?",
4  "What is the Capital Conservation Buffer requirement?",
5  "How often is the credit scorecard recalibrated?"
6)
7
8# Hàm hỗ trợ: chạy truy vấn có căn cứ và trả về chuỗi
9ask_rag <- function(question) {
10  ch <- chat_ollama(
11    model  = "llama3.1",
12    system = "Answer using only retrieved context. Be concise."
13  )
14  ragnar_register_tool_retrieve(ch, store)
15  ch$chat(question)
16}
17
18# Hỏi đáp hàng loạt dưới dạng tibble gọn gàng
19qa_results <- tibble(
20  question = questions,
21  answer   = purrr::map_chr(questions, ask_rag)
22)

1A significant increase in credit risk relative to initial recognition triggers 
2an IFRS 9 Stage 2 migration.

1According to the information retrieved, a loan is classified as non-performing 
2when it has a payment that is more than 90 days past due.

1The Capital Conservation Buffer requirement is to hold a minimum Common Equity 
2Tier 1 ratio of 2.5% above the regulatory minimum, as per Basel III 
3regulations. This means that banks must maintain a strong capital base to 
4mitigate risks and ensure their financial stability.

1The credit scorecard is recalibrated quarterly.

1qa_results

1# A tibble: 4 × 2
2  question                                             answer                   
3  <chr>                                                <chr>                    
41 What triggers an IFRS 9 Stage 2 migration?           A significant increase i…
52 When is a loan classified as non-performing?         According to the informa…
63 What is the Capital Conservation Buffer requirement? The Capital Conservation…
74 How often is the credit scorecard recalibrated?      The credit scorecard is …