> For the complete documentation index, see [llms.txt](https://docs.bcbi.brown.edu/codiac-for-health/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bcbi.brown.edu/codiac-for-health/foundations/knowledge-discovery.md).

# Knowledge Discovery

One of the true powers behind the collection of EHR data is that large health datasets may contain patterns that can improve our understanding of diseases, treatments and healthcare delivery. However, collecting data itself does not automatically produce useful knowledge. Knowledge discovery in databases (KDD) is the broader process of finding insights from patterns that can be observed in data \[1]. 

Knowledge discovery is sometimes used as a synonym for data mining, but the terms have different meanings. Data mining is described as the application of specific algorithms or computational methods for extracting patterns from data and is a specific step in the knowledge discovery process. KDD encompasses the overall process which includes selection and preparing of data, applying data mining methods, interpreting the results, and determining whether the discovered patters reepsent useful knowledge \[1]. 

<h2 align="center">Overview of Disease Knowledge Discovery Process</h2>

<img src="https://2177175876-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FkVnTNRf2wRJPzjJa6nlU%2Fuploads%2Fopp38VlwCCWS9AOThWwA%2Funknown.png?alt=media&amp;token=40369c94-4b45-4984-a93e-7f6df655e355" alt="" height="406" width="854">

ADT = Admit Discharge Transfer, CDR = Clinical Data Repository, EHR = Electronic Health Record, EDW = Enterprise Data Warehouse.

Overview of knowledge discovery using EHR data. Data from clinical and external sources are selected, preprocessed, transformed, and analyzed to identify potentially meaningful patterns. These patterns must then be interpreated and evaluated before they can contribute to knowledge. Data standards, privacy, security, information exchange, legislation all affect this process \[2].

## The Knowledge Discovery Process \[2]

The following section is adapted from "Mining the electronic health record for disease knowledge"

### Understand the Problem

The process begins by identifying a potential area of interest to explore. Researchers must determine what they hope to learn, what prior knowledge already exists, and how the resulting information might be used.&#x20;

### Data Selection

Researchers identify a target dataset by creating a cohort definition which includes exclusion/inclusion criteria and then extracting that data. Criteria that can be used for cohort identification are patient demographics, encounters, diagnoses, procedures, medications, and laboratory results.&#x20;

EHR-related data can come from:

* Electronic health records
* Clinical data repositories
* Enterprise data warehouses
* Laboratory and pharmacy systems
* Billing and claims data
* Patient registries
* Biobanks
* Biomedical literature

The selected data may contain structured, unstructured and semi structured data types. &#x20;

### Data Extraction

Once the cohort has been identified, researchers determine which data are needed, what information is available, and how the extracted dataset will be obtained and securely stored.&#x20;

The formal data request should specify the type of dataset requested (aggregate, deidentified, or identified) as well as the required data types, elements, and restrictions. Restrictions may include limiting records to a defined period, including only active problem-list entries, or retrieving only the most recent clinical note. The sensitivity and identification level of the resulting dataset determine the computing environment in which it must be stored and analyzed.

These sources provide different perspectives on patient health. Billing diagnoses may associate multiple clinical codes with an encounter, while problem-list entries may include a diagnosis name, code, status, and onset date. Clinical notes contain unstructured information accompanied by metadata such as the note type and author. Structured and unstructured data may be analyzed separately or combined to provide a more complete representation of the study cohort.

After extraction, researchers should verify that the delivered data elements, date ranges, restrictions, and cohort records match the approved request \[2].

### Preprocess the Data

Once the dataset is obtained, preprocessing prepares it for analysis. Major tasks may include deidentification, data cleaning, standardization, integration, and enrichment.

* Deidentification: Remove protected health information when required for the approved research use and computing environment.
* Data cleaning: Identify and address missing, invalid, inconsistent, duplicate, or implausible values. Decisions to retain, correct, or remove these values should depend on the research question and protocol.
* Standardization and integration: Convert data into consistent formats and reconcile differences among sources. This may include standardizing dates, separating concatenated values, and mapping local or source codes to standardized terminologies.
* Data enrichment: Add useful information through external resources or computational methods. For example, natural language processing may extract and encode clinical concepts from notes, while resources such as the UMLS may help connect equivalent concepts across different vocabularies.

These steps improve the consistency and usability of the dataset while helping researchers combine structured and unstructured information from multiple clinical sources \[2].

### Transform the Data

### Apply Data-Mining Methods

### Interpret and Evaluate the Patterns

### Knowledge to Wisdom

## References

1. Fayyad U, Piatetsky-Shapiro G, Smyth P. [From data mining to knowledge discovery in databases.](https://doi.org/10.1609/aimag.v17i3.1230) *AI Magazine*. 1996;17(3):37–54.&#x20;
2. Chen ES, Sarkar IN. [Mining the electronic health record for disease knowledge. ](https://doi.org/10.1007/978-1-4939-0709-0_15)In: Kumar V, Tipney H, editors. *Biomedical Literature Mining*. Methods in Molecular Biology. Vol. 1159. Humana Press; 2014:269–286.&#x20;

<br>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.bcbi.brown.edu/codiac-for-health/foundations/knowledge-discovery.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.