# Queue Scan

**Queue Scan** is a mechanism that enables real-time synchronization between your **source cloud storage** and the corresponding **dataset**. When enabled, the system actively listens to the configured message queue service — such as **Amazon SQS**, **Kafka**, or **Redis** — for incoming notifications or events related to data changes.

Whenever new data is **added**, **modified**, or **deleted** in the source storage, Queue Scan automatically updates the associated dataset to reflect these changes. This ensures near real-time data consistency without requiring a full data refresh.

Unlike traditional refresh mechanisms, Queue Scan intelligently updates **only the affected directories or files**, leaving the rest of the dataset untouched. This targeted approach significantly reduces processing time and resource consumption.

By leveraging queue-based event notifications, Queue Scan provides:

* **Real-time updates** – keeps datasets synchronized with source changes as they occur.
* **Efficient processing** – updates only new or changed data instead of reloading the entire dataset.
* **Scalability** – works seamlessly with distributed message queues to handle large data volumes.
* **Selective refresh** – Refreshes only the required directories instead of reloading the entire dataset.

<figure><picture><source srcset="/files/tGiIpv8PRrrbs58EDd1C" media="(prefers-color-scheme: dark)"><img src="/files/trAZ0uxA8D3zgDJYJQh8" alt=""></picture><figcaption></figcaption></figure>

#### ⚙️Steps to Enable Queue Scan

{% stepper %}
{% step %}
While creating a new dataset, mark the **Queue Scan** checkbox to activate the feature.
{% endstep %}

{% step %}
From the available options, select the **type of queue service** (e.g., Kafka, Redis, or SQS) that you have configured in **Queue Configs**.
{% endstep %}

{% step %}
Save the dataset configuration — Queue Scan will now automatically listen for changes and update the dataset in real time.
{% endstep %}
{% endstepper %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://supercloudnow.gitbook.io/supercloudnow-docs/undefined/usage-guide/datasets/advance/queue-scan.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.