Collector
1. Overview
The Collector module in Sumo Ledger manages data collection pipelines — the mechanisms for ingesting data from external sources into the ledger system and exporting data to external destinations. Collectors are organized into two categories: Sinks (data destinations/outputs) and Sources (data origins/inputs).
Collectors automate the flow of data between Sumo Ledger and external systems such as file storage, databases, APIs, or messaging queues.
Navigation: Collector (main sidebar menu)
URL: https://las.ledger.raralabs.dev/collector
2. Collector Page
The Collector page has two tabs: Sink and Source.
2.1 Sink Tab
The Sink tab manages data export destinations — where Sumo Ledger sends data to.
Table Columns:
| Column | Description |
|---|---|
| Name | The name of the sink collector. |
| Type | The type of destination (e.g., file system, API endpoint, database, message queue). |
| Shared Folder | The shared folder path used by this sink, if applicable. |
Controls:
- Filter: Dropdown filter for narrowing results.
- Search: Free-text search across sink entries.
- View: Toggle table layout.
2.2 Source Tab
The Source tab manages data ingestion sources — where Sumo Ledger pulls data from.
Table Columns:
| Column | Description |
|---|---|
| Name | The name of the source collector. |
| Type | The type of data source (e.g., file system, API, SFTP, database). |
| Active | Whether this source collector is currently active and running. |
| Cron Expression | The schedule on which this source collector runs, expressed as a cron expression (e.g., 0 0 * * * for daily at midnight). |
3. How Collectors Work
Source Collectors
Source collectors run on a defined schedule (cron expression) to fetch data from external sources. When triggered, a source collector:
- Connects to the configured external source.
- Retrieves files or data matching the configured pattern.
- Passes the data to the configured process for ingestion into the ledger.
Sink Collectors
Sink collectors export data from Sumo Ledger to external destinations. They can be triggered by process completion events or run on a schedule.
Collector Logs
The Collector logs section tracks the execution history of collectors, including status, file counts, and error messages. See the Collector Logs guide for details.
4. Best Practices
- Set up source collectors with appropriate cron schedules. Ensure the schedule aligns with when new data is expected from the external source.
- Monitor the Collector Logs regularly. Failed collector runs can indicate connectivity issues or changes in the source data format.
- Use descriptive names for collectors. Names should reflect the data source or destination (e.g.,
daily_cbs_extract,monthly_report_export). - Test collectors after configuration changes. Any changes to file patterns, connection details, or schedules should be verified by checking the next execution in the Collector Logs.
- Deactivate unused collectors. If a data source is no longer needed, set the collector to inactive rather than deleting it, in case you need to re-enable it later.