Salesforce File Storage limits are easy to breach as organizations accumulate sales notes, call transcripts, and file attachments over time. Because ContentNote and ContentVersion objects quickly consume allocated file storage, engineering teams often face a tradeoff between purchasing costly additional storage or permanently deleting historical CRM data.
A more sustainable approach is to offload cold CRM content into an enterprise data platform while preserving data integrity and queryability.
In this post, we’ll build an automated pipeline to extract Salesforce ContentNotes and attachments into Databricks Delta Lake. We’ll cover:
- Handling Salesforce API limits and Base64 ingestion via a dual REST and Lakeflow Connect strategy.
- Cleaning and structuring unstructured notes into Delta Medallion layers (Bronze Silver Gold).
- Applying Databricks AI Functions and Vector Search to keep archived notes searchable for downstream analytics.
- Safely executing hard deletes in Salesforce to release storage space without losing data lineage.
KEY IDEA | THE IDEA IN ONE LINE
Preserve → model → verify → release → manage. Databricks is an archive, not a mirror.
Why Notes and Files Hurt
- Storage Limit : Salesforce's current allocations page shows 10 GB of file storage per org for Professional, Enterprise, Unlimited and similar editions (1 GB for Essentials and Starter). The only number that matters is the one in your org
- Notes can eat your file storage : Salesforce separates data storage from file storage. While legacy Notes count toward data storage, Enhanced Notes are stored through ContentNote/ContentVersion.
- Storage priced for cold data : Salesforce doesn't publish a list price for storage beyond your allocation, so your cost depends on your contract. Industry write-ups commonly cite about $5 per GB-month for extra file storage. At that rate, 1 TB costs roughly $5,000 a month. Cloud object storage prices by access pattern instead: AWS lists S3 Glacier Deep Archive at $0.00099 per GB-month, about $1 per TB-month. Even S3 Standard at $0.023 per GB-month is over 200 times cheaper than that $5 figure. Check your own contract rate, then compare.
| S3 storage class | Price / GB-month | 500 GB / month | 500 GB / year |
|---|---|---|---|
| Standard (first 50 TB) | $0.023 | $11.50 | $138.00 |
| Glacier Instant Retrieval | $0.004 | $2.00 | $24.00 |
| Glacier Deep Archive | $0.00099 | $0.495 | $5.94 |
How Salesforce Stores a Note
Four objects do the work, and getting the relationships between them right decides whether the archive is complete and correct.
- ContentDocument is the file. It represents the document and acts as the parent for its versions.
- ContentVersion holds the actual version. This is where file metadata and the content reference live. Enhanced Notes also use ContentVersion.
- ContentDocumentLink tells you where the file belongs. A single file can be linked to multiple Salesforce records, so never assume a simple ParentId relationship.
- ContentNote identifies Enhanced Notes. Notes use the Salesforce content model too, which means they can consume file storage.
The Pipeline
Figure 1 — Clean Enterprise Medallion Architecture: Salesforce Extraction → Bronze Auto Loader → Silver Decoded → Gold AI Functions → Databricks AI Search.
Lakeflow Connect or REST? Use both, for different jobs
The goal is simple: get Salesforce content into Databricks without losing relationships, versions, or file bytes.
That requires two extraction paths. Lakeflow Connect handles the structured Salesforce objects,
while REST/Bulk API 2.0 retrieves the file content that the connector cannot ingest.
The flow looks like this:
Salesforce → Bronze → Silver → Gold → AI Search
Databricks' Salesforce connector (Lakeflow Connect) is excellent for structured objects, but its documented limits shape this design:
| Need | Lakeflow Connect | REST / Bulk API 2.0 |
|---|---|---|
| VersionData, Content (base64) | Not ingested: base64 fields are left out of the destination schema | Required: the way to get the bytes |
| ContentDocumentLink | Listed among objects with query restrictions; ingestion needs Query All Files | Query by document ID in batches, or unfiltered with Query All Files on API 59+ |
| Deleted records | Soft deletes flow into Bronze at the next sync; hard deletes need a full refresh | You control it: append-only, no delete logic |
| File metadata | Supported as structured fields | Also available in the same query as the bytes |
WARNING | WHY THIS MATTERS FOR AN ARCHIVE
Lakeflow Connect mirrors soft deletes into its Bronze tables. If you ingest ContentDocumentLink with it and later delete documents in Salesforce, the link rows can disappear with them. For archive data, land your own append-only tables, or copy managed tables into an archive table before cleanup.
Extraction
1.Set Up REST Authentication: To make REST/Bulk API calls from Databricks, Salesforce needs to trust and authenticate the pipeline. Set up a dedicated integration user with an External Client App using the OAuth 2.0 Client Credentials Flow.
- Enable the required API scope and assign the integration user as Run As.
- Grant API Enabled and Query All Files to the integration user.
- Store the client ID and secret securely in a Databricks secret scope.
- Keep the extraction identity read-only; use a separate identity for deletion.
This gives the Databricks pipeline a controlled, repeatable way to authenticate and extract Salesforce content.
Before making Salesforce REST calls, the pipeline first exchanges the client ID and secret for an OAuth access token. That token, along with the Salesforce instance URL and API version, is then used to authenticate subsequent API requests.
2. Extract and Preserve the Raw Data:
The first rule of an archive is don't transform the source before preserving it.
The extraction layer pulls Salesforce content through REST/Bulk APIs and stores the raw response before transforming anything This gives us an immutable copy of what Salesforce actually returned and allows the pipeline to reprocess data without calling Salesforce again
Bronze
With the raw data safely landed, the next step is to turn it into incremental Delta tables.
Rather than loading everything again on every run, use the Salesforce record ID as the key and its modification timestamp to determine what has changed.
This gives us an incremental Bronze archive without repeatedly processing unchanged Salesforce records
Silver — Decode, Clean, and Rebuild
With the data safely preserved in Bronze, the Silver layer turns the raw Salesforce payloads into clean, usable records.
For Notes, the pipeline decodes the Base64 body, removes HTML formatting, and converts it into readable text:
The pipeline keeps the latest Note version, preserves key ContentVersion metadata, and transforms ContentDocumentLink into a relationship layer that identifies the linked Salesforce object—such as Account, Contact, Opportunity, Case, or Lead.
The result is three clean Silver datasets:
- silver_notes — decoded and cleaned Note content
- silver_files — file and version metadata
- silver_note_links — file-to-Salesforce-record relationships
Gold — Turn Archived Notes into Insights
With the Notes cleaned and structured in Silver, the Gold layer adds business value using Databricks AI Functions.
The pipeline analyzes only Notes with usable content and generates sentiment, summaries, and risk/next-step insights
For deeper analysis, ai_query() extracts key risks and next steps from each Note.
Step 4 (Optional): Make Every Note Searchable
Create a Vector Search index so agents and Genie can search notes by meaning, not just keywords.
Step 5: Verify, Then Release Salesforce Storage
This step is where the savings come from. Copying the data alone frees nothing.
- Reconcile. Compare record counts and total bytes per object between Salesforce and Silver. Spot-check note text. In Databricks:
In Salesforce:
- Delete in batches with the Bulk API. Hard-delete, or empty the Recycle Bin, so storage is actually released
DANGER | HARD DELETE IS PERMANENT
A hard-deleted file cannot be restored from the Recycle Bin. The archive is your only copy, which is exactly why Step 3 exists. Confirm the exact permission your org needs for hard delete in a sandbox first.
- Confirm. Re-run the Salesforce storage report and record the reduction.
Governance
Notes and Files may contain sensitive information. Use Unity Catalog access controls, column masks, and row filters where required.
Deletion requests should propagate across Salesforce → Bronze → Silver → Gold → Vector/Search indexes.
What This Doesn't Solve
- API limits still apply. Use incremental extraction.
- Post-archive edits matter. Sync changes using LastModifiedDate.
- Deletion isn't always possible. Salesforce dependencies or asset references may block some files.
- AI output isn't the source of truth. Preserve the original content.
- AI costs apply. Process content incrementally and selectively.
- Retention rules still apply. Check legal holds and retention policies before deletion.
Results
We ran this pipeline on a client org and measured the outcome after the final reconciliation.
Architectural Takeaways
Archiving Salesforce content to Databricks Delta Lake resolves the constant pressure of storage limits without sacrificing business visibility:
Dual-Path Extraction avoids API Gaps: Using Lakeflow Connect for structured metadata alongside REST/Bulk APIs for Base64 binaries ensures full version history without payload drops.
Storage Cost Compression: Offloading gigabytes or terabytes from Salesforce (5/GB/mo)intocloudobjectstorage(0.00099/GB/mo on Glacier) yields immediate, recurring ROI.
Unlocking Dark CRM Data: Once extracted into Delta Lake, unstructured notes transition from forgotten storage overhead into active AI assets via AI Functions (`ai_summarize`, `ai_query`) and Vector Search.
Next Steps with Astrea IT
Managing enterprise data retention requires balancing regulatory compliance, security policies, and cost control.
At Astrea IT Services, we bridge Salesforce CRM architectures with Databricks Lakehouse pipelines to help organizations safely archive, govern, and analyze historical data.
Looking to evaluate your org's storage growth?
Reach out to our engineering team for a Salesforce Data Storage & Retention Assessment to identify archive candidates and design an automated cleanup strategy for your enterprise.
If you're interested in exploring more Databricks solutions, visit our Databricks Page.