Skip to main content

Import, export, and migrate data

Move data in and out of a table. Use TaruviBase Console for one-off files, or write a migration script that validates every record and sends it in batches through an SDK or the REST API.

Choose a data-movement workflow​

For a one-off import or export, use TaruviBase Console: open your app, select Datatables, open the table's menu, and select Import data or Export data. Imports run as a job; check progress and download any error report under History. Use the SDK or REST workflow below for large, repeatable, or automated migrations.

TaskRecommended workflow
Import recordsNormalize the source, then use bounded create or update batches
Export recordsRead stable, bounded pages from an unchanged source set and write each completed page to the destination
Resume workPersist an application-owned checkpoint after each verified batch
Report rejected rowsKeep a redacted application report with the source key and validation result
Retry an unknown outcomeRead the stable destination IDs before resending a write

Using the SDK or REST API means every batch goes through the same access checks as the rest of your application. Save your progress after each batch and keep a report of rejected rows so you can resume and check the transfer.

1. Prepare the destination​

  1. Save the destination table's exact stored schema JSON in the migration manifest. If a checksum is useful, compute one over that saved file and record both the checksum and algorithm.
  2. Confirm the site, app, and table selected for the migration.
  3. Create a disposable app with the same schema for a rehearsal.
  4. Back up the source and record its row count.

For retryable imports, supply an application-owned UUID and keep a stable mapping between the source identifier and TaruviBase identifier. This prevents an ambiguous retry from creating a second logical record.

2. Normalize and validate records​

Convert each source row to the destination field names and value types before sending it to TaruviBase. Reject or quarantine a row when it:

  • omits a required field;
  • violates a length, range, enum, unique, or foreign-key constraint;
  • contains an unknown field; or
  • cannot be mapped to one explicit destination identifier.

Test one synthetic record first, read it back, and compare the stored values.

3. Send a bounded batch​

Use bulk create or bulk update for records whose outcome can be tracked as one batch. Keep every batch at 1,000 records or fewer and keep the ordered list of record IDs.

await database.from('tasks').create(batch).execute();

Here, batch is the validated list for one checkpoint. Compare the returned record count with the batch size before advancing the checkpoint.

For an export, pause source writes or capture an immutable record-ID manifest, then page through the unchanged source set with stable ordering. Write each completed page to the destination file or system. Do not rely on a single unbounded list response.

await database
  .from('tasks')
  .sort('id', 'asc')
  .page(page)
  .pageSize(100)
  .execute();

4. Checkpoint progress​

After every successful batch, persist:

  • the source range, ID manifest, or page within a quiesced source;
  • the destination record IDs;
  • the saved schema file and its application-owned checksum, when used;
  • the attempted, accepted, and rejected counts; and
  • the completion timestamp.

Resume from the last verified checkpoint. When a request outcome is unknown, read the explicit IDs before retrying the write.

5. Check and finish​

  1. Compare the source, accepted, rejected, and destination counts.
  2. Read a sample from every batch, including boundary values.
  3. Verify foreign-key references after all dependent records are present.
  4. Retain a redacted error report for rejected rows.
Protect destination data
  • Affected resource and cascade: Cleanup targets only the reviewed rehearsal or destination record IDs in the named site, app, and table. Review foreign-key behavior before cleanup.
  • Reversibility: Record deletion is permanent.
  • Authorization: The cleanup caller needs the table policy's delete action for every target record.
  • Backup or export: Retain the verified source backup, migration manifest, and exported cleanup records.
  • Confirmation: Compare the exact cleanup ID list and count with the reconciled rehearsal or destination records.
  • Success response and postcondition: Use the canonical single-record or bulk-delete response contract, compare the deleted count when present, and verify every target ID is absent.
  • Recovery: This guide documents no undelete operation. Recreate cleanup records only from the retained export and re-run reconciliation.
  1. Remove rehearsal data only after reconciliation succeeds.

Cross-batch atomicity is not assumed. Design the migration so every completed batch is independently verifiable and every rejected row can be corrected and replayed without repeating successful writes.

Continue with query pagination, reliability, or the interface map.