Performance Optimization

Migration performance depends on a small number of high-impact factors. This article covers the most effective techniques for reducing execution time and avoiding common bottlenecks.

Filter Optimization

Filters are the highest-impact performance lever available to you. Every record you do not extract is a record you do not need to transform, map, or load. Fewer records means faster execution, less memory usage, and lower API consumption.

Be as specific as possible. If you only need incidents from the last 90 days, filter on the creation date. If you only need active users, filter on the status field. Avoid extracting entire tables when you only need a subset.

Filter on fields the source can resolve server-side. Filters that the source system can evaluate against its own index or query engine are far faster than full extractions filtered in memory. For database adaptors this means indexed columns; for SaaS systems (ServiceNow, Jira, etc.) this means fields supported by the system's query language. If you are unsure, check with your source system administrator or DBA, or use the Native filter mode (below) to push the query down to the source.

Combine filters. Multiple filter conditions narrow the result set further. A filter like [category] = "Hardware" AND [priority] = 1 AND [sys_created_on] > date(2025, 1, 1) is far more efficient than extracting all incidents and filtering in memory.

Native vs. Standard Filters

Precision Bridge supports three filter modes (in addition to None):

• Basic filters perform simple key-value equality matching. They exist mainly for internal use and CSV filtering — Standard is the better choice for most user-authored filters.
• Standard filters are translated by Precision Bridge into the source system's query language. This translation works well in most cases but may not always produce the most optimal query.
• Native filters are passed directly to the source system's API or query engine. The source system optimizes and executes the query natively, which is typically faster.

For performance-critical extractions, test both Standard and Native modes and compare execution times. Native filters are especially beneficial when using complex filter logic that the source system handles efficiently, such as encoded queries in ServiceNow.

Batch Sizing

Batch size controls how many records are processed in each API call or database query. The default batch size works well for most migrations, but tuning it can help in specific scenarios.

Out-of-the-box defaults:

• Database extraction chunk size: 1000 records per chunk
• CSV extraction chunk size: 1000 records per chunk
• CSV row load limit: 10,000 rows (used when slurping a whole file into memory)
• API endpoints: Each adaptor configures its own per-endpoint page size; check the adaptor's endpoint metadata.

These are global settings; you can also override retrieve, insert, and update batch sizes on individual Migrate Records steps via the override toggles on the Advanced tab.

Reduce batch size when:

• You are hitting API rate limits and receiving throttling errors.
• Individual records are large (many fields, large text content) and consuming too much memory per batch.
• The source system responds slowly to large requests.

Increase batch size when:

• You are performing database-to-database migrations where both systems can handle large result sets.
• Network latency is high and you want to minimize the number of round trips.
• The source system performs better with fewer, larger requests.

Monitor memory usage during large migrations. If Precision Bridge consumes excessive memory, reduce the batch size to lower the in-memory footprint.

Keyset Pagination

For large datasets (100,000+ records), use keyset pagination instead of offset pagination.

Why offset pagination is problematic at scale: Offset pagination requests page N by skipping the first (N-1) pages of results. If the source data changes during extraction (records are added or deleted), pages can shift, causing records to be skipped or duplicated. At high offsets, the source system may also perform poorly because it still has to scan all skipped records.

How keyset pagination works: Instead of requesting "page 5", keyset pagination requests "all records where the key field is greater than the last key value I received." This approach is stable even when source data changes during extraction and performs consistently regardless of how deep into the dataset you are.

Enable keyset pagination via the toggle on your Migrate Records or Extract Records step. PB uses the source table's pre-configured primary key as the cursor (for example, sys_id in ServiceNow or the primary key column in a database table) — there is no separate "key field" selector. The toggle is only available when the source table's adaptor declares keyset pagination support.

Lookup Performance

Lookup tables are used to translate values during field mapping (e.g., looking up a user's email from a user table). Large lookup tables can significantly slow down processing because each record in the migration triggers a lookup query.

Pre-load lookup data into a variable. Instead of querying the lookup table for every record, use an Extract Records step at the beginning of your procedure to load the lookup data into an output variable. Then use formula-based lookups against that in-memory variable. This converts thousands of individual queries into a single bulk extraction.

Filter lookup tables. If you only need a subset of the lookup data, filter the Extract Records step to load only relevant records. For example, if you are mapping user IDs but only need active users, filter on active = true.

Cache lookup results across steps. If multiple steps in the same procedure reference the same lookup data, extract it once and pass the variable to all steps that need it rather than re-extracting in each step.

API Rate Limits and Connection Management

Enterprise systems like ServiceNow and Jira enforce API rate limits. Precision Bridge handles rate limiting automatically by retrying throttled requests, but you can minimize throttling by:

• Using reasonable batch sizes that stay within the source system's comfort zone.
• Avoiding running multiple procedures against the same source system simultaneously.
• Scheduling large migrations during off-peak hours when the source system has more capacity.

Database connections are pooled by Precision Bridge for efficiency. You do not need to manage connection lifecycle manually.

General Tips

• Validate with a small sample before running at full scale. Use the Limit option on the Filter tab — or a tightly-scoped filter such as a single record number — to migrate a handful of records end-to-end. Once mappings look correct, remove the limit. (Note: the Sampling Validation option on the Validation tab is a separate, post-migration accuracy check — see Advanced Options.)
• Monitor execution logs. The execution report shows timing for each step. If one step is disproportionately slow, investigate its filters, batch size, and lookup configuration.
• Profile before optimizing. Do not guess at bottlenecks. Run the migration, review the execution report, and focus your optimization effort on the slowest steps.
• Consider data volume growth. A migration that runs in 10 minutes today may take hours next quarter if the source dataset grows. Build filters and pagination strategies that scale.

Summary

• Filters are the most effective optimization: extract only what you need.
• Use native filters for performance-critical extractions.
• Tune batch size based on your specific systems and data characteristics.
• Use keyset pagination for large datasets to ensure reliable, performant extraction.
• Pre-load lookup data into variables to avoid per-record query overhead.
• Monitor execution reports to identify and address actual bottlenecks.