Bulk Operations Business

Bulk operations enable you to update, delete, or modify thousands of documents in a single operation, with built-in safety features and progress tracking.

Overview

Use bulk operations to:

• Update many documents - Modify fields across multiple documents
• Delete in bulk - Remove documents matching criteria
• Insert multiple documents - Create many documents at once
• Transform data - Apply calculations or transformations at scale
• Migrate schemas - Update document structures across collections

Document Limits

Bulk Update

Simple Field Updates

Update specific fields across multiple documents:

{
  "$set": {
    "status": "archived",
    "archivedAt": new Date()
  }
}

Update Operators

Use MongoDB update operators:

// Set field values
{
  "$set": {
    "status": "active",
    "updatedAt": new Date()
  }
}
 
// Increment numeric fields
{
  "$inc": {
    "viewCount": 1,
    "score": 10
  }
}
 
// Multiply values
{
  "$mul": {
    "price": 1.1  // 10% price increase
  }
}
 
// Rename fields
{
  "$rename": {
    "oldFieldName": "newFieldName"
  }
}
 
// Remove fields
{
  "$unset": {
    "deprecatedField": "",
    "temporaryData": ""
  }
}
 
// Add to array
{
  "$push": {
    "tags": "featured"
  }
}
 
// Remove from array
{
  "$pull": {
    "tags": "deprecated"
  }
}
 
// Add to set (no duplicates)
{
  "$addToSet": {
    "categories": "electronics"
  }
}

Conditional Updates

Update based on current values:

// Update only if field doesn't exist
{
  "$setOnInsert": {
    "createdAt": new Date()
  }
}
 
// Set minimum value
{
  "$min": {
    "lowestPrice": 99.99
  }
}
 
// Set maximum value
{
  "$max": {
    "highestBid": 150.00
  }
}
 
// Current date
{
  "$currentDate": {
    "lastModified": true,
    "timestamp": { "$type": "date" }
  }
}

Complex Transformations

Use aggregation pipeline updates:

// Calculate new field from existing fields
[
  {
    "$set": {
      "fullName": {
        "$concat": ["$firstName", " ", "$lastName"]
      },
      "discountedPrice": {
        "$multiply": [
          "$price",
          { "$subtract": [1, { "$divide": ["$discountPercent", 100] }] }
        ]
      }
    }
  }
]
 
// Conditional field updates
[
  {
    "$set": {
      "tier": {
        "$switch": {
          "branches": [
            { "case": { "$gte": ["$points", 1000] }, "then": "gold" },
            { "case": { "$gte": ["$points", 500] }, "then": "silver" }
          ],
          "default": "bronze"
        }
      }
    }
  }
]

Bulk Delete

Deleting Documents

Remove multiple documents matching criteria:

Safe Deletion Practices

Protect against accidental data loss:

1. Filter carefully - Double-check your filter criteria
2. Preview first - Review sample documents before deletion
3. Export backup - Export data before bulk delete
4. Use dry run - Test with dryRun: true option
5. Soft delete - Mark as deleted instead of removing

// Soft delete example
{
  "$set": {
    "isDeleted": true,
    "deletedAt": new Date(),
    "deletedBy": "{{current_user}}"
  }
}

Bulk Insert

Inserting Multiple Documents

Create many documents at once:

[
  {
    "name": "Product 1",
    "price": 99.99,
    "category": "electronics"
  },
  {
    "name": "Product 2",
    "price": 149.99,
    "category": "electronics"
  }
]

Import from File

Bulk insert from CSV or JSON:

Bulk Replace

Replacing Documents

Replace entire documents:

// Find documents
{
  "filter": {
    "version": 1
  },
 
  // Replace with new structure
  "replacement": {
    "version": 2,
    "data": {
      // New document structure
    }
  }
}

Progress Tracking

Real-Time Progress

Monitor bulk operation execution:

Displayed metrics:

• Documents processed / total
• Success rate
• Error count
• Elapsed time
• Estimated time remaining
• Operations per second

Operation History

View past bulk operations:

{
  "operationId": "bulk_op_abc123",
  "type": "update",
  "collection": "users",
  "startTime": "2024-02-24T10:00:00Z",
  "endTime": "2024-02-24T10:05:23Z",
  "status": "completed",
  "statistics": {
    "matched": 10000,
    "modified": 9987,
    "failed": 13,
    "duration": "5m 23s"
  },
  "filter": {
    "status": "pending"
  },
  "update": {
    "$set": { "status": "processed" }
  }
}

Error Handling

Handling Failures

Configure how to handle errors:

Ordered Operations:

• Stop on first error
• Guarantees order of operations
• Slower but safer

Unordered Operations:

• Continue despite errors
• Faster execution
• Collects all errors

Error Reporting

View detailed error information:

{
  "errors": [
    {
      "documentId": "507f1f77bcf86cd799439011",
      "error": "Validation failed: email is required",
      "operation": {
        "$set": { "status": "active" }
      }
    },
    {
      "documentId": "507f1f77bcf86cd799439012",
      "error": "Duplicate key error",
      "operation": {
        "$set": { "username": "existinguser" }
      }
    }
  ],
  "successCount": 9987,
  "errorCount": 13
}

Retry Failed Operations

Retry only failed documents:

Performance Optimization

Batch Size Configuration

Optimize throughput:

{
  "batchSize": 1000,  // Documents per batch
  "maxConcurrency": 4  // Parallel batches
}

Guidelines:

• Small documents: 500-1000 per batch
• Large documents: 100-500 per batch
• Complex updates: 50-200 per batch

Indexing Considerations

Improve bulk operation speed:

1. Disable indexes temporarily - For massive inserts
2. Use covered queries - Ensure filters use indexes
3. Avoid large arrays - Array operations are slower
4. Use projection - Only fetch needed fields

Parallel Processing

Execute operations in parallel:

{
  "parallel": true,
  "maxWorkers": 8,
  "batchSize": 500
}

Safety Features

Dry Run Mode

Test operations without making changes:

Backup Before Operation

Automatic backup option:

{
  "operation": "update",
  "backup": {
    "enabled": true,
    "location": "s3://backups/bulk-ops/2024-02-24/",
    "format": "json"
  }
}

Operation Audit Trail

All bulk operations are logged:

{
  "timestamp": "2024-02-24T10:00:00Z",
  "user": "admin@company.com",
  "operation": "bulk_update",
  "collection": "users",
  "filter": { "status": "pending" },
  "update": { "$set": { "status": "processed" } },
  "documentsAffected": 9987,
  "duration": "5m 23s"
}

Common Bulk Operations

Data Migration

Migrate to new schema:

// Add new field based on existing data
[
  {
    "$set": {
      "fullAddress": {
        "$concat": [
          "$address.street", ", ",
          "$address.city", ", ",
          "$address.state", " ",
          "$address.zip"
        ]
      }
    }
  },
  {
    "$unset": "address"
  }
]

Price Updates

Bulk price changes:

// 10% price increase on specific category
{
  "filter": {
    "category": "electronics",
    "price": { "$exists": true }
  },
  "update": {
    "$mul": { "price": 1.10 },
    "$set": { "lastPriceUpdate": new Date() }
  }
}

Status Updates

Batch status changes:

// Archive old records
{
  "filter": {
    "status": "pending",
    "createdAt": { "$lt": new Date("2023-01-01") }
  },
  "update": {
    "$set": {
      "status": "archived",
      "archivedAt": new Date(),
      "archivedReason": "auto-archive-old-records"
    }
  }
}

Cleanup Operations

Remove obsolete data:

// Remove temporary fields
{
  "filter": {},
  "update": {
    "$unset": {
      "tempCache": "",
      "processingData": "",
      "__v": ""
    }
  }
}

API Access

Perform bulk operations via API:

# Bulk update
curl -X POST https://api.mongodash.com/v1/collections/users/bulk-update \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {"status": "pending"},
    "update": {"$set": {"status": "processed"}},
    "options": {
      "ordered": false,
      "batchSize": 1000
    }
  }'
 
# Bulk delete
curl -X POST https://api.mongodash.com/v1/collections/users/bulk-delete \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {"isDeleted": true, "deletedAt": {"$lt": "2023-01-01"}},
    "options": {
      "dryRun": false
    }
  }'

Best Practices

Planning

1. Test on subset - Validate with small sample first
2. Schedule off-peak - Run during low-traffic periods
3. Export backup - Always backup data first
4. Document changes - Record what was changed and why
5. Monitor resources - Watch database CPU and memory

Execution

1. Use dry run - Test before executing
2. Start small - Begin with small batches
3. Monitor progress - Watch for errors and slowdowns
4. Have rollback plan - Know how to undo changes
5. Communicate - Inform team of bulk operations

After Execution

1. Verify results - Sample check updated documents
2. Check metrics - Ensure expected number of changes
3. Review errors - Investigate and fix failures
4. Update documentation - Record schema changes
5. Clean up - Remove temporary fields or backups

What's Next?

• Custom Views - Save bulk operation filters as views
• Data Sync - Automate data synchronization
• Import/Export - Learn more about data import/export