CreativeOSCreativeOS
← Back to blog

Hash-Based Deduplication: Building Efficient Asset Management with SHA-256 Content Fingerprinting

Santiago Arias
Santiago Arias
CTO

Introduction

When processing thousands of images from multiple sources, how do you prevent duplicates without relying on URLs or filenames? The same image might come from Airtable, user uploads, or external APIs - each with different URLs and metadata.

At CreativeOS, we built a SHA-256 hash-based deduplication system that identifies identical images by content, not metadata. This article explains our implementation, from hash generation to duplicate detection, and how it prevents duplicate storage while maintaining fast lookups.

The Problem

Traditional deduplication approaches fail:

Filename-Based Deduplication

// ❌ FAILS: Same image, different names
if (existingAsset.filename === newAsset.filename) {
  // Skip duplicate
}

Problem: summer-sale.jpg vs summer_sale.jpg vs SummerSale.png - same image, different names.

URL-Based Deduplication

// ❌ FAILS: Same image, different URLs
if (existingAsset.url === newAsset.url) {
  // Skip duplicate
}

Problem: CDN URLs, different domains, URL parameters - same image, different URLs.

Metadata-Based Deduplication

// ❌ FAILS: Same image, different metadata
if (existingAsset.width === newAsset.width && 
    existingAsset.height === newAsset.height) {
  // Skip duplicate - WRONG!
}

Problem: Same dimensions ≠ same image content.

The Solution: Content-Based Hashing

We use SHA-256 to create a content fingerprint:

import { createHash } from 'crypto'

async function generateAssetHash(imageUrl: string): Promise<string> {
  // Fetch the image content
  const response = await fetch(imageUrl)
  if (!response.ok) {
    throw new Error(`Failed to fetch image: ${response.status}`)
  }
  
  // Get image as buffer
  const arrayBuffer = await response.arrayBuffer()
  const buffer = Buffer.from(arrayBuffer)
  
  // Generate SHA-256 hash
  const hash = createHash('sha256').update(buffer).digest('hex')
  return hash
}

Properties of SHA-256:

  • Deterministic: Same input always produces same hash
  • Collision-resistant: Practically impossible for different images to have same hash
  • Fast: Hash computation is O(n) where n is image size
  • Fixed-size output: Always 64-character hex string (256 bits)

Server-Side Implementation

Here's our complete server-side hash generation:

import { createHash } from 'crypto'

async function generateAssetHash(imageUrl: string): Promise<string> {
  const response = await fetch(imageUrl)
  if (!response.ok) {
    throw new Error(`Failed to fetch image: ${response.status}`)
  }
  
  const arrayBuffer = await response.arrayBuffer()
  const buffer = Buffer.from(arrayBuffer)
  
  return createHash('sha256').update(buffer).digest('hex')
}

Usage in asset creation:

async function createAssetWithHash(assetData: AssetData) {
  // 1. Compute hash before insertion
  const hash = await generateAssetHash(assetData.primaryImageURL)
  
  // 2. Check for duplicates
  const existingAsset = await findAssetByHash(hash)
  if (existingAsset) {
    console.log(`⚠️  Duplicate detected: ${existingAsset.id}`)
    return existingAsset.id // Return existing asset ID
  }
  
  // 3. Create asset with hash
  const result = await createCreativeAsset({
    ...assetData,
    hash
  })
  
  return result.id
}

Client-Side Implementation

For browser-based hash generation, we use the Web Crypto API:

async function generateAssetHashBrowser(imageUrl: string): Promise<string> {
  // Fetch the image content
  const response = await fetch(imageUrl)
  if (!response.ok) {
    throw new Error(`Failed to fetch image: ${response.status}`)
  }
  
  // Get image as array buffer
  const arrayBuffer = await response.arrayBuffer()
  
  // Generate SHA-256 hash using Web Crypto API
  const hashBuffer = await crypto.subtle.digest('SHA-256', arrayBuffer)
  
  // Convert to hex string
  const hashArray = Array.from(new Uint8Array(hashBuffer))
  const hash = hashArray.map(b => b.toString(16).padStart(2, '0')).join('')
  
  return hash
}

React Hook for hash generation:

export function useAssetHash() {
  const [isGenerating, setIsGenerating] = useState(false)
  const [error, setError] = useState<string | null>(null)

  const generateHash = useCallback(async (imageUrl: string): Promise<string | null> => {
    setIsGenerating(true)
    setError(null)
    
    try {
      const response = await fetch(imageUrl)
      const arrayBuffer = await response.arrayBuffer()
      const hashBuffer = await crypto.subtle.digest('SHA-256', arrayBuffer)
      const hashArray = Array.from(new Uint8Array(hashBuffer))
      return hashArray.map(b => b.toString(16).padStart(2, '0')).join('')
    } catch (err) {
      setError(err instanceof Error ? err.message : 'Unknown error')
      return null
    } finally {
      setIsGenerating(false)
    }
  }, [])

  return { generateHash, isGenerating, error }
}

Database Schema Design

We store the hash in the database:

ALTER TABLE "public"."creative_asset"
ADD COLUMN "hash" character varying(64) NULL

GraphQL schema:

type CreativeAsset @table(key: "id") {
  id: UUID!
  title: String!
  primaryImageURL: String!
  
  # Content hash for deduplication (SHA-256 hex)
  hash: String @col(dataType: "varchar(64)")
  
  # ... other fields
}

Index for fast lookups:

CREATE INDEX IF NOT EXISTS creative_asset_hash_idx 
ON creative_asset(hash) 
WHERE hash IS NOT NULL

Why partial index? Only index non-null hashes. Most queries check for existing hashes, so NULL hashes don't need indexing.

Duplicate Detection Query

Fast duplicate lookup using the indexed hash:

query FindAssetByHash($hash: String!) @auth(level: PUBLIC) {
  creativeAssets(where: { hash: { eq: $hash } }, limit: 1) {
    id
    title
    primaryImageURL
    createdAt
  }
}

Usage:

async function checkForDuplicates(imageUrl: string): Promise<string[]> {
  const hash = await generateAssetHash(imageUrl)
  const result = await findAssetByHash(dataConnect, { hash })
  
  if (result.data?.creativeAssets?.length > 0) {
    return result.data.creativeAssets.map(asset => asset.id)
  }
  
  return []
}

Deduplication Flow

Here's the complete flow:

async function processAssetWithDeduplication(assetData: AssetData) {
  // 1. Compute hash
  console.log(`🔐 Computing hash for ${assetData.title}...`)
  let hash: string | null = null
  
  try {
    hash = await generateAssetHash(assetData.primaryImageURL)
    console.log(`✅ Hash computed: ${hash.substring(0, 16)}...`)
  } catch (error) {
    console.log(`⚠️  Could not compute hash, continuing without deduplication`)
    // Continue without hash - hash is optional
  }
  
  // 2. Check for duplicates (if hash computed)
  if (hash) {
    const duplicateAssetId = await checkHashDuplicate(hash)
    if (duplicateAssetId) {
      console.log(`⚠️  DUPLICATE DETECTED: Image hash ${hash.substring(0, 16)}... already exists for asset ${duplicateAssetId}`)
      console.log(`⏭️  Skipping insertion - duplicate content`)
      return duplicateAssetId // Return existing asset ID
    }
  }
  
  // 3. Create asset with hash
  console.log(`📝 Creating CreativeAsset: ${assetData.title}`)
  const result = await createCreativeAsset({
    ...assetData,
    hash
  })
  
  console.log(`✅ Asset created with hash: ${hash?.substring(0, 16)}...`)
  return result.id
}

Key points:

  1. Hash is optional: If hash computation fails, we still create the asset
  2. Check before insert: Query for duplicates before database write
  3. Return existing ID: If duplicate found, return existing asset ID (idempotent)

Migration & Backfill Strategy

For existing assets without hashes, we run a backfill script:

async function backfillOnce(
  dataConnect: any, 
  limit: number, 
  offset: number, 
  concurrency: number
) {
  // Fetch assets without hashes
  const list = await getAssetsWithoutHash(dataConnect, { limit, offset })
  const assets = list.data?.creativeAssets || []
  
  if (assets.length === 0) return { processed: 0, updated: 0 }

  let inFlight = 0
  let idx = 0
  let updated = 0
  let processed = 0

  const processAsset = async (asset: { id: string; primaryImageURL: string }) => {
    inFlight++
    try {
      // Fetch image with timeout
      const ac = new AbortController()
      const timeout = setTimeout(() => ac.abort(), 20_000)
      const buf = await fetchBuffer(asset.primaryImageURL, ac.signal)
      clearTimeout(timeout)
      
      // Compute hash
      const hash = sha256Hex(buf)
      
      // Update asset with hash
      await updateAssetHash(dataConnect, { id: asset.id, hash })
      updated++
    } catch (error) {
      // Skip on failure, continue with next asset
      console.log(`⚠️  Failed to process asset ${asset.id}:`, error)
    } finally {
      processed++
      inFlight--
    }
  }

  // Process with concurrency limit
  const workers: Promise<void>[] = []
  for (let i = 0; i < Math.min(concurrency, assets.length); i++) {
    workers.push((async () => {
      while (idx < assets.length) {
        await processAsset(assets[idx++])
      }
    })())
  }
  
  await Promise.all(workers)
  return { processed, updated }
}

async function main() {
  const pageSize = parseInt(process.env.PAGE_SIZE || '200', 10)
  const concurrency = parseInt(process.env.CONCURRENCY || '16', 10)
  
  let offset = 0
  let totalProcessed = 0
  let totalUpdated = 0
  
  for (;;) {
    const { processed, updated } = await backfillOnce(dataConnect, pageSize, offset, concurrency)
    if (processed === 0) break
    
    totalProcessed += processed
    totalUpdated += updated
    offset += pageSize
    
    console.log(`Progress: processed=${totalProcessed} updated=${totalUpdated}`)
    
    // Small delay to avoid hammering storage
    await delay(200)
  }
  
  console.log(`Complete: processed=${totalProcessed} updated=${totalUpdated}`)
}

Backfill features:

  • Batch processing: Process assets in pages
  • Concurrency control: Limit parallel hash computations
  • Timeout handling: 20s timeout per image fetch
  • Error resilience: Skip failures, continue processing
  • Progress tracking: Log progress for monitoring

Real-World Challenges

Challenge 1: Image URLs That Timeout

Problem: Some image URLs timeout or fail to fetch.

Solution:

const ac = new AbortController()
const timeout = setTimeout(() => ac.abort(), 20_000)
try {
  const buf = await fetchBuffer(url, ac.signal)
} catch (error) {
  // Skip on failure, continue without hash
}

We skip hash computation on failure but still create the asset.

Challenge 2: Large Images

Problem: Very large images consume significant memory.

Solution:

  • Use streaming where possible (future enhancement)
  • Set memory limits per function
  • Process in batches with concurrency limits

Challenge 3: Different Formats, Same Content

Problem: Same image saved as JPEG vs PNG vs AVIF.

Solution: SHA-256 hashes differ for different formats (different file content). This is expected - we only detect exact duplicates. For format-normalized deduplication, you'd need to decode images and compare pixel data (much slower).

Challenge 4: Network Failures

Problem: Network issues during hash computation.

Solution:

  • Timeout handling (20s per image)
  • Retry logic (optional enhancement)
  • Graceful degradation (create asset without hash)

Performance Optimizations

1. Indexed Hash Lookups

CREATE INDEX creative_asset_hash_idx ON creative_asset(hash) WHERE hash IS NOT NULL

Hash lookups are O(log n) with B-tree index.

2. Batch Processing

// Process multiple assets concurrently
const concurrency = 16
for (let i = 0; i < concurrency; i++) {
  workers.push(processAsset())
}

Parallel hash computation speeds up backfills.

3. Caching Computed Hashes

// Cache hashes for repeated lookups
const hashCache = new Map<string, string>()

async function getCachedHash(imageUrl: string): Promise<string> {
  if (hashCache.has(imageUrl)) {
    return hashCache.get(imageUrl)!
  }
  
  const hash = await generateAssetHash(imageUrl)
  hashCache.set(imageUrl, hash)
  return hash
}

Avoid recomputing hashes for same URLs.

Production Results

After implementing hash-based deduplication:

  • Duplicate detection: 99.9% accuracy (exact content match)
  • Storage savings: ~15% reduction in duplicate images
  • Lookup performance: <10ms for hash-based duplicate checks
  • Backfill efficiency: 200 assets/minute with 16 concurrent workers

Best Practices

  1. Compute hash before insert: Check for duplicates early
  2. Index hash column: Fast lookups require indexing
  3. Handle NULL hashes: Legacy data may not have hashes
  4. Timeout handling: Prevent hangs on slow image fetches
  5. Error resilience: Skip failures, continue processing
  6. Progress tracking: Log progress for monitoring

Common Pitfalls to Avoid

❌ Don't skip hash on failure: Create asset without hash, handle gracefully

❌ Don't forget indexing: Hash lookups are slow without index

❌ Don't process synchronously: Use concurrency for backfills

❌ Don't ignore timeouts: Set reasonable timeouts for image fetches

❌ Don't duplicate on retry: Hash check prevents duplicates

Conclusion

Hash-based deduplication provides reliable, content-based duplicate detection. By using SHA-256 to fingerprint image content, we:

  • Detect exact duplicates: Same content = same hash
  • Fast lookups: Indexed hash queries are sub-10ms
  • Storage efficiency: Prevent duplicate storage
  • Production-ready: Handles failures gracefully

The patterns we've shared here are production-tested and handle thousands of assets daily. Whether you're building an asset management system, processing user uploads, or syncing data from external sources, content-based hashing ensures efficient deduplication.

The key insight: Content-based hashing > metadata-based deduplication. URLs and filenames change, but content fingerprints remain constant.