Caching and downsampling images in Swift

Say we have a list of items, each one with a bit of text and an image, and a details screen that shows the same item with more fields. The details screen refetches the object, because the list response doesn't carry everything, and it draws the same image, only smaller. Same item, same URL, two sizes (I know the server or the CDN could serve a different size, but for the purpose of this post, let's say they don't; and it'll prove to actually be better).

The usual route

We want to be efficient with the calls, and we want the images cached. First, the fetch itself:

let (data, _) = try await URLSession.shared.data(from: url)

What comes back is Data, not an image, so the usual route is to turn it into a UIImage and resize it for the view it goes in, since a 4000×4000 image is 64MB of bitmap, and you don't want that in a 200pt cell.

The first thing we reach for is a cache, so we don't download twice:

actor ImageCache {
 
   static let shared = ImageCache()
 
   private let cache = NSCache<NSURL, UIImage>()
 
   func image(for url: URL, size: CGSize) async throws -> UIImage? {
      if let cached = cache.object(forKey: url as NSURL) { // ❶
         return cached
      }
 
      let (data, _) = try await URLSession.shared.data(from: url)
 
      guard let original = UIImage(data: data) else { return nil } // ❷
 
      let resized = UIGraphicsImageRenderer(size: size).image { _ in // ❸
         original.draw(in: CGRect(origin: .zero, size: size))
      }
 
      cache.setObject(resized, forKey: url as NSURL) // ❹
 
      return resized
   }
 
}

We check the cache first (❶): on a miss, we download, make a UIImage out of the data (❷), redraw it at the size of the view (❸) and store the resized one under the URL (❹).

Next, the calls.

On a slow connection, the list can still be downloading an image when someone taps the item, and the details screen asks for the same URL. It happens the other way around too: a deeplink opens the details screen, and we go back to the list while that download is still running.

As it is, that's a second download of the same data, competing with the first one for the same slow connection, so we keep track of the downloads already in progress, and let a second caller wait on the first one:

actor ImageCache {
 
   // [...]
   private var inFlight: [URL: Task<Data, Error>] = [:]
 
   private func data(for url: URL) async throws -> Data {
      if let existing = inFlight[url] { // ❶
         return try await existing.value
      }
 
      let task = Task {
         let (data, _) = try await URLSession.shared.data(from: url)
         return data
      }
 
      inFlight[url] = task // ❷
 
      defer { inFlight[url] = nil } // ❸
 
      return try await task.value
   }
 
}

If a download for the URL is already running, we wait on its Task instead of starting another one (❶); if not, we start one and put it in the table (❷). The defer (❸) takes it out whether the download succeeded or threw, so one failure doesn't poison the URL, the next request just tries again.

This table is the reason ImageCache is an actor: there's no await between the lookup and the insert, so two callers can't both miss it and both start a download. image(for:size:) now calls data(for:) instead of URLSession directly.

Both screens now go through the same cache with the same URL, so they hit the same entry, and that entry has exactly one size: the size of whichever screen got there first.

If that's the list, the details screen gets a 200pt image for an 80pt slot. It works, but it either draws a much larger image than it needs, or it resizes an image that was already resized once.

But if we have deeplinks, though, the details screen can be the first one that runs. Nothing is cached yet, so it downloads the image, resizes it to 80pt, stores it under the URL, and then the list asks for that same URL and gets an 80pt image to draw at 200, which will be blurry.

We could put the image size in the cache's key, but then the details screen would miss the list's entry and download the same image again.

So, what does this cache actually need to do?

Download a URL once, no matter which screen asks first.
Not download it twice while the first download is still running.
Give each screen an image at its own size, without either one deciding for the other.
Not burn CPU turning one screen's image into the other's or juggling Data and UIImage around.
Never hold a full-size bitmap in memory just to draw a thumbnail.
Survive a relaunch.

Where's the issue, then?

A UIImage is already rendered, it has a size, and we're filing it under something that never mentions the size. The Data, on the other hand, really is the same for both screens, so that's what should go under the URL, and each screen downsamples from it at its own size.

Caching the `Data`

That gives us two caches. The Data goes in a file, keyed by the URL, so it survives a relaunch; we read it back whenever a screen needs a size we don't have yet in the memory cache.

The decoded UIImages go in an NSCache, keyed by the URL plus the pixel size, so a cell that scrolls back into view doesn't downsample again, but at the same time, both the list and details screens have their own correct size. Data only turns into a UIImage in one place, the downsample, which only runs when that NSCache has nothing for the URL at that size.

Nothing in here ever expires, though, so let's assume for the purpose of this post that the image behind a URL never changes. That holds for most CDN URLs, which change when the image does, but not for something like an avatar that keeps its URL when the user picks a new one.

Why not URLCache, which URLSession already has? By default it follows the server's cache headers, so whether an image gets cached isn't really our call, and it doesn't merge requests that are already in flight either.

In code, that's a couple of new properties:

actor ImageCache {
 
   static let shared = ImageCache()
 
   private let images: NSCache<NSString, UIImage> = {
      let images = NSCache<NSString, UIImage>() // ❶
 
      images.totalCostLimit = 50 * 1024 * 1024 // ❷
 
      return images
   }()
   private var inFlight: [URL: Task<Data, Error>] = [:]
 
   private let directory: URL = {
      let directory = URL.cachesDirectory.appending(path: "Images") // ❸
 
      try? FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true)
 
      return directory
   }()
 
}

images (❶) is that NSCache, and it only ever holds downsampled images. Its totalCostLimit (❷) is the memory budget for them, 50MB here: every image goes in with a cost, and when the total goes over the limit, NSCache evicts images to get back under it (the docs say it's not a strict limit). It also evicts on its own when the system is low on memory, limit or not.

We chose for the files to go in the caches directory (❸), but it might very well be any folder, but then you'd have to add logic to purge old/unused images; the caches folder is somewhat handled by the system (which might also be a downside, since we don't get to decide when they get purged, and it only happens when the device is low on space), so a real app still wants its own size limit on that folder.

In ImageCache.data(for:), the only change is inside the Task:

let file = fileURL(for: url)
let task = Task {
   if let onDisk = try? Data(contentsOf: file) { // ❶
      return onDisk
   }
 
   let (data, response) = try await URLSession.shared.data(from: url)
 
   guard let http = response as? HTTPURLResponse, (200..<300).contains(http.statusCode) else { // ❷
      throw URLError(.badServerResponse)
   }
 
   try? data.write(to: file, options: .atomic) // ❸
 
   return data
}

The disk read (❶) sits inside the same Task, so it gets deduplicated as well, and it stays off the main thread. It's a synchronous read, so it does hold the actor while it runs, which for thumbnails is fine.

(❷) matters a lot more now that the data lives on disk. URLSession doesn't throw on a 404 or a 500, it hands back the error page as Data, and without this check we'd write that page to the file: every later request would find it there, and the image would never load again. Throwing goes through the same defer as any other failure, so the next request just tries again.

(❸) is a try? on purpose, a full disk shouldn't fail the image, but in a real project that might be a log line rather than a discarded error.

The filename has to be stable across launches, which rules out hashValue: Swift seeds it per process, so yesterday's files would be unreachable today. The URL's path isn't enough either: 2 hosts can serve the same path, a query like ?v=2 can change the image while the path stays the same, and a long path can go past the 255-byte limit for a filename. A hex digest from CryptoKit can do the job, and it's always the same length:

extension ImageCache {
 
   private nonisolated func fileURL(for url: URL) -> URL {
      let digest = SHA256.hash(data: Data(url.absoluteString.utf8))
      let name = digest.map { String(format: "%02x", $0) }.joined()
 
      return directory.appending(path: name)
   }
 
}

Downsampling

Now, the bread and butter. image(for:size:) gets a scale and stops resizing:

extension ImageCache {
 
   func image(for url: URL, size: CGSize, scale: CGFloat) async throws -> UIImage? {
      let pixels = Int(max(size.width, size.height) * scale) // ❶
      let key = "\(url.absoluteString)|\(pixels)" as NSString
 
      if let cached = images.object(forKey: key) {
         return cached
      }
 
      let original = try await data(for: url)
 
      guard let image = await downsample(original, maxPixelSize: pixels, scale: scale) else {
         return nil
      }
 
      let cost = image.cgImage.map { $0.bytesPerRow * $0.height } ?? 0 // ❷
 
      images.setObject(image, forKey: key, cost: cost)
 
      return image
   }
 
}

We take a size in points and a scale, and turn both into one number, the longest edge in pixels (❶). That number goes in the key, so the list's 600px entry and the details screen's 240px entry (200pt and 80pt, on a 3x screen) are separate entries over the same file, and neither one can overwrite the other.

Why not w×h? Because the longest edge is the only size ImageIO takes: it keeps the source's aspect ratio and only limits the longest edge, so 2 slots with the same longest edge get exactly the same image, and a w×h key would just store it twice. This does assume the image fits inside its slot, like our square thumbnails; filling a slot whose aspect ratio differs from the source's needs a longer edge than this, worked out from the source's ratio.

The cost we hand to NSCache (❷) is the decoded bitmap's size in bytes, bytesPerRow * height, which is what the 50MB limit is measured in.

Lastly, the downsampling itself, which is more or less the recipe from Image and Graphics Best Practices:

extension ImageCache {
 
   @concurrent
   private nonisolated func downsample(_ data: Data, maxPixelSize: Int, scale: CGFloat) async -> UIImage? {
      let sourceOptions = [kCGImageSourceShouldCache: false] as CFDictionary // ❶
 
      guard let source = CGImageSourceCreateWithData(data as CFData, sourceOptions) else {
         return nil
      }
 
      let thumbnailOptions = [
         kCGImageSourceCreateThumbnailFromImageAlways: true, // ❷
         kCGImageSourceCreateThumbnailWithTransform: true, // ❸
         kCGImageSourceShouldCacheImmediately: true, // ❹
         kCGImageSourceThumbnailMaxPixelSize: maxPixelSize // ❺
      ] as CFDictionary
 
      guard let thumbnail = CGImageSourceCreateThumbnailAtIndex(source, 0, thumbnailOptions) else {
         return nil
      }
 
      return UIImage(cgImage: thumbnail, scale: scale, orientation: .up)
   }
 
}

CGImageSourceCreateWithData doesn't decode anything by itself, ❶ keeps it that way, so a 4000×3000 JPEG never becomes 48MB of bitmap on its way to a 240px thumbnail. ❷ tells it to build from the full image instead of the embedded EXIF thumbnail, which some cameras include. ❸ applies the EXIF orientation while it thumbnails, which is what makes the .up at the end valid.

❹ makes the decode happen right here, instead of lazily at the first draw. ❺ is in pixels, not points, which is the whole reason we computed pixels a moment ago.

The @concurrent on top (Swift 6.2) makes downsample always run on the concurrent thread pool, off the actor. Without it, a plain synchronous call from image(for:size:scale:) would run on the actor itself, so every decode would wait for the one before it, and so would a cell whose image is already in images.

And all of this also saves CPU, the list most of all. There's no resizing an image that was already resized, because we never threw the data away. The usual route, UIImage(data:) plus a redraw, decodes the full-size image first and then draws all of it again at the new size, while downsampling from the data never builds the full-size bitmap at all. On one details screen it hardly matters, but on a list you're scrolling through, it's one full-size decode per image that doesn't happen anymore.

The call site doesn't care about any of this:

let image = try await ImageCache.shared.image(for: item.imageURL, size: thumbnailSize, scale: displayScale)

The displayScale comes from the view (traitCollection.displayScale, or @Environment(\.displayScale) in SwiftUI), because the view knows its screen and the cache doesn't. It isn't always 3, either: an iPad with an external display, or an iPad app running on a Mac, can have another one.

Both screens call the same line with their own thumbnailSize. With the list first, it goes like this:

The list asks for url at 200pt (600px). images has nothing and there's no file, so we download the Data, write it to the file, downsample it to 600px and put that UIImage in images.
You open the item, and the details screen asks for the same url at 80pt (240px). images has no 240px entry, but the file is there, so we read the Data from disk, downsample it to 240px and put that in images as well. No download.
A cell that scrolls back into view finds its image in images, and nothing else happens.
After a relaunch, images is empty but the file isn't, so each screen reads it and downsamples again, still with no download.

With a deeplink it's the same steps with the screens swapped, and nothing ends up blurry. On a slow connection, the second screen might ask while the first download is still running, in which case it waits on that one, then does its own downsample.

This is also where the server serving a single size turns out to be better: once the first screen has the file, the second one never touches the network for the image. The cost is that the first screen downloads more pixels than it draws, and a URL per size would mean fewer bytes overall, but also another request, and a wait, on each screen.

One thing this doesn't do is cancel. A cell that scrolls off screen keeps its download running to the end and lands it in the cache, which is free if you scroll back up and pure waste if you jumped 200 rows.

To cancel properly, the shared Task would need to know how many cells and screens are still waiting on it, and cancel only when the last of them closes or scrolls away. Otherwise a cell that scrolls away cancels a download the details screen still wants.

As always, let me know if there's anything that can be improved @roland.leth.ro or @rolandleth.

The usual route

Caching the Data

Downsampling

Caching the `Data`