UI doesn’t have to look like a rectangle. You might want a different shape for your icon. You can do that with layer masks. Let’s learn how to set them up.
Download the example files. You’ll find an app with two square buttons.

If you head over to the assets, you see I have a thumbs-up and a thumbs-down image and its inverse image.

I’ll take those images and make them mask images cutting the solid buttons.
Head to the ViewController.swift code. I’ve already added images in the assets folder and assigned them to constants in the view controller.
I also created a function for you configureMask, for adding masks to the buttons.

 func configureMask(button:UIButton, with image:UIImage!){ 

A mask is an area, in our case a CALayer, where its opacity determines visibility to an underlying layer. Solid areas appear, areas with an alpha of zero disappear.
Mask layers are CALayers embedded in another CALayers. Instantiate a new CALayer

let maskLayer = CALayer()

Next, set its frame. This frame will be relative to its view, which will be the button. I’ll start with the origin at (0,0) to start in the upper left corner of the button.

maskLayer.frame.origin = CGPoint(x: 0, y: 0)  

I’ll make the frame the same size as the button. I’ll come back to this in a minute.

maskLayer.frame.size = button.frame.size

Once you have a frame, set the contents of the maskLayer to an image. This is a CGImage so I’ll use the image’s cgImage property to get one.

  maskLayer.contents = image.cgImage

Finally, in the button, add  to the layer’s mask property maskLayer.

   button.layer.mask = maskLayer

I’ll use this method to configure both buttons. For the auto layout I defined, the size of the button is not known until the subviews are laid out. I need that for the mask to be correct. So I’ll add it in the method viewDidLayoutSubviews.

override func viewDidLayoutSubviews() {
        configureMask(button: upButton, with: upImage)
        configureMask(button: downButton, with: downImage)
    }

I’ll change the mask when we tap a button by changing the button’s mask contents. I’ll do that in the upButton IBAction.

@IBAction func upButton(_ sender: UIButton) {
    upButton.layer.mask?.contents = inverseUpImage?.cgImage
    downButton.layer.mask?.contents = downImage?.cgImage
}

The configureMask method replaces the maskLayer. I could use it to change the button’s mask as well.

@IBAction func downButton(_ sender: UIButton) {
    configureMask(button: upButton, with: upImage)
    configureMask(button: downButton, with: inverseDownImage)
}

We are ready to run. You can use any simulator, but I’ll use an iPad 11 inch for this one. Run, and you get two buttons with the first mask.

Click a button. The mask changes.

Click the other button. The mask changes again.

Stop the app. I added two more backgrounds for you. Go to the storyboard and change the button backgrounds to Tile Floor and Wood table. Run again, tap a button,  and you get those photos masked.

That’s the very basics of CALayer masks. With a few lines of code, you can make masks to make any shape, from icons like these to circles and squares. Since masks are a CALayer, You can even make more complex masks by masking the mask’s layer. Play around with masks a bit, You’ll find them a very useful tool to make any view something other than a rectangle.

The Whole Code

Here’s the code for this lesson. If you want to download the full prjoect you can do so on GitHub.

//
//  ViewController.swift
//  CALayerMask Demo
//
//  An exercise file for iOS Development Tips Weekly
//  by Steven Lipton (C)2018, All rights reserved
//  For videos go to http://bit.ly/TipsLinkedInLearning
//  For code go to http://bit.ly/AppPieGithub
//

import UIKit

class ViewController: UIViewController {
    let upImage = UIImage(named:"thumb_up_48pt")
    let downImage = UIImage(named: "thumb_down_48pt")
    let inverseUpImage = UIImage(named: "invert_thumb_up_48pt")
    let inverseDownImage = UIImage(named: "invert_thumb_down_48pt")

   
    
    @IBOutlet weak var upButton: UIButton!
    @IBOutlet weak var downButton: UIButton!
    
    @IBAction func upButton(_ sender: UIButton) {
       sender.layer.mask?.contents = inverseUpImage?.cgImage
        downButton.layer.mask?.contents = downImage?.cgImage
    }
    
    @IBAction func downButton(_ sender: UIButton) {
        configureMask(button: upButton, with: upImage)
        configureMask(button: downButton, with: inverseDownImage)
    }
    
    func configureMask(button:UIButton, with image:UIImage!){
        let masklayer = CALayer()
        masklayer.frame.origin = CGPoint(x: 0, y: 0 )
        masklayer.frame.size = button.frame.size
        masklayer.contents = image.cgImage
        button.layer.mask = masklayer
    }
    
    override func viewDidLayoutSubviews() {
        configureMask(button: upButton, with: upImage)
        configureMask(button: downButton, with: downImage)
    }
    
    override func viewDidLoad() {
        super.viewDidLoad()
        // Do any additional setup after loading the view, typically from a nib.
        
    }

}