I’m guessing a hardware problem.
What happens if you unplug the guitar and use a different sound source?
Try to trace the signal path through from guitar to computer to narrow down where the problem starts.
The mixer seems to be perfectly happy with it’s green flashing lights, but the computer sound levels are to-the-moon too high. There are “normal” things that people do wrong when they connect things to a computer, but this is even far worse than that.
How are you getting into the computer? I don’t see anywhere you tell us. Newer Macs no longer have a Stereo Line-In like the older ones did and you’re stuck with an adapter.
This is what I use and it works with both Macs and Windows.
Apple (Upper Left) > System Preferences > Hardware > Sound > Input. Make sure your USB device is selected and see if a rational number of lights in the blue ball light meter flash when you play music. They should not all come on and stay on.
The USB connection may not have any provision to change the volume. This is probably normal.
OS X USB audio support doesn’t usually come with the ability to change the input level of the device in the system sound preferences. It depends on the device, but for most devices that’s the case. Since your device users the Core Audio USB drivers that come with Mac, it almost certainly is the case.