What you have proposed has some issues:

1. running the input straight into the volume pot is OK, but then follow it with a buffer stage, not straight to the pan pot. Also, a 10k linear pot with about 1k5 to ground is a good enough approximation of a log pot, and more accurate and reliable (log pots have very thin tracks at one end).
2. Your mute switch should have the wiper connected to the input of a stage, the NC to the signal, and the NO to ground - so that the signal is truly 0V when you MUTE.
3. A "pan pot" is normally used to route a mono signal to stereo. You want a "balance" control I think? Yours seems overly complex.
4. There also seems to be a problem that it is possible to short the input signal to ground by turning the vol pot and pan pot so that pin 3 shorts to pin2. The source might not like this.

An improved circuit which resolves these issues is below (only one channel shown):

^{simulate this circuit – Schematic created using CircuitLab}

The volume pot R5 is 10k dual linear, the balance R1 is a single 10k linear. The balance has a loss of 5/8 at centre, which is made up in the following stage, so unity gain is preserved (ideally R3 would be 6k, a better combination of R3/R4 giving a gain of 8/5 probably exists and is left as an exercise). You might also provide HF rolloff somewhere between 50kHz and 100kHz, by means of a parallel capacitor to R3.