Is it possible to use RFCOMM API to write customize AT command to use bluetooth headset

Like switch radio channel by AT command

Depending on the Platform / Available API you can write your own custom AT commands or send anything to peer over RFCOMM for sure. (Like in Android using the SPP APIs) Not sure what you mean by switching Radio channels, Bluetooth is a frequency hopping technology constantly hoping around different channels