Generally, two ways come to mind:

1. Emulating a HTTP Server using AHK

Using an own local web server (ideally a firsthand AHK implementation), you can directly communicate via AJAX calls. The server can then receive and delegate any kind of commands. For AHK, there exists Sparrow, but the project seems to be dead, and the source code links in the post are down; you'd have to get them from somewhere else (e.g. some svn repository). You could also implement a command line tool like KF Web Server and use it in conjunction with your script.

2. Defining your own protocol

Another solution would be to simply specify your own protocol and assign it to an executable, e.g. your script or some kind of dispatcher, that starts other applications depending on the parameters you pass it. This answer describes how to define your own protocol in windows. You can also check out the respective windows documentation.
If you're able to call custom protocols from within your browser with JavaScript strongly depends on your browser, and maybe its security settings. You'll need to check that out for your environment.

Security
No matter what solution you go with, giving a webpage the ability to run applications or even execute code opens up serious vulnerabilities. If an attacker finds out about your protocol, webserver, etc., they could easily inject their own stuff when you visit a website controlled by them. That's why you should at least implement two things:
a) Use some kind of authentication mechanism (e.g. HMAC). This makes exploitation at least a bit harder.
b) Never directly execute arguments as code. Always parse them and check their integrity.