1st solution:

See: QuickGO can supply GO term information and GO annotation data via REST web services. http://www.ebi.ac.uk/QuickGO/WebServices.html

goid GO identifiers either directly or indirectly (descendent GO identifiers) applied in annotations

example:

$ curl -s 'http://www.ebi.ac.uk/QuickGO/GAnnotation?goid=GO:0005515&format=tsv' |   awk -F '   ' '($4!="-")' | head -n 3 | ~/src/variationtoolkit/bin/verticalize  
>>>    2
$1    DB           UniProtKB
$2    ID           A0A097
$3    Splice       -
$4    Symbol       Tbeta-a
$5    Taxon        7955
$6    Qualifier    -
$7    GO ID        GO:0003779
$8    GO Name      actin binding
$9    Reference    GO_REF:0000002
$10    Evidence     IEA
$11    With         InterPro:IPR001152
$12    Aspect       Function
$13    Date         20120616
$14    Source       InterPro
<<<    2

>>>    3
$1    DB           UniProtKB
$2    ID           A0A097
$3    Splice       -
$4    Symbol       Tbeta-a
$5    Taxon        7955
$6    Qualifier    -
$7    GO ID        GO:0003779
$8    GO Name      actin binding
$9    Reference    GO_REF:0000002
$10    Evidence     IEA
$11    With         InterPro:IPR016323
$12    Aspect       Function
$13    Date         20120616
$14    Source       InterPro
<<<    3

2nd solution:

use NCBI esearch with db=gene and the GO modifier:

http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=gene&term=%225515%22[GO]

You don't mention whether you want programmatic access or not. For web-based access, you can go to Amigo from the Gene Ontology folks themselves. For example:

http://amigo.geneontology.org/cgi-bin/amigo/term-assoc.cgi?term=GO:0005515

From that page, you can filter by species, evidence, etc.

For programmatic access, Pierre's answers are great. As another alternative, my group maintains mygene.info, which is optimized for fast JSON queries (suitable for AJAX applications). For example:

http://mygene.info/query?q=go:GO\:0005515

The downside is that it only supports nine of the most common species.