matted almost got it. This is what worked:

# exactly one end mapped
samtools view -bh -F 4 -f 8 in.bam > out1.bam
samtools view -bh -F 8 -f 4 in.bam > out2.bam

# exactly two ends mapped
samtools view -bh -F 12 in.bam > out3.bam

then merge and sort -n

Thank you all!

I wrote a java tool filtering SAM/BAm file using javascript (rhino). It's available at: https://github.com/lindenb/jvarkit#samjs-filtering-a-sambam-file-with-javascript )

To get "the reads unmapped or the mate unmapped" you can filter the BAM with:

java -jar dist/samjs.jar I=input.bam \
    VALIDATION_STRINGENCY=SILENT \
    SCRIPT_EXPRESSION="record.readUnmappedFlag || record.mateUnmappedFlag;" \
    OUT=result.bam

and then get back the FASTQs using picard/samToFastq.jar

To get "exactly one end or two ends mapped" you can filter the BAM with:

java -jar dist/samjs.jar I=input.bam \
    VALIDATION_STRINGENCY=SILENT \
    SCRIPT_EXPRESSION="!(record.readUnmappedFlag && record.mateUnmappedFlag);" \
    OUT=result.bam

This command will remove the unmapped reads and will output all the mapped reads into a new.bam file.

samtools view -bh -F 4 original.bam -> new.bam

-F option will skip the alignment that match a given flag and -f option will include those alignments.

Since it sounds like you want both reads of a pair that meet your conditions, I think you'll have to break it into several steps:

# exactly one end mapped
samtools view -b -F 4 -f 8 in.bam > out1.bam

# exactly two ends mapped
samtools view -b -F 12 in.bam > out2.bam

You can convert out1.bam and out2.bam to paired-end fastq files for use in assembly. Each read present in these bams should also have its mate present in the bam.

See http://picard.sourceforge.net/explain-flags.html for an easy way to play with flag settings. You may need to also filter out something like -F 1792 (no secondary alignments, failed reads, or PCR duplicates), depending on what flags the aligner set in generating in.bam.

sambamba has a powerful filtering syntax that allows it to be done in one step

https://github.com/biod/sambamba/wiki/%5Bsambamba-view%5D-Filter-expression-syntax

sambamba view -f bam -F "not (unmapped and mate_is_unmapped)" ${bam} > ${output.bam}