I'll admit that I was a bit, um... chemically handicapped... last night when I tried to solve this puzzle. Don't drink and bitshift. So i've tidied it up a bit this morning and posted it as a more coherent answer. A refactor of shame.

There is no need for an operation to reverse compliment DNA that you've encoded - it is, by nature of being DNA, already reversed complimented. Complementation is built into the genetic code, and reversing a binary series is another way to say decode it backwards. Therefore, we can encode our DNA in any 2bit format we like, then by changing how we conceptualise what we have just written to memory, it is already reverse complimented.

>>> def twobit(DNA): return sum([('A','C','G','T').index(base)<<shift for shift,base in zip((0,2,4,6,8,10,12,14,16,18),DNA)])
>>> def bittwo(bits):  return ''.join([('A','C','G','T')[(bits >> x)&3] for x in (0,2,4,6,8,10,12,14,16,18) ])
>>> def revcomp(bits): return ''.join([('T','G','C','A')[(bits >> x)&3] for x in (18,16,14,12,10,8,6,4,2,0) ])
>>> twobit('AAACCGGGTT')
1026368
>>> bittwo(1026368)
'AAACCGGGTT'
>>> revcomp(1026368)
'AACCCGGTTT'

Note that decoding to the original string, and decoding to the reverse compliment, takes exactly the same number of operations.

If you want to do a 2-to-1 mapping of kmers and their reverse compliment to a single binary value, thats a different problem entirely. The old trick is to find the lesser of the two encoded or decoded values and only write/print that. Many people do this on the encoding step to reduce their kmer table size by half, for example:

>>> min( twobit('AAACCGGGTT'), twobit('AACCCGGTTT') )
1026368 # only this gets written to the kmer table.

However, it is but my opinion that such code suggests a design decision mistake. If you care about performance, this will hurt performance as it only takes 1 extra op to binary intersect a table twice as large, where it takes many ops to encode the string two ways and find out which is the smaller. On the other hand, if you care about memory, then there are much more efficient data structures than a kmer table, although of course they will be slower for many operations. This method of basically using computationally expensive lossy encoding is a poor compromise.

I think that the problem is solved using this code using incrementation.

inline unsigned int  X1( const std::string A,const int k ){
  unsigned int  result=V(A[0]);
  for( int i=1;i<k;i++) {
    result+= pow(4,i)*V(A[i]);
  }
  return result;
}


  // [[Rcpp::export]]
  inline IntegerVector rc_kmer4(const std::string A,const int n,const int k)
  {
    const int np=k-1;
    const int Lim=n-np;             //---- number of subsequence 
    const int beta=pow(4,k);
    const int N=pow(4,np);               
    unsigned short int fo,ba;// forword and backword of substring
    IntegerVector P(beta);    
    unsigned int x=X1(A,k);
    unsigned int y=X0(A,k,k);
    P[beta-x-1]++;
    P[y]++;
    for( int i=1;i<Lim;i++){
      fo=V(A[i-1]);
      ba=V(A[i+np]);
      x=(x-fo)/4+ba*N;
      y=4*y-beta*fo+ba;
      P[beta-x-1]++;
      P[y]++;
    }
    return P;
  }

It works with linear complexity. Think you for ideas @John