The reduced residue system modulo n used here is the set of numbers k from the set {0,1,...,n-1} which satisfy gcd(k,n)=1. There are phi(n) = A000010(n) such numbers k. Cf. A038566. See also the Apostol reference p. 133, and the Wikipedia link.
This is the m=3 member of a family of sequences, call them rmnS(m) (reduced mod n sum), with entries rmnS(m;n):=sum(binomial(k+m-1,m),0<=k<=n-1 with gcd(k,n)=1), m>=0, n>=1. Recall gcd(0,n)=n.
The members for m=0, 1, and 2 are A000010(n), A023896(n) and A127415(n), respectively, where in the last two the offset for n=1 should be taken as 0 (not 1).
REFERENCES
T. Apostol, Introduction to Analytic Number Theory, Springer, 1986.
a(n) = (n*(n+2)/4!) *{n*(n+2) + mu(rad(n))*rad(n)} *phi(n)/n, n>=2, with rad(n) = A007947(n) the squarefree kernel of n, mu(n)=A008683(n), and phi(n)= A000010(n).
Note that phi(n)/n = A076512(n)/A109395(n) = phi(rad(n))/rad(n).
